This is a saddle point. I would suggest using derivative operators to spot such discontinuities. In fact, Haralick presents a broad overview of these methods in his Topographic Primal Sketch.
The use of approximate Facet model would give you a speed boost. I used them for edge detection here.
There is also one implementation available here.