I'm trying to improve my algorithm for selecting points in perspective view mode (OpenGL/Qt/C++). The current implementation works as follows.
- The user clicks on a certain
(x,y)position (in Window space) - Traversing the pipeline from Window space back to World space (that is, Window → NDC → Clip → Eye → World), this defines a point
aon thenearclipping plane and a pointbon thefarclipping plane - The distances from the line between
aandband the vertices (available in World space) are determined (see e.g. this article on Wikipedia) - The point with the minimum distance to the line is selected
Now, this works fine when using orthogonal view mode. However, when visualizing something in perspective view mode, there are some cases where the result might be a bit off.
As example, let's consider the wireframe of a cube. In the illustration below, the point (x,y) is the point where all white lines meet. These lines are orthogonal to the line from a to b which is not visible in this orientation — the second illustration shows a rotated view.
The point that eventually gets selected is highlighted in green. However, as the blue point seems to be closer to (x,y) in perspective mode, the user probably expects this point to get selected instead. Somehow the computed distances have to be scaled in order to take this effect into account. How is this usually done?

