If you want to do it geometrically...

The inverse of the view-projection matrix, $K^{-1}$ is the matrix you want. Where $K = View * Projection$

If $\vec{v}$ is a point in homogeneous screen coordinates, $[x, y, 1]$ where $x$ and $y$ are whatever coordinates your $K$ matrix projects onto.

$$\vec{d} = K^{-1} * \vec{v}$$

Where $\vec{d}$ now represents a line in homogeneous coordinates, coming out of your camera origin. Now you'd iterate through the objects in view, and choose the closest one in front of the camera that this line intersects with.

But there are other ways!

Color Picking

Here, we render (into a buffer) each object in view with a slightly different color, which corresponds to the object ID. We then read it using glReadPixels(mouse_coordinates...) and decide which object we clicked. This is pretty easy to do in the new pipeline - you just write a color passthrough shader. Would not recommend pre OpenGL 3.0 (You'll have to do shenanigans to render the scene without interference)

Selection Buffer (Old pipeline)

If you're on the fixed pipeline, you get a nice little toy called the selection buffer to do the work for you.

Edit: This answer is only helpful in 3D

If you want to do it geometrically...

The inverse of the view-projection matrix, $K^{-1}$ is the matrix you want. Where $K = View * Projection$

If $\vec{v}$ is a point in homogeneous screen coordinates, $[x, y, 1]$ where $x$ and $y$ are whatever coordinates your $K$ matrix projects onto.

$$\vec{d} = K^{-1} * \vec{v}$$

Where $\vec{d}$ now represents a line in homogeneous coordinates, coming out of your camera origin. Now you'd iterate through the objects in view, and choose the closest one in front of the camera that this line intersects with.

But there are other ways!

Color Picking

Here, we render (into a buffer) each object in view with a slightly different color, which corresponds to the object ID. We then read it using glReadPixels(mouse_coordinates...) and decide which object we clicked. This is pretty easy to do in the new pipeline - you just write a color passthrough shader. Would not recommend pre OpenGL 3.0 (You'll have to do shenanigans to render the scene without interference)

Selection Buffer (Old pipeline)

If you're on the fixed pipeline, you get a nice little toy called the selection buffer to do the work for you.

If you want to do it geometrically...

The inverse of the view-projection matrix, K^{-1} is the matrix you want. Where $K = View * Projection$

If $\vec{v}$ is a point in homogeneous screen coordinates, $[x, y, 1]$ where x and y are whatever coordinates your K matrix projects onto.

$\vec{d} = K^{-1} * \vec{v}$

Where $\vec{d}$ now represents a line in homogeneous coordinates, coming out of your camera origin. Now you'd iterate through the objects in view, and choose the closest one in front of the camera that this line intersects with.

But there are other ways!

Color Picking

Here, we render (into a buffer) each object in view with a slightly different color, which corresponds to the object ID. We then read it using glReadPixels(mouse_coordinates...) and decide which object we clicked. This is pretty easy to do in the new pipeline - you just write a color passthrough shader. Would not recommend pre Opengl 3.0 (You'll have to do shenanigans to render the scene without interference)

Selection Buffer (Old pipeline)

If you're on the fixed pipeline, you get a nice little toy called the selection buffer to do the work for you.

If you want to do it geometrically...

The inverse of the view-projection matrix, $K^{-1}$ is the matrix you want. Where $K = View * Projection$

If $\vec{v}$ is a point in homogeneous screen coordinates, $[x, y, 1]$ where $x$ and $y$ are whatever coordinates your $K$ matrix projects onto.

$$\vec{d} = K^{-1} * \vec{v}$$

Where $\vec{d}$ now represents a line in homogeneous coordinates, coming out of your camera origin. Now you'd iterate through the objects in view, and choose the closest one in front of the camera that this line intersects with.

But there are other ways!

Color Picking

Here, we render (into a buffer) each object in view with a slightly different color, which corresponds to the object ID. We then read it using glReadPixels(mouse_coordinates...) and decide which object we clicked. This is pretty easy to do in the new pipeline - you just write a color passthrough shader. Would not recommend pre OpenGL 3.0 (You'll have to do shenanigans to render the scene without interference)

Selection Buffer (Old pipeline)

If you're on the fixed pipeline, you get a nice little toy called the selection buffer to do the work for you.