I am drawing an empty texture over a framebuffer with a fragment shader active to try and draw a sky, but I need the direction each pixel is facing in (for correct linear perspective) to draw the right colours.
How can I calculate the direction vector for each fragment (pixel) from coords in the empty texture, aspect ratio, FOV, and camera orientation?
Other approaches I have tried have the sky misalign to the objects in the scene when rotating.
The basic idea is that you can construct a vector representing a position in clip space, and then "unproject" it to get that position in "world space", then normalise that vector to get a direction. World space is in quotes because the camera is always at the origin in said "world space". Perhaps sky space would be a better term.
If you construct the unprojected vector in the vertex shader for each vertex of the empty texture you're drawing, and set a varying vec3 to it, you can then let the varying variable be interpolated and normalise it in the fragment shader, cutting out potentially hundreds of thousands of fragment shader matrix multiplications and only have them done for the corners of the screen.
The GLSL vertex shader code, where VertexTexCoord.st is the UV coords on the stretched empty texture being drawn (a vec2, so it fills both x and y in the vec4 constructor) and clipToSky is the "unprojection matrix" explained below:
varying vec3 directionPreNormalise; // ... vec3 directionPreNormalise = ( clipToSky * vec4( VertexTexCoord.st * 2.0 - 1.0, -1.0, 1.0 ) ).xyz; And then in the fragment shader, if you even need to normalise it:
vec3 direction = normalize(directionPreNormalise); VertexTexCoord's components are in the range 0 to 1, but we want them in the range -1 to 1, since that's what clip space has.
If you want to understand the effect of the z, and whether it should be -1.0 or 0.0 for you: The -1.0 z coordinate can be anything, and this still works. The XY-aligned plane in clip space based on your stretched texture is transformed into a (where XYZ are aligned with the view frustum) XY-aligned plane in the view frustum, and the Z coordinate of that output plane depends on the Z coordinate in clip space pre-transformation. The relationship is nonlinear. -1.0 corresponds to a position on the near plane* (which I feel is non-arbitrary), assuming your graphics API has clip space go from -1 to 1 on the z axis. Having a z of -2.0 pre-transform also works for -1 to 1 clip spaces, so you should be OK to use whatever still, but 0.0 would more likely correspond to your near plane if you want something non-arbitrary (I haven't used an API like that). The XY-aligned plane in the view frustum grows as you increase the pre-transform z, "sliding along the edges of the view frustum", at such a rate that direction in the code above doesn't change with the pre-transform z.
*Note that the use of .xyz with no division by the w coordinate means that directionPreNormalise is no longer a position on the near plane, but it all still works fine.
The 1.0 w coordinate is just for homogenous coordinates. Changing it appears to have a FOV-changing effect.
The "unprojection matrix" would be, in Lua code using my maths library:
mat4.inverse(projectionMatrix * cameraMatrixStationary)
Where projectionMatrix is the matrix that takes vectors from camera space to clip space (which is where FOV and aspect ratio come in) and cameraMatrixStationary is a world space -> camera space matrix constructed from your camera orientation quaternion, but the position is set to the origin. If it is off the origin, the sky that gets rendered gets offset in a curious fashion.
Source for the research: I made a matrix visualiser and I used it to draw various wireframe graphics representing matrices, planes, the clip space cube, etc to see exactly what was going on. (It was originally intended for something else.)