The two thingsTwo alternatives I'd recommend would be either:
A)A) Calculate all the matrices by hand. ThisIt doesn't require any matrix4 classes or anything, just a lot of math. You'd have to calculate something like
NDC_Coordinates=PerspectiveDivide(ProjectionCameraviewModelview*Vector)
NDC_Coordinates=PerspectiveDivide(Projection*Cameraview*Modelview*Vector) The ModelviewModelview one is optional, so it'll be the cameraviewCameraView that will probably contain all the annoying rotation and translation stuff.
B)B) Make a Doom styleDoom-style renderer. I won't elaborate much here because there are tons of resources online.
http://www.permadi.com/tutorial/raycast/index.html
http://lodev.org/cgtutor/raycasting.html