Playing with XNA Triangle Picking Sample I found out that it does not work well if you scale the world matrix of the objects you want to pick. When I dug into the implementation I found this comment in the RayIntersectsModel method:
// The input ray is in world space, but our model data is stored in object // space. We would normally have to transform all the model data by the // modelTransform matrix, moving it into world space before we test it // against the ray. That transform can be slow if there are a lot of // triangles in the model, however, so instead we do the opposite. // Transforming our ray by the inverse modelTransform moves it into object // space, where we can test it directly against our model data. Since there // is only one ray but typically many triangles, doing things this way // around can be much faster. After the comment they actually transformed the ray:
ray.Position = Vector3.Transform(ray.Position, inverseTransform); ray.Direction = Vector3.TransformNormal(ray.Direction, inverseTransform); With this implementation, picking suffered from "short-sightedness" if you scaled the models: it could only pick those objects, that were close enough to it. Even the ray-boundingSphere intersection test, which implementation is hardcoded into XNA, failed in the same way.
I fixed this by "doing the wrong thing" - I actually started transforming every vertex by the model's world matrix and to fix the boundingSphere test I added this code:
Quaternion rot; Vector3 scale, trans; modelTransform.Decompose(out scale, out rot, out trans); float maxScale = Math.Max(Math.Max(scale.X, scale.Y), scale.Z); boundingSphere.Center = Vector3.Transform(boundingSphere.Center, modelTransform); boundingSphere.Radius *= maxScale; This obviously is not optimal and I wanted to know if there is a way to actually transform the ray back to the model's space with the model's inverted matrix, while making it work for scaled matrices?
SOLUTION: Thanks to Nathan's answer I found a way to fix the ray scaling - just renormalize the ray direction:
ray.Position = Vector3.Transform(ray.Position, inverseTransform); ray.Direction = Vector3.TransformNormal(ray.Direction, inverseTransform); //ADD THE FOLLOWING LINE: ray.Direction.Normalize(); SOLUTION UPDATE: As I tested the app, I found that Nathan was indeed completely right and another change was necessary. Here is the full code for the correct RayIntersectsModel() method:
static float? RayIntersectsModel(Ray ray, Model model, Matrix modelTransform, out bool insideBoundingSphere, out Vector3 vertex1, out Vector3 vertex2, out Vector3 vertex3) { vertex1 = vertex2 = vertex3 = Vector3.Zero; ... Matrix inverseTransform = Matrix.Invert(modelTransform); // STORE WORLDSPACE RAY. Ray oldRay = ray; ray.Position = Vector3.Transform(ray.Position, inverseTransform); ray.Direction = Vector3.TransformNormal(ray.Direction, inverseTransform); ray.Direction.Normalize(); // Look up our custom collision data from the Tag property of the model. Dictionary<string, object> tagData = (Dictionary<string, object>)model.Tag; if (tagData == null) { throw new InvalidOperationException( "Model.Tag is not set correctly. Make sure your model " + "was built using the custom TrianglePickingProcessor."); } // Start off with a fast bounding sphere test. BoundingSphere boundingSphere = (BoundingSphere)tagData["BoundingSphere"]; if (boundingSphere.Intersects(ray) == null) { // If the ray does not intersect the bounding sphere, we cannot // possibly have picked this model, so there is no need to even // bother looking at the individual triangle data. insideBoundingSphere = false; return null; } else { // The bounding sphere test passed, so we need to do a full // triangle picking test. insideBoundingSphere = true; // Keep track of the closest triangle we found so far, // so we can always return the closest one. float? closestIntersection = null; // Loop over the vertex data, 3 at a time (3 vertices = 1 triangle). Vector3[] vertices = (Vector3[])tagData["Vertices"]; for (int i = 0; i < vertices.Length; i += 3) { // Perform a ray to triangle intersection test. float? intersection; RayIntersectsTriangle(ref ray, ref vertices[i], ref vertices[i + 1], ref vertices[i + 2], out intersection); // Does the ray intersect this triangle? if (intersection != null) { // RECOMPUTE DISTANCE IN WORLD SPACE: Vector3 vertexA = Vector3.Transform(vertices[i], modelTransform); Vector3 vertexB = Vector3.Transform(vertices[i+1], modelTransform); Vector3 vertexC = Vector3.Transform(vertices[i+2], modelTransform); RayIntersectsTriangle(ref oldRay, ref vertexA, ref vertexB, ref vertexC, out intersection); // If so, is it closer than any other previous triangle? if ((closestIntersection == null) || (intersection < closestIntersection)) { // Store the distance to this triangle. closestIntersection = intersection; // Store the three vertex positions in world space. vertex1 = vertexA; vertex2 = vertexB; vertex3 = vertexC; } } } return closestIntersection; } } 
