@gmagno the approach is usually to project the mouse movement on the objects or scene and calculate deltas in that space, you do this by casting a ray and calculate intersection. Search for ray casting, project and unproject, I am rough on the details as I didn't work in CG for years now. hope that helps.

@underscore_d No, I take this on case by case. I wrote this 3 yrs ago. I dont remember my rational and dont have time to go through that again. But i think the op probably used new which did alot of random allocations and causing alot of cache misses. You are trying to generalize what am saying which is not true. I am also not in game dev anymore....