Thanks to Håkon Hægland for asking and answering this question, and especially for providing the key information from the hard-to-obtain paper: Wolfgang Heidrich, 2005, Computing the Barycentric Coordinates of a Projected Point, Journal of Graphics Tools, pp 9-12, 10(3).
I was web searching for exactly this information. Among the top hits were a question on the gamedev.stackexchange.com site, and this one. After implementing the technique presented here by Håkon, I posted that code and a link to this question at the game dev site: Easy way to project point onto triangle (or plane). Later it occurred that the C++/Eigen implementation might be useful to readers of this question:
bool pointInTriangle(const Eigen::Vector3f& query_point, const Eigen::Vector3f& triangle_vertex_0, const Eigen::Vector3f& triangle_vertex_1, const Eigen::Vector3f& triangle_vertex_2) { // u=P2−P1 Eigen::Vector3f u = triangle_vertex_1 - triangle_vertex_0; // v=P3−P1 Eigen::Vector3f v = triangle_vertex_2 - triangle_vertex_0; // n=u×v Eigen::Vector3f n = u.cross(v); // w=P−P1 Eigen::Vector3f w = query_point - triangle_vertex_0; // Barycentric coordinates of the projection P′of P onto T: // γ=[(u×w)⋅n]/n² float gamma = u.cross(w).dot(n) / n.dot(n); // β=[(w×v)⋅n]/n² float beta = w.cross(v).dot(n) / n.dot(n); float alpha = 1 - gamma - beta; // The point P′ lies inside T if: return ((0 <= alpha) && (alpha <= 1) && (0 <= beta) && (beta <= 1) && (0 <= gamma) && (gamma <= 1)); }