There is an excellent answer to this question on the Math community of Stack Exchange: Determine if projection of 3D point onto plane is within a triangle. Which is, in turn, just a summary of this paper in Journal of Graphics Tools: Wolfgang Heidrich, 2005, Computing the Barycentric Coordinates of a Projected Point, Journal of Graphics Tools, pp 9-12, 10(3).
The transformation of the original query point to barycentric coordinates can be thought of allowing a simple conversion to a point (P’) on the plane of the triangle, or equivalently, of testing whether the query point is within an infinite triangular prism extending perpendicularly from the triangle. Here is some C++ code using the Eigen library, returning a boolean value based on four points: the query point and the vertices of the triangle. (The comments come directly from the math.stackexchange.com page cited above, except that \$\vec u\$ is written as \$u\$ because of typesetting issues.)
bool pointInTriangle(const Eigen::Vector3f& query_point, const Eigen::Vector3f& triangle_vertex_0, const Eigen::Vector3f& triangle_vertex_1, const Eigen::Vector3f& triangle_vertex_2) { // u=P2−P1 Eigen::Vector3f u = triangle_vertex_1 - triangle_vertex_0; // v=P3−P1 Eigen::Vector3f v = triangle_vertex_2 - triangle_vertex_0; // n=u×v Eigen::Vector3f n = u.cross(v); // w=P−P1 Eigen::Vector3f w = query_point - triangle_vertex_0; // Barycentric coordinates of the projection P′of P onto T: // γ=(u×w)⋅n/‖n‖² float gamma = u.cross(w).dot(n) / n.dot(n); // β=(w×v)⋅n/‖n‖² float beta = w.cross(v).dot(n) / n.dot(n); float alpha = 1 - gamma - beta; // The point P′ lies inside T if: return ((0 <= alpha) && (alpha <= 1) && (0 <= beta) && (beta <= 1) && (0 <= gamma) && (gamma <= 1)); }