The question is a slight extension of the question given here: Check whether a point is within a 3D Triangle
There is an elegant solution to this given by W. Heidrich, Journal of Graphics, GPU, and Game Tools,Volume 10, Issue 3, 2005.
Let $\vec{u}=P_2-P_1$, $\vec{v}=P_3-P_1$, $\vec{n}=\vec{u}\times\vec{v}$, $\vec{w}=P-P_1$. We then have directly the barycentric coordinates of the projection $P'$ of $P$ onto $T$ as
- $\gamma=\left[(\vec{u}\times\vec{w})\cdot\vec{n}\right]/\vec{n}^2$$\gamma=(\vec{u}\times\vec{w})\cdot\vec{n}/\left\lVert\vec{n}\right\rVert^2$
- $\beta=\left[(\vec{w}\times\vec{v})\cdot\vec{n}\right]/\vec{n}^2$$\beta=(\vec{w}\times\vec{v})\cdot\vec{n}/\left\lVert\vec{n}\right\rVert^2$
- $\alpha=1-\gamma-\beta$
The coordinates of the projected point is
- $P'=\alpha P_1+\beta P_2 +\gamma P_3$
The point $P'$ lies inside $T$ if
- $0\leq\alpha\leq 1$,
- $0\leq\beta\leq 1$, and
- $0\leq\gamma\leq 1$.