I have a 3D plane defined by three points: $P_0$ , $P_1$ and $P_2$. How to check whether a point $P$ is located right on and inside the 3D triangle?
So, for example, if I have a plane defined by $({0,0,0})$, $({10,0,0})$ and $({0,10,0})$, then the point $({50,0,0})$ is considered not located on the plane, whereas the point $({5,0,0})$ is.
You can find if a point is in a triangle using barycentric coordinates.
If you have a triangle with vertices $A$, $B$ & $C$, & a point P in the plane of the triangle, you simply need to find:
$AreaABC = \frac{ \left| \overline{AB} \times \overline{AC} \right| }{ 2 }$
$\alpha = \frac{ \left| \overline{PB} \times \overline{PC} \right| }{ 2AreaABC }$
$\beta = \frac{ \left| \overline{PC} \times \overline{PA} \right| }{ 2AreaABC }$
$\gamma = 1 − \alpha − \beta$
Here $\alpha$ is the ratio of the area of a subtriangle $PBC$ over the area of the whole triangle $ABC$, as shown in this image from Peter Shirley's book:
If ALL of the following 4 restrictions are met:
then the point P is inside the triangle.
If ANY of $\alpha$,$\beta$,$\gamma$ are outside those ranges, or if the sum of $ \alpha + \beta + \gamma \ne 1 $ then the point P is not inside the triangle.
Notice how we exploit the 4th condition ($\alpha + \beta + \gamma = 1$) in finding $\gamma$.
Even though we find $\gamma$ that way, we still must check that $ 0 \le \alpha \le 1 $ and $ 0 \le \beta \le 1 $ and $ 0 \le \gamma \le 1 $ for the point to be inside the triangle.
$\gamma$ can easily be outside $[0, 1]$. Say $\alpha=0.99$ and $\beta=0.85$. Then $\gamma = 1 - 0.99 - 0.85 = -0.84$, and the point that got us those barycentric coordinates would be outside the triangle.
Things to observe:
When one of ($\alpha$, $\beta$, $\gamma$) is 1 and the other two are 0, then the point P is exactly at a vertex of the triangle.
When one of ($\alpha$, $\beta$, $\gamma$) is 0, and the other 2 coordinates are between 0 and 1, the point P is on an edge of the triangle.
Try to solve the system $$ x * (P_1 - P_0) + y * (P_2 - P_0) = P - P_0 $$ If it is solveable, the point $P$ lies on the plane. If in addition $x \geq 0$, $y \geq 0$, and $x + y \leq 1$, then $P$ lies inside the triangle.
Vectors $V_{01}=P_1-P_0$ and $V_{02}=P_2-P_0$ lie in the plane of the triangle, and $V_{01}\times V_{02}$ is normal to this plane. Let $V_{0p}=P-P_0$, and if $V_{0p} \cdot (V_{01}\times V_{02}) = 0$ then $P$ lies in the plane.
Let $P=cP_0+aP_1+bP_2$ where $c=1-a-b$. Then $P=(1-a-b)P_0+aP_1+bP_2$ or $V_{0p}=aV_{01}+bV_{02}$. If $0 \le a,b,c \le 1$ then $P$ lies in the triangle or on its edge.
Vector $V_{01} \times (V_{01}\times V_{02})$ is orthogonal to $V_{01}$ so $V_{02} \cdot(V_{01} \times (V_{01}\times V_{02}))b=V_{0p} \cdot(V_{01} \times (V_{01}\times V_{02}))$ can be solved for b.
Likewise $V_{02} \times (V_{01}\times V_{02})$ is orthogonal to $V_{02}$ so $V_{01} \cdot(V_{02} \times (V_{01}\times V_{02}))a=V_{0p} \cdot(V_{02} \times (V_{01}\times V_{02}))$ can be solved for a.
There may well be a more efficient algorithm, but here's a way to check that P is on the triangle defined by the three points
Compare the cross products $\vec {P_0 P_1} \times \vec {P_0 P_2} = \vec a$ and $\vec {P P_1} \times \vec {P P_2} = \vec b$. If $\vec b = k_1 \vec a$ with $k_1 \geq 0$, then P is on the plane and is on the correct side of the line through P1 and P2.
Now, compute $\vec {P P_2} \times \vec {P P_0} = \vec c$ and $\vec {P P_0} \times \vec {P P_1} = \vec d$. If $\vec c = k_2 \vec a$ and $\vec d = k_3 \vec a$ with $k_2, k_3 \geq 0$, then P lies within the triangle.
Of course, there's a lot of redundancy in this calculation. Once we know P is on the plane, we could just check one coordinate of the $\vec c$ and $\vec d$ to see that they have the correct sign. This is just projecting the problem onto one of the coordinate planes.
My old answer below explains how to check that P is on the infinite plane defined by the first three points, not how to check if it lies inside the triangle defined by the first three points, which is what the question intended.
A general plane equation is Ax + By + Cz = D. The 3-dimensional vector <A,B,C> is perpendicular to the plane.
(Note that there is not a unique equation of this form; you can multiply the equation by any nonzero number and get another equation of the plane. --So you can try to find a solutions with D=1, and if that doesn't work, use D=0.)
If you find an equation of the plane defined by your first three points, you can just plug in the fourth point to check if it satisfies the equation.
You can solve for the plane equation by hand by setting up equation from the first three points; but a more efficient method is to take the cross product of the vector $\vec {P_0 P_1}$ and the vector $\vec {P_0 P_2}$, and to use the resulting vector as <A,B,C>. You'll find it very useful to understand the cross product if you like working in 3-D space.
