Most efficient way to get the closest point to a 3d rectangle

This is actually a pretty simple problem if you first rotate the rectangle so it lies flat and at right angles on one plane, and also apply the same rotation to the player location. Now, everything can be done with Cartesian coordinates to determine the closest point. In Cartesian space, it's trivial to find the point on the rectangle, then reverse the rotation to get your final answer.

Here's a basic illustration of how this works...

Let's say you rotate the rectangle so it lies flat on the Z plane, so you're working just with X and Y. Given that and looking only at the player location in X and Y (ignoring Z), the player will either be inside the rectangle, or outside the rectangle.

If inside, the point on the rectangle is simply the player X, Y coordinates on the rectangle.

If outside, it's also surprisingly trivial as well. Rather than answer the outside case, I'm going to point you to https://math.stackexchange.com/questions/356792/how-to-find-nearest-point-on-line-of-rectangle-from-anywhere which has a nice, extremely simple answer to that problem.

Here's a key illustration from the above linked Math stack answer, which shows how for a point outside the rectangle, it's either a simple line to an edge, or one of the corners, depending on the point location...

Now with the closest point on the rectangle located in Cartesian coordinates, you reverse the rotation on that point to find the desired point on the rectangle in 3D space.

The only thing I've left you as a challenge is to compute the rotational transformation needed for your rectangle. I'm going to say though that defining a plane (to work out the needed rotation) with 4 points is going to be a problem, because you'll never have enough precision to ever have 4 points truly lying on a flat plane (unless it's already flat on X, Y, or Z). The best thing would be for you to pick 3 of the points on the rectangle to define a triangle, then compute the normal from them (and from there the rotation).

Let \$\vec{n}\$ be a normalized normal vector of the plane (say, the cross-product of \$\vec{DA}\$ and \$\vec{DC}\$ divided by its length).

Let \$Q = P - (\vec{DP}\cdot \vec{n}) \vec{n}\$. (I.e., \$Q\$ is the projection of \$P\$ into the plane of \$ABCD\$. You seem to have a function for this already. The only missing part is to make sure that \$Q\$ belongs to the rectangle.)

Let \$x=(\vec{DQ}\cdot \vec{DC})/|DC|^2\$ and \$y=(\vec{DQ}\cdot \vec{DA})/|DA|^2\$. (That means, we find the place of \$Q\$ in a coordinate system in the rectangle that has its origin in \$D\$ and \$(1,0)\$ and \$(0,1)\$ in \$C\$ and \$A\$, respectively.) If \$x>1\$ or \$x<0\$, set \$x=1\$ or \$x=0\$, respectively; the same for \$y\$.

The point \$D + x\vec{DC} + y\vec{DA}\$ is the point you are looking for.

(Admittedly, this answer still leaves open the question of "most efficient way".)

Edit: In order to make this somewhat more efficient: The first step is unnecessary (the following steps would ignore the component perpendicular to the plane anyway); and it is possible to compute the projections (probably the most expensive parts of the algorithm) only when necessary.

Let \$a= \vec{DP}\cdot \vec{DC}\$. If \$a<0\$, let \$\vec{x}=\vec{0}\$. Else if \$a>l_c\$, where \$l_c=\vec{DC}\cdot\vec{DC}\$, let \$\vec{x}=\vec{DC}\$. Else let \$\vec{x}\$ be the projection of \$\vec{DP}\$ in the direction of \$\vec{DC}\$ (like above, e.g. as \$\frac{a}{l_c}\, \vec{DC}\$.).

Let \$b= \vec{DP}\cdot \vec{DA}\$. If \$b<0\$, let \$\vec{y}=\vec{0}\$. Else if \$b>\vec{DA}\cdot\vec{DA}\$, let \$\vec{y}=\vec{DA}\$. Else let \$\vec{y}\$ be the projection of \$\vec{DP}\$ in the direction of \$\vec{DA}\$.

The solution is \$D+\vec{x}+\vec{y}\$.

While this is certainly more efficient than my first attempt (source: count the operations of each type), on which it is based, I still cannot say how close it is to the 'most efficient' algorithm (if such exists) - and am looking forward to reading another answer that addresses this question (ideally comparing with all other possible algorithms and not only examples given here).