(Essentially this question, but in 3 dimensions.)
In 3D space, points $a$ and $b$ are known. In addition, a unit vector $\hat{t}$ is known. Assume $b-a$ and $\hat{t}$ are not parallel.
The goal: determine the center point of a circle which passes through both $a$ and $b$ and is tangent to $\hat{t}$ at point $a$.
(The circle will then lie on the plane spanned by $b-a$ and $\hat{t}$.)
My attempt:
Let $c$ be the center of the circle (the goal).
Get a vector in the direction of the normal of the plane $n$:
$$n=(b-a)\times\hat{t}$$
Find three equations that can be used to solve for $[c_x, c_y, c_z]$.
First, using the fact that $(a-c)$ and $\hat{t}$ must be perpendicular, $(a-c)\cdot\hat{t}=0$, or
$t\cdot c = t \cdot a \tag{1}$
Second, $(b-c)$ must be perpendicular to $n$:
$n \cdot c = n \cdot b \tag{2}$
Third, $c$ must lie on the perpendicular bisecting plane of $a$ and $b$:
$(b-a)\cdot c = (\frac{b-a}{2})\cdot (b-a) \tag{3}$
This gives the equation
$$ \begin{bmatrix} t_x & t_y & t_z \\ n_x & n_y & n_z \\ b_x-a_x & b_y-a_y & b_z-a_z \end{bmatrix} \begin{bmatrix} c_x \\ c_y \\ c_z \end{bmatrix} = \begin{bmatrix} t \cdot a \\ n \cdot b \\ (\frac{b-a}{2})\cdot (b-a) \end{bmatrix} $$
However, solving that (by plugging in $a$, $b$, and $\hat{t}$ and doing it on a computer) is giving me wildly off results. $c$ is in the correct plane, but that's about it.
