Condition when angle between two lines is ${\pi\over 3}$

As clarified in the comments, the correct formulation of the problem is:

Show that the angle between the lines whose direction cosines $(l_1,m_1,n_1)$ and $(l_2,m_2,n_2)$ satisfy the equations $$l+m+n=0\tag1$$ and $$fmn+gnl+hlm=0\tag2$$ is ${\pi\over 3}$ if $${1\over f}+{1\over g}+{1\over h}=0.\tag3$$

In fact as will be shown below even stronger statement with "if" replaced by "if and only if" holds. It will be additionally assumed $(l_1,m_1,n_1)\nparallel(l_2,m_2,n_2)$, $(f,g,h)\ne(0,0,0)$. Otherwise extra solutions are possible.

The entirety of the conditions implies that neither of $l_i,m_i,n_i$ is $0$. Indeed an assumption that any of the components - say $n_1$ - is $0$ results in combination with $(1)$ and $(2)$ in the equality $(f,g,h)=(0,0,0)$.

Observe now that equality $(1)$ $$ (l,m,n)\cdot(1,1,1)=0 $$ means that the vectors ${\bf a}_1=(l_1,m_1,n_1)$ and ${\bf a}_2=(l_2,m_2,n_2)$ are orthogonal to the vector ${\bf 1}=(1,1,1)$. This implies ${\bf a}_1\times {\bf a}_2 \parallel {\bf 1}$ or component-wise: $$ m_1n_2-n_1m_2=n_1l_2-l_1n_2=l_1m_2-m_1l_2=\frac{\sin\alpha}{\sqrt3},\tag4 $$ where $\alpha$ is the angle between ${\bf a}_1$ and ${\bf a}_2$ which is to be found. The last equality is due to $$ \sin^2\alpha=|{\bf a}_1\times {\bf a}_2|^2=\sum_{i=x,y,z}({\bf a}_1\times {\bf a}_2)_i^2. $$

Similarly the equation $(2)$ $$ (f,g,h)\cdot(mn,nl,lm)=0 $$ means that the vectors ${\bf b}_1=(m_1n_1,n_1l_1,l_1m_1)$ and ${\bf b}_2=(m_2n_2,n_2l_2,l_2m_2)$ are orthogonal to the vector ${\bf c}=(f,g,h)$. This in turn implies that the vector ${\bf c}$ is collinear to the vector product ${\bf b}_1\times {\bf b}_2$, which reads component-wise: $$\begin{align} (f,g,h)& =A\big(l_1l_2(n_1m_2-m_1n_2),m_1m_2(l_1n_2-n_1l_2),n_1n_2(m_1l_2-l_1m_2)\big)\\ &=\frac{A\sin\alpha}{\sqrt3}(l_1l_2,m_1m_2,n_1n_2)\tag5 \end{align} $$ where $A$ is a non-zero constant.

We start now with the proof of the "if" part. The equations $(3)$ and $(5)$ imply: $$ \frac{1}{l_1l_2}+\frac{1}{m_1m_2}+\frac{1}{n_1n_2}=0\tag6 $$ or $$ m_1m_2n_1n_2+n_1n_2l_1l_2+l_1l_2m_1m_2=0.\tag7 $$ With help of $(1)$ the equation reads: $$\begin{align} 0&=(l_1+m_1)(l_2+m_2)(l_1l_2+m_1m_2)+l_1l_2m_1m_2\\ &=(l_1l_2+m_1m_2+l_1m_2)(l_1l_2+m_1m_2+m_1l_2).\tag8 \end{align} $$ Now we can proceed with computation of $\cos\alpha={\bf a}_1\cdot {\bf a}_2$: $$\begin{align} \cos\alpha &=l_1l_2+m_1m_2+n_1n_2\\ &=l_1l_2+m_1m_2+(l_1+m_1)(l_2+m_2)\\ &=2(l_1l_2+m_1m_2)+l_1m_2+m_1l_2\\ &\stackrel{(8)}=\pm(l_1m_2-m_1l_2)\stackrel{(4)}=\pm\frac{\sin\alpha}{\sqrt3}.\tag9 \end{align} $$ Squaring the equation one finally obtains: $$ \cos^2\alpha=\frac13 \sin^2\alpha\implies \cos^2\alpha=\frac14 \implies \alpha=\frac\pi3.\tag{10} $$

As already mentioned the inverse implication $\alpha={\pi\over 3}\implies {1\over f}+{1\over g}+{1\over h}=0$ holds as well, since, provided the equality $(1)$, the equations $(6)-(10)$ are equivalence relations, so that the whole argument can be easily reversed.

In general if a, b, c and d, e, f are components of 2 vectors then the direction cosines are for the first vector : $\frac{a}{\sqrt{a^2 + b^2 + c^2}}$ and $\frac{b}{\sqrt{a^2 + b^2 + c^2}}$ and $\frac{c}{\sqrt{a^2 + b^2 + c^2}}$ similarly for the second vector.

Then the angle $\theta$ between these vectors is

$$cos\theta = \frac{ad + be + cf}{\sqrt{a^2 + b^2 + c^2}\sqrt{d^2 + e^2 + f^2}}$$

Using the definition of dot product. In other words $cos\theta$ equals the sum of product of the directional cosines of corresponding vectors. In this question we have

$$cos\theta = l(fmn) + m(gnl) + n(hlm) = lmn(f + g + h)$$

The next step is to prove this equal 1/2.

From the question we have

$$l + m + n = 0$$ $$l^2 + m^2 + n^2 = 1$$ Since there are three unknowns but only 2 equations, we can only express l and n in terms of m :

$$\implies l^2 + lm + m^2 = \frac{1}{2}$$

$$\implies l = \frac{-m + D}{2}$$ $$\implies n = \frac{-m - D}{2}$$ Where $D = \sqrt{2 - 3m^2}$. It does not matter which root we take for l the other root will be for n.

Also we have other conditions:

$$fmn + gnl + hlm = 0 ... (1)$$ $$(fmn)^2 + (gnl)^2 + (hlm)^2 = 1 ... (2)$$ $$\frac{1}{f} + \frac{1}{g} + \frac{1}{h} = 0 \implies fg + gh + fh = 0... (3)$$

Put l and n into (1) we have $$-fm(m + D) - g(1 - 2m^2) + hm(-m + D) = 0...(4)$$

Put l and n into (2) we have $$f^2m^2(2 - 2m^2 + 2mD) + g^2(1 - 2m^2)^2 + h^2m^2(2 - 2m^2 -2mD) = 4...(5)$$

Eliminate f from (3) and (4) we have $$f^2m(m + D) + fg(2mD - 2m^2 + 1) + g^2(1 - 2m^2) = 0$$ $$\implies f = \frac{2m^2 - 2mD - 1\pm\sqrt{(2mD -2m^2 + 1)^2 - 4(m + D)m(1 - 2m^2)}}{2m(m + D)}g$$ $$\implies f = \frac{2m^2 - 2mD - 1 \pm 1}{2m(m + D)}g$$ We take plus 1 for simplicity since the other root may give a result making $cos\theta$ greater than 1 or less than -1. $$f = \frac{m - D}{m + D}g$$

Eliminate h from (4) and (5) gives $$f^2m^2(2 - 2m^2 + 2mD) + g^2(1 - 2m^2)^2 + fgm(m + D)(1 - 2m^2) = 2$$ Put f into this equation we get $$m^2(m - D)^2g^2 + g^2(1 - 2m^2)^2 + (m - D)mg^2(1 - 2m^2) = 2$$ $$\implies g^2(1 - Dm - m^2) = 2$$ $$\implies g = \sqrt{\frac{2}{1 - Dm - m^2}}$$ $$\implies f = \frac{m - D}{m + D}\sqrt{\frac{2}{1 - Dm - m^2}}$$ $$\implies h = \frac{D - m}{2m}\sqrt{\frac{2}{1 - Dm - m^2}}$$

Hence put l, m, n, f, g, h to the cosine expression we get $$cos\theta = lmn(f + g +h) = \frac{-(m - D)}{2}m\frac{-(m + D)}{2}\sqrt{\frac{2}{1 - Dm - m^2}}(\frac{m - D}{m + D} + 1 + \frac{D - m}{2m})$$ $$=\frac{m - D}{4}\sqrt{\frac{2}{1 - Dm - m^2}}$$ $$=\frac{1}{4}\sqrt{\frac{2(m^2 + 2 - 3m^2 - 2Dm)}{1 - Dm - m^2}} = \frac{1}{2}$$