Radius of the spherical image of a circle

As you can see from the picture, the geometry is symmetric about the vertical axis, in other words the answer depends only on $R$ and $A=|a|$. So if we take our circle to be centered on the $x$-axis, as @Hagen has suggested, we only need to look at the intersection of the picture with the $(x,z)$-plane. The map from the $x$-axis to the circle $x^2+z^2=1$ is: $$ \xi\mapsto \left(\frac{2\xi}{\xi^2+1},\frac{\xi^2-1}{\xi^2+1}\right)\,. $$ Now you have to plug in $A+R$ and $A-R$ for $\xi$, find the two points the formula gives you, and the distance between them is the diameter of the circle you want, again making use of @Hagen’s suggestion. Looks like very messy algebra, and I do wonder whether there’s a slicker way to do it.

I suppose I should boast that I learned this stuff out of Ahlfors’s book, with Ahlfors himself standing in front of the class. He was a superb teacher.

Ahlfors gives a slick derivation of the chordal distance formula $$ d(z,z') = \frac{2|z-z'|}{\sqrt{(1+|z|^2)(1+|z'|^2)}} $$ on page 20 of his book. Here $z,z' \in \mathbb C$ and the chordal distance is by definition the distance of $z$ to $z'$ after applying stereographic projection to them.

Taking $z= A+R$ and $z'=A-R$ as suggested by @lubin this simplifies to $$ d(A+R,A-R) = \frac{2R}{\sqrt{(1+(A+R)^2)(1+(A-R)^2)}} = \frac{2R}{\sqrt{1+2A^2+2R^2+(A^2-R^2)^2}} $$ and dividing this by $2$ yields the formula you're looking for.

Since you ask in a comment how to see that the diametrically opposite points $A+R$ and $A-R$ are mapped to diametrically opposite points: observe that the circles around the origin through $A\pm R$ are tangent to the circle $C$ of radius $R$ around $A$. The stereographic projection maps them to circles of latitude tangent to the image of the circle $C$.

The algebra is not all that messy:

Let $x = (x_1,x_2,x_3)$ and $x'=(x_1',x_2',x_3')$ be the points on $S^3$ corresponding under stereographic projection to $z$ and $z'$, respectively. Then $\lVert x\rVert^2 = 1=\lVert x'\rVert^2$ and $$ \begin{align*} \lVert x - x'\rVert^2 &= (x_1 - x_1')^2 + (x_2-x_2')^2 + (x_3-x_3')^2 \cr &= 2 - 2(x_1 x_1' + x_2 x_2' + x_3 x_3'). \end{align*} $$ After reminding the reader of $\DeclareMathOperator{\Re}{Re}\DeclareMathOperator{\Im}{Im}$ $$ (x_1,x_2,x_3) = \frac{1}{1+|z|^2}\left(2\Re z, 2\Im z, |z|^2-1\right) = \frac{1}{1+|z|^2}\left(z+\bar{z}, z-\bar{z}, |z|^2-1\right), $$ Ahlfors computes $$ \begin{align*} x_1 x_1' + x_2 x_2' + x_3 x_3' &= \frac{(z+\bar{z})(z'+\bar{z'}) + (z-\bar{z})(z'-\bar{z'}) + (|z|^2-1)(|z'|^2-1)}{(1+|z|^2)(1+|z'|^2)} \cr &= \frac{(1+|z|^2)(1+|z'|^2) - 2|z-z'|^2}{(1+|z|^2)(1+|z'|^2)} \end{align*} $$ and obtains $$ \lVert x - x'\rVert = \frac{2|z-z'|}{\sqrt{(1+|z|^2)(1+|z'|^2)}}. $$

I think I managed to write it properly.

Let's denote the stereographic projection from $\Bbb C$ to the sphere by $\varphi$. Let's denote $C(a,R)$ the circle with center $a$ and radius $R$.

Assume first that $a\in[0,\infty[$. Then since $\varphi$ maps circles into circles, the diameter of the circle image is $$D = \sup\{d(\varphi(z),\varphi(a+R)):z\in C(a,R)\}.$$

The approach is to find a bound for $$\{d(\varphi(z),\varphi(a+R)):z\in C(a,R)\}$$

and then see that the bound is reached at a point in the circle so it has to be the sup.

So, observe that

\begin{align} d(\varphi(z),\varphi(a+R)) &= \frac{2\lvert z - (a+R)\rvert}{\sqrt{(1+\lvert z\rvert^2)(1+\lvert a + R\rvert^2)}} \\ &\leq \frac{2(\lvert z-a\rvert + R)}{\sqrt{(1+\lvert z\rvert^2)(1+\lvert a + R\rvert^2)}} \\ &= \frac{4R}{\sqrt{(1+\lvert z\rvert^2)(1+\lvert a + R\rvert^2)}} \tag{1} \end{align}

In the other hand, if $z = x + iy$ is in $C(a,R)$ then $$ R^2 = (x-a)^2 + y^2,$$

so

\begin{align} \lvert z \rvert^2 &= x^2 + R^2 - (x-a)^2 \\ &= x^2 + R^2 - x^2 + 2xa - a^2 \\ &= R^2 + 2xa - a^2\tag{2} \end{align}

but since $z$ is in the circle $\Re z = x \in [a-R,a+R]$. So by (2) \begin{align} \lvert z \rvert^2 &= R^2 + 2xa - a^2 \\ &\geq R^2 + 2(a-R)a - a^2 \\ &= (a-R)^2. \end{align}

So

\begin{align} 1 + \lvert z\rvert^2 &\geq 1 + (a-R)^2 \\ \frac{1}{\sqrt{1 + \lvert z\rvert^2}} &\leq \frac{1}{\sqrt{1 + (a-R)^2}}. \end{align}

By (1)

$$ d(z,a+R) \leq \frac{4R}{\sqrt{(1+\lvert a - R\rvert^2)(1+\lvert a+R\rvert^2)}} $$

and the bound is reached at $z=a-R$, so the radius is $$\frac{2R}{\sqrt{(1+\lvert a - R\rvert^2)(1+\lvert a+R\rvert^2)}}.$$

Now, notice that the formula $$d(\varphi(z),\varphi(z'))=\frac{2|z-z'|}{\sqrt{(1+|z|^2)(1+|z'|^2)}}$$ is invariant under rotations.

So if $a\in\Bbb C \setminus [0,\infty[$, in particular it is not $0$, so rotate by $\frac{\bar a}{\lvert a \rvert}$ and we are done.

One Approach: We want $F:\mathbb R^2 \to \mathbb S^2$ where $\mathbb S^2 = x^2+y^2+z^2=2$.

You can calculate the image of $(x,y)$.While trying to solve,I got something like: $$ F(x,y)=\left(\frac{2x}{x^2+y^2+1},\frac{2y}{x^2+y^2+1},\frac{x^2+y^2-1}{x^2+y^2+1}\right)$$

Now,As far as I know,circles are sent to circles under this projection,so,get any 3 points of one circle and find their images.As any three points completely describe the cirle,we are done.

P.S:I am not sure about the formula,so please check it from the book you are using.