A more general formula exists, for star polygon too and its smoother variations:
$$1=\frac{\sqrt{x^2+y^2} \cos \left(\frac{2 \sin ^{-1}\left(k \cos \left(n \tan ^{-1}(x,y)\right)\right)+\pi m}{2 n}\right)}{\cos \left(\frac{2 \sin ^{-1}(k)+\pi m}{2 n}\right)}$$
or, with polar coordinates:
$$\rho = \frac{\cos \left(\frac{2 \sin ^{-1}(k)+\pi m}{2 n}\right)}{\cos \left(\frac{2 \sin ^{-1}(k \cos (n \phi ))+\pi m}{2 n}\right)}$$
where
$\phi$ - angle;
$\rho$ - radius;
$n$ - number of convex vertices;
$m$ - determines how many vertices the sides will lie on one straight line;
$k$ - hardness - for $k=0$ we get a circle regardless of other parameters, for $k=1$ - a polygon with straight lines, with intermediate values from $0$ to $1$ - intermediate figures between the circle and the polygon.
For more details see this paper (at russian)
