"Clean code" is not an end in itself, it is a means to an end. The main purpose of refactoring bigger functions into smaller ones and cleaning up the code in other ways is to keep the code evolvable and maintainable.
When picking such a very specific mathematical algorithm like the "Miller-Rabin" prime test from a text book, most programmers do not want to "evolve it" or changeevolve it (except they are working as mathematician in that field and try to change or improve the algo). Their typicalstandard goal is to transfer it from the pseudo code of the text book correctly into the programming language of their environment. For this purpose, I would recommend to follow the text book as close as possible, which typically means not to refactor.
However, for someone working as mathematician in that field who is really workingtrying to work on that algorithm and trying to change or improve it, IMHO splitting up this function into smaller, well named ones, or replacing the bunch of "magic numbers" by named constants, can help to make changes to the code easier like for any other kind of code as well.