The reason it never reaches the form you want despite of been minimal by [`LeafCount`](https://reference.wolfram.com/language/ref/LeafCount.html) is that that form is never tried. Look at the `attempts` with a fresh kernel (as [`Simplify`](https://reference.wolfram.com/language/ref/Simplify.html) is cashed)
{sol, {attempts}} = Reap@Simplify[
1 - (2 b^2)/a^2 + b^4/a^4
, ComplexityFunction -> ((Sow[#]; LeafCount[#]) &)
];
[![enter image description here][1]][1]
<hr>
You can force that form by
CompleteSquare[f_, x_] := Module[
{a, b, c},
{c, b, a} = CoefficientList[f, x];
Assuming[
Sqrt[a] > 0,
(FullSimplify[Sqrt[a] x] +
FullSimplify[b/(2 Sqrt[a])])^2 + (FullSimplify[(a c - b^2/4)])
]]
CompleteSquare[1 - (2 b^2)/a^2 + b^4/a^4, b^2]
(* (-1 + b^2/a^2)^2 *)
Or using [`Simplify`](https://reference.wolfram.com/language/ref/Simplify.html)
Simplify[
1 - (2 b^2)/a^2 + b^4/a^4
, TransformationFunctions -> {Automatic, CompleteSquare[#, b^2] &}
, ComplexityFunction -> LeafCount
]
(-1 + b^2/a^2)^2
[![enter image description here][2]][2]
Based on the answer by Ulrich Neumann, also
transf[expr_] := Module[
{vars, tvar},
vars = Variables[expr];
ReplaceAll[
Simplify@ReplaceAll[expr, vars[[1]] -> vars[[2]]/tvar]
, tvar -> Divide @@ vars[[{2, 1}]]
]
]
Simplify[
1 - (2 b^2)/a^2 + b^4/a^4
, TransformationFunctions -> {Automatic, transf}
, ComplexityFunction -> LeafCount
]
(* (-1 + b^2/a^2)^2 *)
[1]: https://i.sstatic.net/XMaFq.png
[2]: https://i.sstatic.net/kKSVC.png