Return to Answer

The reason it never reaches the form you want despite of being minimal by LeafCount is that that form is never tried, so its LeafCount is never measured nor compared. Look at the attempts with a fresh kernel (as Simplify is cashed).

We use TransformationFunctions to tell Simplify to explire the form you want, and ComplexityFunction to define how these alternatives are scored.

Or based on the answer by Ulrich Neumann, also

The undocumented PolynomialForm allows placing the terms in the desired order with the option TraditionalOrder->True.

The reason it never reaches the form you want despite of being minimal by LeafCount is that that form is never tried, so its LeafCount is never measured nor compared. Look at the attempts with a fresh kernel (as Simplify is cashed).

We use TransformationFunctions to tell Simplify to explore the form you want, and ComplexityFunction to define how these alternatives are scored.

Or based on the answer by Ulrich Neumann, also

The undocumented PolynomialForm allows placing the terms in the desired order with the option TraditionalOrder->True.

or

Format[bda] = DisplayForm@FractionBox["b", "a"]; PolynomialForm[ (-1 + b^2/a^2)^2 /. b -> bda a , TraditionalOrder -> True ] 

Explanation of the problem

The reason it never reaches the form you want despite of been minimal by LeafCount is that that form is never tried, so its LeafCount is never measured nor compared. Look at the attempts with a fresh kernel (as Simplify is cashed).

So the solution is to find a way for Simplify to explore the form you want using TransformationFunctions and score it as optimal as per your definition ComplexityFunction -> LeafCount.

{sol, {attempts}} = Reap@Simplify[ 1 - (2 b^2)/a^2 + b^4/a^4 , ComplexityFunction -> ((Sow[#]; LeafCount[#]) &) ]; 

Solution requested: Simplify

We use TransformationFunctions to tell Simplify to explire the form you want, and ComplexityFunction to define how these alternatives are scored.

One way is to use the CompleteSquare described below

Simplify[ 1 - (2 b^2)/a^2 + b^4/a^4 , TransformationFunctions -> {Automatic, CompleteSquare[#, b^2] &} , ComplexityFunction -> LeafCount ] (-1 + b^2/a^2)^2 

Or based on the answer by Ulrich Neumann, also

transf[expr_] := Module[ {vars, tvar}, vars = Variables[expr]; ReplaceAll[ Simplify@ReplaceAll[expr, vars[[1]] -> vars[[2]]/tvar] , tvar -> Divide @@ vars[[{2, 1}]] ] ] Simplify[ 1 - (2 b^2)/a^2 + b^4/a^4 , TransformationFunctions -> {Automatic, transf} , ComplexityFunction -> LeafCount ] (* (-1 + b^2/a^2)^2 *) 

PolynomialForm

The undocumented PolynomialForm allows placing the terms in the desired order with the option TraditionalOrder->True.

PolynomialForm[(-1+b^2/a^2)^2,TraditionalOrder->True] (* (b^2/a^2-1)^2 *) 

Alternative Solution: CompleteSquare

Other people have suggested way to complete the square. You can force that form by

CompleteSquare[f_, x_] := Module[ {a, b, c}, {c, b, a} = CoefficientList[f, x]; Assuming[ Sqrt[a] > 0, (FullSimplify[Sqrt[a] x] + FullSimplify[b/(2 Sqrt[a])])^2 + (FullSimplify[(a c - b^2/4)]) ]] CompleteSquare[1 - (2 b^2)/a^2 + b^4/a^4, b^2] (* (-1 + b^2/a^2)^2 *) 

Explanation of the problem

The reason it never reaches the form you want despite of being minimal by LeafCount is that that form is never tried, so its LeafCount is never measured nor compared. Look at the attempts with a fresh kernel (as Simplify is cashed).

So the solution is to find a way for Simplify to explore the form you want using TransformationFunctions and score it as optimal as per your definition ComplexityFunction -> LeafCount.

{sol, {attempts}} = Reap@Simplify[ 1 - (2 b^2)/a^2 + b^4/a^4 , ComplexityFunction -> ((Sow[#]; LeafCount[#]) &) ]; 

Solution requested: Simplify

We use TransformationFunctions to tell Simplify to explire the form you want, and ComplexityFunction to define how these alternatives are scored.

One way is to use the CompleteSquare described below

Simplify[ 1 - (2 b^2)/a^2 + b^4/a^4 , TransformationFunctions -> {Automatic, CompleteSquare[#, b^2] &} , ComplexityFunction -> LeafCount ] (-1 + b^2/a^2)^2 

Or based on the answer by Ulrich Neumann, also

transf[expr_] := Module[ {vars, tvar}, vars = Variables[expr]; ReplaceAll[ Simplify@ReplaceAll[expr, vars[[1]] -> vars[[2]]/tvar] , tvar -> Divide @@ vars[[{2, 1}]] ] ] Simplify[ 1 - (2 b^2)/a^2 + b^4/a^4 , TransformationFunctions -> {Automatic, transf} , ComplexityFunction -> LeafCount ] (* (-1 + b^2/a^2)^2 *) 

PolynomialForm

The undocumented PolynomialForm allows placing the terms in the desired order with the option TraditionalOrder->True.

PolynomialForm[(-1+b^2/a^2)^2,TraditionalOrder->True] (* (b^2/a^2-1)^2 *) 

Alternative Solution: CompleteSquare

Other people have suggested way to complete the square. You can force that form by

CompleteSquare[f_, x_] := Module[ {a, b, c}, {c, b, a} = CoefficientList[f, x]; Assuming[ Sqrt[a] > 0, (FullSimplify[Sqrt[a] x] + FullSimplify[b/(2 Sqrt[a])])^2 + (FullSimplify[(a c - b^2/4)]) ]] CompleteSquare[1 - (2 b^2)/a^2 + b^4/a^4, b^2] (* (-1 + b^2/a^2)^2 *)