Seemingly, you need only the second derivative of the expression for $\delta = 0$ and $L \to \infty$.
First, I replace L by λ^-1:
A = (gp0 + 1/L^2 gf0) + δ (gp1 + 1/L^2 gf1) + δ^2 (gp2 + 1/L^2 gf2) /. L -> λ^-1; // AbsoluteTiming det = Sqrt[Det[A]]; // AbsoluteTiming {0.000957, Null}
{0.392288, Null}
Now, let's take the derivative with respect to δ twice and let us substitute δ -> 0 and λ -> 0:
a2 = D[det, δ, δ]; // AbsoluteTiming a2at0 = a2 /. δ -> 0; // AbsoluteTiming a2at00 = a2at0 /. λ -> 0; // AbsoluteTiming {2.00551, Null} {13.2472, Null} {0.020978, Null}