Note that PlotPoints -> 2 means 2 plot points in each direction, so I'd expect 4 evaluations for your input (8 seconds total).
We can use EvaluationMonitor to see how many times the function is being evaluated:
cnt = 0; Plot3D[a + b, {a, 0, 1}, {b, 0, 1}, PlotPoints -> 2, Mesh -> All, MaxRecursion -> 0, EvaluationMonitor :> (cnt++) ]; cnt
16
Plot3D does post processing to make the plot better looking. One part of this is figuring out surface normals to give the plot a smoother appearance, and this involves evaluating your function. Luckily we can turn this off with NormalsFunction:
cnt = 0; Plot3D[a + b, {a, 0, 1}, {b, 0, 1}, PlotPoints -> 2, Mesh -> All, MaxRecursion -> 0, NormalsFunction -> None, EvaluationMonitor :> (cnt++) ]; cnt
6
I don't know where the extra 2 evaluations are coming from, but let's at least profile a function that takes 2 seconds to evaluate and see if we're at 12 seconds. Perhaps there are more options we can turn off.
Plot3D[Pause[2]; a + b, {a, 0, 1}, {b, 0, 1}, PlotPoints -> 2, Mesh -> All, MaxRecursion -> 0, NormalsFunction -> None ]; // AbsoluteTiming
{12.2104, Null}
If you'd like to have smooth shading, you could manually postprocess by estimating the surface normals from the triangular faces.
An example:
plot = Plot3D[Sin[x + y], {x, 0, 3π}, {y, 0, 3π}, NormalsFunction -> None, Mesh -> None]

(* hack to get the right plot style... *) color = plot[[1, 2, 1, 1, 2]]; Show[DiscretizeGraphics[plot, MeshCellStyle -> {2 -> color}, PlotTheme -> "SmoothShading"], AbsoluteOptions[plot]]

And for comparison, here's the plot with default shading:
Plot3D[Sin[x + y], {x, 0, 3π}, {y, 0, 3π}, Mesh -> None]

ListPlot3D, instead, i.e. evaluate your points, first, and then feed them toListPlot3D. $\endgroup$Compile), or vectorize, or replaced by a more efficiet algorithm... $\endgroup$