Both results indicate the both integrals are numericallyresult in zero.
The integrand is dominated by the part Cosh[(ArcSinh[a] - ArcSinh[b])/2]*Exp[-1/2*(a^2 + b^2)]
Plot3D[ Cosh[(ArcSinh[a] - ArcSinh[b])/2]*Exp[-1/2*(a^2 + b^2)] , {a, -10 , +10 }, {b, -10 , +10 }, PlotPoints -> 100, PlotRange -> All, AxesLabel -> {a, b}] For numerical reasons it is sufficient to decrease the integration range accordingly.
inf=10; f[x_, y_] :=NIntegrate[ Cosh[(ArcSinh[a] - ArcSinh[b])/2]*Exp[-1/2*(a^2 + b^2)]* Cos[x*(a - b) +y*(Sqrt[1 + a^2] -Sqrt[1 + b^2])], {a, -inf, +inf}, {b, -inf, +inf}, Method -> "LocalAdaptive" , IntegrationMonitor :> ((errors = Through[#1@"Error"]) &)]; f[20, 10 ] (* result 1.86704*10^-45*) Total@errors (* error 1.93674*10^-46*) Hope it helps!
