Integrate[r*Exp[a*x^2 + b*x*y + c*y^2] /. {x -> r*Cos[\[Phi]]r*Cos[ϕ],y -> r*Sin[\[Phi]]r*Sin[ϕ]}, {r, 0, R}]
(-1 + E^(R^2 (a Cos[\[Phi]]^2Cos[ϕ]^2 + b Cos[\[Phi]]Cos[ϕ] Sin[\[Phi]]Sin[ϕ] + c Sin[\[Phi]]^2Sin[ϕ]^2)))/(a + c + (a - c) Cos[2 \[Phi]]ϕ] + b Sin[2 \[Phi]]ϕ])
It is clear the finite limit of the function defined by the above expression as R->Infinity exists only if ForAll[\[Phi]ForAll[ϕ, \[Phi]ϕ > -Pi && \[Phi]ϕ <= Pi, a Cos[\[Phi]]^2Cos[ϕ]^2 + b Cos[\[Phi]]Cos[ϕ] Sin[\[Phi]]Sin[ϕ] + c Sin[\[Phi]]^2Sin[ϕ]^2 < 0] is valid as the result of
Integrate[r*Exp[a*x^2 + b*x*y + c*y^2] /. {x -> r*Cos[\[Phi]]r*Cos[ϕ], y -> r*Sin[\[Phi]]r*Sin[ϕ]}, {r, 0,Infinity},Assumptions -> {a, b, c} \[Element]∈ Reals && \[Phi]ϕ > -Pi && \[Phi]ϕ <= Pi]
ConditionalExpression[-(1/( 2 (a Cos[\[Phi]]^2Cos[ϕ]^2 + b Cos[\[Phi]]Cos[ϕ] Sin[\[Phi]]Sin[ϕ] + c Sin[\[Phi]]^2Sin[ϕ]^2))), a Cos[\[Phi]]^2Cos[ϕ]^2 + b Cos[\[Phi]]Cos[ϕ] Sin[\[Phi]]Sin[ϕ] + c Sin[\[Phi]]^2Sin[ϕ]^2 < 0]
Resolve[ForAll[\[Phi]Resolve[ForAll[ϕ, \[Phi]ϕ > -Pi && \[Phi]ϕ <= Pi, a Cos[\[Phi]]^2Cos[ϕ]^2 + b Cos[\[Phi]]Cos[ϕ] Sin[\[Phi]]Sin[ϕ] + c Sin[\[Phi]]^2Sin[ϕ]^2 < 0], Reals] Integrate[(-1)/(a + c + (a - c) Cos[2 \[Phi]]ϕ] + b Sin[2 \[Phi]]ϕ]), {\[Phi]ϕ, -Pi, Pi}, Assumptions -> c < 0 && a < b^2/(4 c) && a < 0 && b \[Element]∈ Reals] , thinking a few hours. In fact, this is $$\frac{2 \pi }{\sqrt{4 a c-b^2}} $$ as Simplify under the conditions c < 0 && a < b^2/(4 c) && a < 0 && b \[Element]∈ Reals] shows.