Question in the proof of Hankel's integral representation of the Bessel function of the first kind

Method 1: Integration by Parts and Analytic Continuation Directly

Let $\mathcal{C}$ be a contour with the following properties:

• it is parametrized by a $C^1$ curve $\gamma:(a,b)\to\Bbb{C}$ whose image lies in $\Bbb{C}\setminus(-\infty,0]$,
• $\mathrm{Re}(\gamma(s))$ remains bounded as $s$ varies in $(a,b)$, while $\mathrm{Im}(\gamma(s))\to -\infty$ as $s\to a^+$, and $\mathrm{Im}(\gamma(s))\to +\infty$ as $s\to b^-$,
• for each $p>1$, we have $\int_{\mathcal{C}}\frac{1}{|t|^p}\,|dt|<\infty$.

The straight line from $c-i\infty$ to $c+i\infty$ with $c>0$ satisfies all these conditions (first two obvious, and the third because $c>0$). Put $\Omega= \{(x,\alpha)\in\Bbb{C}^2\,:\, \mathrm{Re}(\alpha)>-1\}$, and define the function $F:\Omega\to\Bbb{C}$ as the improper integral \begin{align} F(x,\alpha):=\int_{\mathcal{C}}\frac{1}{t^{\alpha+1}}\exp\left(t-\frac{x^2}{4t}\right)\,dt. \end{align} At the moment, it doesn’t matter which holomorphic branch of $t^z$ one uses, but for later comparison with the Bessel function, we’ll want the principal branch. I claim that $F$ is well-defined and actually holomorphic in $\Omega$. To see this, write \begin{align} \frac{1}{t^{\alpha+1}}\exp\left(t-\frac{x^2}{4t}\right)&=\frac{d}{dt}\left[\frac{1}{t^{\alpha+1}}\exp\left(t-\frac{x^2}{4t}\right)\right]- e^t\frac{d}{dt}\left(\frac{1}{t^{\alpha+1}}\cdot e^{-x^2/4t}\right)\\ &= \frac{d}{dt}\left[\frac{1}{t^{\alpha+1}}\exp\left(t-\frac{x^2}{4t}\right)\right]+ \left(\alpha+1+\frac{x^2}{4t}\right)\frac{1}{t^{\alpha+2}}\exp\left(t-\frac{x^2}{4t}\right). \end{align}

Fix any compact set $K\subset\Omega$. Then, $|x|$ and $|\alpha|$ remain bounded, and there is an $\alpha_0\in (-1,\infty)$ (depending on $K$) such that for all $(x,\alpha)\in K$, we have $\mathrm{Re}(\alpha)\geq \alpha_0$. Now observe that the exponential remains uniformly bounded as $(x,\alpha)$ vary in $K$ and $t$ varies in $\mathcal{C}$ (since $|x|$ remains bounded, and $|t|\to\infty$ so $\frac{|x|^2}{4t}$ becomes very small for large $|t|$, hence $t$ having bounded real part implies so does $t-\frac{x^2}{4t}$). Keeping this in mind, observe that

• For the first term on the RHS, integrate over a finite portion of the contour, $\mathcal{C}_{a_0,b_0}$, with $[a_0,b_0]\subset (a,b)$, and then take limits. We get only the boundary terms. But since the exponential remains bounded, and $|t|\to \infty$ and $\mathrm{Re}(\alpha)+1>0$, it follows these boundary terms vanish.
• The second term on the RHS is bounded by $\frac{M_{\mathcal{C},K}}{|t|^{\alpha_0+2}}$, for some fixed constant $M_{\mathcal{C},K}>0$ depending only on the contour and $K$. Since $\alpha_0+2>1$, it follows this is (absolutely) integrable over $\mathcal{C}$.

This shows that $F$ is actually well-defined, and we have \begin{align} F(x,\alpha)&= 0 +\int_{\mathcal{C}}\left(\alpha+1+\frac{x^2}{4t}\right)\frac{1}{t^{\alpha+2}}\exp\left(t-\frac{x^2}{4t}\right)\,dt. \end{align} Now, observe that the integrand on the RHS is holomorphic as a function of $(x,\alpha)\in \Omega$, and as we mentioned above, for each compact $K\subset\Omega$, the integrand is bounded by $\frac{M_{\mathcal{C}, K}}{|t|^{\alpha_0+2}}$, which is actually (absolutely (Lebesgue)) integrable over $\mathcal{C}$. Thus by the holomorphic version of differentiation under the integral sign, it follows that $F$ is indeed holomorphic in $\Omega$ (in that link I, for convenience, only discussed holomorphic functions of one variable, but it generalizes verbatim to any number of variables, even if we consider infinite-dimensional complex Banach spaces). A side note: your question is about fixed $x$ and varying $\alpha$, so if you’re uncomfortable with several complex variables, you can simply stick to this case (then, the result in my link above applies directly). I only mentioned holomorphy as a function of two variables as a sort of “FYI” because it immediately followed from our estimates without any actual extra work.

Regardless, what we now have is that $f_{\alpha}(x):=\frac{(x/2)^{\alpha}}{2\pi i}\int_{\mathcal{C}}\frac{1}{t^{\alpha+1}}\exp\left(t-\frac{x^2}{4t}\right)\,dt$ is holomorphic the infinite half-plane $\{\alpha\in\Bbb{C}\,:\, \mathrm{Re}(\alpha)>-1\}$. On the other hand, the Bessel function $J_{\alpha}(x)$ is (for fixed $x$) an entire function of $\alpha$ (just by looking at the series and using the ratio+ Weierstrass $M$-test). Hence, if you manage to find some non-empty open interval $I\subset (-1,\infty)$ such that for all $\alpha\in I$, $J_{\alpha}(x)=f_{\alpha}(x)$, then by uniqueness of analytic continuation it follows that $J_{\alpha}(x)=f_{\alpha}(x)$ for all $\mathrm{Re}(\alpha)>-1$ and all $x\in\Bbb{C}$. Now indeed, once the real part of $\alpha$ is large enough (in your case as long as it is in $(0,\infty)$) you were able to directly apply Fubini-Tonelli. Hence, this completes the proof.

An important remark

For many purposes, the contour $\mathcal{C}$ above is not ideal. In particular, the purely vertical strip $c-i\infty$ to $c+i\infty$ is not good. Part of the reason why we deform contours in the first place is to exploit the rapid decay of the integrand in certain portions of the complex plane. As a result, we really “should” use contours whose real parts tend to $-\infty$. Then, we can exploit the exponential damping; otherwise we have to unnecessarily deal with these oscillatory integrals.

As another example is Airy’s integral. If you naively adopt the definition only along the positive real axis, then we have \begin{align} \mathrm{Ai}(x)&:=\lim_{R\to\infty}\frac{1}{\pi}\int_0^R\cos\left(\frac{t^3}{3}+xt\right)\,dt. \end{align} This is an improper integral, whose convergence is not a-priori obvious. One has to integrate by parts to see convergence, or use something like Dirichlet’s test (something which I don’t really want to use unless I really have to… why deal with oscillatory conditionally-convergent integrals when we can very easily deal with absolutely convergent ones). From this representation, the properties of $\mathrm{Ai}$ are not at all obvious. But, when you analyze the phase function $\frac{t^3}{3}+xt$, you’ll see there are certain sectors where the exponential decays rapidly; it is into these regions that we “should” deform our contour into. This yields for instance \begin{align} \mathrm{Ai}(z)&=\frac{1}{2\pi i}\int_{\mathcal{C}}\exp\left(\frac{t^3}{3}-zt\right)\,dt, \end{align} where $\mathcal{C}$ is a doubly-infinite contour starting from $e^{-i\pi/3}\infty$ and ending at $e^{i\pi/3}\infty$ (cf. eqn 9.5.4). Now, the integrand has exponential decay; this allows us to easily justify things like holomorphy of this integral, differentiation under the integral sign (and hence also allows us to show that this function satisfies an ODE (Airy’s equation); and all boundary terms will obviously vanish, etc).

Method 2: Integrating by parts for each term in series

In other words, we’re essentially continuing from where you left off. Btw, the final series to be bounded is NOT the one you wrote: it seems like you forgot to include the factorials in front (without the factorials the series is of course infinite, since each term is like $\frac{m}{m+ \alpha+1}\sim 1$ as $m\to\infty$).

Anyway, here’s how we can finish off. First, we simplify the integrand somemore with a naive estimate. Observe that $\frac{(1+\omega)^2}{c^2+\omega^2}$ is a quotient of continuous functions which has finite limits as $\omega\to 0^+$ (since $c>0$) and as $\omega\to \infty$; thus the quotient is bounded by some $M^2>0$. As a result, for each $\beta>0$, we have \begin{align} \frac{1}{(c^2+\omega^2)^{\beta/2}}&\leq \frac{M^{\beta}}{(1+\omega)^{\beta}}. \end{align} Keeping in mind the elementary integral $\int_0^{\infty}\frac{1}{(1+\omega)^{\beta}}\,d\omega=\frac{1}{\beta-1}$ (provided $\beta>1$ (or even $\mathrm{Re}(\beta)>1$)), it follows that taking $\beta=m+\alpha+2$, we have \begin{align} &\left|\frac{(-1)^m}{m!}\left(\frac{x}{2}\right)^{2m}(m+\alpha+1)\int_{c-iR}^{c+iR}\frac{e^t}{t^{m+\alpha+2}}\,dt\right| \\ &\leq \frac{1}{m!}\left(\frac{|x|^2}{4}\right)^m(m+\alpha+1)\int_0^{\infty}\frac{2e^cM^{m+\alpha+2}}{(1+\omega)^{m+\alpha+2}}\,d\omega\\ &=\frac{1}{m!}\left(\frac{|x|^2}{4}\right)^m(m+\alpha+1)\cdot\frac{2e^cM^{m+\alpha+2}}{(m+\alpha+2)-1}\\ &=M^{\alpha+2}\cdot \frac{1}{m!}\left(\frac{M|x|^2}{4}\right)^m. \end{align} The RHS is independent of $R$ and summable over $m$ (with sum $M^{\alpha+2}e^{M|x|^2/4}$).