Why does causality imply that the system function is analytic?

New answer:

I provide a new answer because I believe that this is a clearer and more direct way of explaining the relation between causality of the impulse response and analyticity of the corresponding transfer function. The old answer is not wrong but it relies on the Hilbert transform, which in hindsight appears to be a rather indirect approach. For the time being I will not delete the old answer, just in case anybody is interested (see below).

I'll show that if a given transfer function $H(s)$ is analytic in the right half-plane, its corresponding impulse response $h(t)=\mathscr{L}^{-1}\{H(s)\}$ must be causal. The only additional assumption is that $\lim_{s\to\infty}H(s)=0$ for $\textrm{Re}\{s\}>0$ (i.e., in the right half-plane).

In order to prove that $h(t)=0$ for $t<0$ if $H(s)$ is analytic in the right half-plane, we consider the integral of $H(s)e^{st}$ over the closed contour $C$ which consists of the line segment on the imaginary axis from $s=-j\Omega$ to $s=j\Omega$ and of the semicircle in the right half-plane $s=\Omega e^{j\theta}$, with $\theta$ from $\pi/2$ to $-\pi/2$. Since $H(s)e^{st}$ is analytic inside $C$, we have from Cauchy's theorem

$$\oint_CH(s)e^{st}ds=0\tag{1}$$

The integral $(1)$ can be split into the integral along the line segment on the imaginary axis and the integral along the semicircle $\Gamma$:

$$\oint_CH(s)e^{st}ds=\int_{-\Omega}^{\Omega}H(j\omega)e^{j\omega t}jd\omega+\int_{\Gamma}H(s)e^{st}ds\tag{2}$$

For $t<0$ and $\textrm{Re}\{s\}>0$ (i.e., in the right half-plane), the real part of the exponent of $e^{st}$ is negative, and since $H(s)$ tends to zero as $s\to\infty$ in the right-half plane, if follows from Jordan's lemma that the contribution of the integral along the semicircle $\Gamma$ tends to zero as $\Omega\to\infty$. With $(1)$ and $(2)$ we obtain

\begin{align} \lim_{\Omega\to\infty}\oint_CH(s)e^{st}ds &= \lim_{\Omega\to\infty}\left[\int_{-\Omega}^{\Omega}H(j\omega)e^{j\omega t}jd\omega+\int_{\Gamma}H(s)e^{st}ds\right] \\ &= j\int_{-\infty}^{\infty}H(j\omega)e^{j\omega t}d\omega \\ &= 2\pi j\, h(t) =0,\qquad t<0 \end{align}

This shows that under the given assumptions (analyticity of $H(s)$ in the right half-plane and $H(s)$ tending to zero as $s\to\infty$ in the right half-plane) the impulse response $h(t)$ is causal.

Old answer:

A causal impulse response $h(t)$ satisfies

$$h(t)=h(t)u(t)\tag{1}$$

where $u(t)$ is the unit step function. With

$$\mathscr{F}\{u(t)\}=\pi\delta(t)+\frac{1}{j\omega}\tag{2}$$

the Fourier transform of Eq. $(1)$ is

\begin{align} H(j\omega)&=\frac{1}{2\pi}H(j\omega)\star\left(\pi\delta(t)+\frac{1}{j\omega}\right)\\ &=\frac{1}{2}H(j\omega)-\frac{j}{2}H(j\omega)\star \frac{1}{\pi\omega}\tag{3} \end{align}

where $H(j\omega)$ is the Fourier transform of $h(t)$ (i.e., the frequency response) and $\star$ denotes convolution. Noting that convolution with $1/(\pi\omega)$ corresponds to the Hilbert transform, we can write Eq. $(3)$ as

$$H(j\omega)=-j\mathscr{H}\{H(j\omega)\}\tag{4}$$

with $\mathscr{H}\{\cdot\}$ denoting the Hilbert transform. Splitting $(4)$ into real and imaginary parts, we arrive at

\begin{align} H_R(\omega) & = \mathscr{H}\{H_I(\omega)\}\\ H_I(\omega) & = -\mathscr{H}\{H_R(\omega)\} \end{align}

where $H_R(\omega)$ and $H_I(\omega)$ are the real and imaginary parts of $H(j\omega)$, respectively.

Since the real and imaginary parts of $H(j\omega)$ are related by the Hilbert transform, the function $H(j\omega)$ can be analytically extended from the imaginary axis to the right half-plane, which results in the transfer function $H(s)$ being analytic for $\textrm{Re}\{s\}>0$, assuming that $H(s)$ decays sufficiently fast for $|s|\to\infty$ and $\textrm{Re}\{s\}>0$. For more mathematical details see ref 1 and especially ref 2, and the second part of this answer.