Evaluating $\int_{-\infty}^\infty \frac{pe^{ipr}}{\sqrt{p^2 + m^2}} dp$ using complex contours

It's a simple application of Cauchy's integral theorem. Consider the complex contour integral $$\oint\limits_C dz \frac{z\, e^{irz}}{\sqrt{z^2+m^2}}, \qquad r\gt 0,\tag{1} \label{1}$$ where the closed curve $C$ (taken counterclockwise) consists of the following pieces: $$\begin{align} \gamma_1&: \, p\to z(p)=p, \, -R\le p \le R, \\ \gamma_2&: \, \varphi \to z(\varphi) = R e^{i \varphi}\!,\, 0\lt \varphi \le \pi/2-\epsilon, \; (\epsilon >0), \\ \gamma_3&: \, \rho \to z(\rho)= i \rho+\epsilon , \, \text{ from} \;\rho=R \; \text{to} \; \rho=m,\\ \gamma_4&: \, s \to z(s)= im -s+\epsilon, \; 0\le s \le 2 \epsilon, \\ \gamma_5&: \, \rho \to z(\rho)= i \rho -\epsilon, \; m \le \rho \le R,\\ \gamma_6&: \varphi \to z(\varphi)=R e^{i\varphi}, \; \pi/2+\epsilon \le \varphi \lt \pi. \end{align} \tag{2} \label{2}$$ The branch cut of $w \to \sqrt{w}$ is chosen along the negative real axis, such that $\lim\limits_{\epsilon \downarrow 0} \sqrt{-|w|\pm i \epsilon}= \pm i |w|$.

As $C$ does not enclose any poles of the integrand, the integral \eqref{1} vanishes. In the limit $R\to \infty$, the contributions from $\gamma_2$ and $\gamma _6$ vanish because of the exponential damping factor $e^{-R r \sin{\varphi}}$ for $r \sin{\varphi} >0$. Taking finally also the limit $\epsilon \downarrow 0$, we are left with $$\underbrace{\int\limits_{-\infty}^{+\infty}\!\! dp \frac{p\, e^{irp}}{\sqrt{p^2+m^2}}}_{\text{from} \; \gamma_1} \;\underbrace{-i \int\limits_m^\infty \! \! d \rho \frac{\rho e^{-r \rho}}{\sqrt{\rho^2-m^2}}}_{\text{from} \; \gamma_3} \; \underbrace{-i\int\limits_m^\infty \! \! d\rho \frac{\rho e^{-r \rho}}{\sqrt{\rho^2-m^2}}}_{\text{from} \; \gamma_5}=0, \tag{3} \label{3} $$ giving the desired result $$\int\limits_{-\infty}^{+\infty} \! \! dp \frac{p \, e^{irp}}{\sqrt{p^2+m^2}} = 2i \int\limits_m^\infty \! \! d\rho \frac{\rho \, e^{-r \rho}}{\sqrt{\rho^2-m^2}}. \tag{4} \label{4}$$ (Note that a factor $i$ is missing in your last formula.)

Peskin and Schroeder use \eqref{4} to demonstrate that naive "relativistic" one-particle quantum mechanics, obtained by simply repacing $H=\vec{p}^2 /2m$ by $H=(\vec{p}^2+m^2)^{1/2}$, is in conflict with the requirement of causality. At the same time, they show how relativistic quantum field theory (being the correct description) avoids this problem.