Eigenvalues of $AB$ and $BA$ where $A$ and $B$ are rectangular matrices

This is a totally different approach, but way more powerful.

I'm going to prove that $\chi_{BA}=(-X)^{n-m}\chi_{AB}$ by elementary means.

Let $r=\operatorname{rank}(A)$

From a well-known theorem, derive that there exists $P,Q$ invertible $m\times m$ and $n \times n$ matrices such that $$A=P\begin{bmatrix}I_r& 0\\ 0 &0\end{bmatrix}Q $$

where $I_r$ denotes the $r\times r$ identity matrix.

By changes of basis, $$B=Q^{-1}\begin{bmatrix}E& F\\ G &H\end{bmatrix}P^{-1}$$

For some submatrices $E,F,G,H$.

Note that $AB=P\begin{bmatrix}E& F\\ 0&0\end{bmatrix}P^{-1}$ and $BA=Q^{-1}\begin{bmatrix}E& 0\\ G&0\end{bmatrix}Q$.

Hence $\chi_{AB}=\det(E-XI_r)(-X)^{m-r}$ and $\chi_{BA}=\det(E-XI_r)(-X)^{n-r}$

Hence $\chi_{BA}=(-X)^{n-m}\chi_{AB}$.

The results in the two other answers are now a simple consequence of the formula.

Also, note that $BA$ will have an eigenvalue of $0$, since it's $n\times n$, but the maximum rank of each of $A$ and $b$ is $m<n$.

the characteristic polynomials of $AB$ and $BA$ are still $\lambda^{m-r}p(\lambda)$ and $\lambda^{n-r}p(\lambda), p(0) \neq 0$ the reason is $tr(AB)^k = tr(BA)^k$ for all $k$. showing the coefficients of the characteristic polynomials are the same.

Eigenvalues are roots of characteristic polynomial. We want to find the connction between characteristic polynomials of AB and BA. Let $\chi_M(x)$ denotes a characteristic polynomial $\chi_M(x) = det(x - M)$

Lets prove the fact: For square matrices $A$ and $B$ holds $det(AB - x) = det(BA - x) \Leftrightarrow \chi_{AB}(x) = \chi_{BA}(x)$.

If $det(A) \neq 0$ then it follows from $det(AB - x) = det(A^{-1}A)det(AB - x) = det(A^{-1})det(AB - x)det(A) = det(BA - x)$.

If $det(A) = 0$ there are finite number of such $s \in \mathbb R$ that $\chi_A(s)=0$ because $\chi_A(s)$ is a finite-degree polynomial. Then there are infinite number of such $s$ that $\chi_A(s) \neq 0$. For all such $s$ we know $\chi_{(A-s)B}(x) = \chi_{B(A-s)}(x)$ as a result of a previous case. For every fixed $x$ we see two finite-degree polynomials ($x$ is fixed, $s$ is variable) $\chi_{(A-s)B}(x)$ and $\chi_{B(A-s)}(x)$ which are equal in infinite number of points. Then we conclude they are equal at every $s$. At $s = 0$ we get the result $\chi_{AB}(x) = \chi_{BA}(x)$ at every $x$.

For square matrices we are done!

Key fact (proof below): If $A$ is $m\times n$, $B$ is $n\times m$ and $n \geq m$ then $\chi_{BA}(x) = \lambda^{n-m}\chi_{AB}(x)$.

Consider $n\times n$ matrices $A' = \left(\dfrac{A}{0}\right)$ and $B' = (B\mid0)$. We just put zero rows and columns to make matrices $n\times n$.

First, $B'A' = BA \Rightarrow x - B'A' = x - BA \Rightarrow \chi_{B'A'}(x) = \chi_{BA}(x)$

Second, $A'$ and $B'$ are square matrices. Then due to the fact above we have $\chi_{B'A'}(x) = \chi_{A'B'}(x)$.

Third, $\chi_{A'B'}(x) = det(x - A'B') = det\begin{pmatrix}x - AB & 0 \\ 0 & \begin{matrix}x & 0 & \ldots & 0 \\ 0 & x & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & x \\\end{matrix}\end{pmatrix} = det(x - AB)x^{n - m} = x^{n-m}\chi_{AB}(x)$

So, we see $\chi_{BA}(x) = \chi_{B'A'}(x) = \chi_{A'B'}(x) = x^{n-m}\chi_{AB}(x)$. Then wee see all eigenvalues of $AB$ are eigenvalues of $BA$ and other $n-m$ eigenvalues are zeros in $BA$.