Given $s, t \in \mathbb{Z}$, define $$ h(s,t) := a^{s} b^ta^{-s} = \begin{pmatrix} 1 & 2^s t \\ 0 & 1\end{pmatrix} \in H.$$ Observe that $h(0,0) = 1$, and that for all $s,t,u,v \in \mathbb{Z}$, we have $$ (a^sb^t)(a^ub^v) = h(s',t')a^{s+u}, \quad \text{where} \quad s' = \min\{s,s+u\}, \ t ' = 2^{s-s'}t + 2^{s+u-s'}v.$$ Since each element of $G$ is a product of matrices of the form $a^mb^n$ with $m,n \in \mathbb{Z}$, the above relations imply that each matrix $g \in G$ can be represented in the form $$g = h(s,t)a^r \quad \text{with} \quad s,t,r \in \mathbb{Z}.$$ Note that for $g \notin \langle a\rangle$, this factorisation can be made unique if we require that $t$ be an odd integer. A matrix $g \in G$ belongs to $H$ if and only if $\det(g) = 2^r = 1$, or in other words, if $g = h(s,t)$ for some $s, t\in\mathbb{Z}$.
For $g = h(s,t) \in H$ such that $g \neq 1$, define $\operatorname{ord}(g)$ to be the unique $s' \in \mathbb{Z}$ such that $g = h(s',t')$ with an odd integer $t'$; for $g = 1$ put $\operatorname{ord}(g) = \infty$. Thus, we have $\operatorname{ord} h(s,t) \geq s$ for all $s,t\in\mathbb{Z}$. Now, suppose that we are given $k \geq 1$ pairs of integers $(s_1,t_1),\ldots,(s_k,t_k)$ with $t_j \equiv 1 \!\pmod 2$ for $1 \leq j \leq k$, and let $s = \min\{s_1,\ldots,s_k\}$. For any $k$-tuple $(m_1,\ldots,m_k) \in \mathbb{Z}^k$, we have $$ h(s_1,t_1)^{m_1}\cdots h(s_k,t_k)^{m_k} = h(s, \,2^{s_1-s}m_1t_1 + \cdots + 2^{s_k-s}m_kt_k).$$ Therefore, $\operatorname{ord}\left(h(s_1,t_1)^{m_1}\cdots h(s_k,t_k)^{m_k}\right) \geq s$ for all $(m_1,\ldots,m_k) \in \mathbb{Z}^k$. On the other hand, $\operatorname{ord} h(s-1,1) = s-1$, and therefore, $h(s-1,1) \in H$ does not belong to the group generated by $h(s_1,t_1), \ldots, h(s_k,t_k)$. Hence, $H$ is not finitely generated.