I have the group presentation
$$ G := \langle a,b |a^8=b^8=1,a^{-1}ba=b^{-1},b^{-1}ab=a^{-1} \rangle. $$
Which group is it? Notably, what about its order?
I have the group presentation
$$ G := \langle a,b |a^8=b^8=1,a^{-1}ba=b^{-1},b^{-1}ab=a^{-1} \rangle. $$
Which group is it? Notably, what about its order?
This is the quaternion group, of order $8$. First, note that $a^2=b^{-2}$, as you've noticed. In particular, $a^2$ is centralised by $b$, but $b$ also inverts $a$ by conjugation, so we find $a^2=(a^2)^b=a^{-2}$, and so $a^4=1$. Now, $\langle a\rangle$ is normal in $G$, has order at most $4$, and the quotient has order at most $2$ (since $b^2\in\langle a\rangle$). This shows the group has order at most $8$, but $Q_8$ clearly satisfies the presentation (with $a=i$ and $b=j$, for example).
Note that we never use the first two relations! Indeed, $\langle a,b\mid a^b=a^{-1}, b^a=b^{-1}\rangle$ is already the quaternion group! (This is often useful to remember.)
(I'm using $a^b$ throughout to denote the conjugate of $a$ under $b$, that is $a^b=b^{-1}ab$.)
Consider the permutation group $Q = \langle \alpha = (1234)(5687), \beta = (1538)(2746)\rangle$. It's easy to check that the mapping $a\mapsto\alpha$, $b\mapsto\beta$ defines a homomorphism, since $\alpha$ and $\beta$ satisfy the corresponding relations ($\alpha^8=\beta^8=1, \alpha^\beta=\alpha^{-1}, \beta^\alpha= \beta^{-1}$), and this is a group of order $8$ (in fact, the quaternion group). It follows that $G$ has order at least $8$.
It seems pretty clear from the defining relations that all the elements of the group can be written in the form $a^mb^n$ with $0\leq m,n\leq 7$. But we also have the deduced relation $a^2 = b^{-2}$ and, using this, we can restrict the elements to those of the form $a^mb^n$ with $m\in\{0,1\}$ and $0\leq n\leq 3$, giving a total of eight elements.
(EDIT: As @verret rightly points out in a comment we also need to deduce, as he/she did, that $b^4 = 1$ to provide a complete argument in the paragraph above.)
It follows that the homomorphism is in fact an isomorphism, so this group is isomorphic to the quaternion group of order $8$.