The key words are "presentation" and "relator"; relators describe loops in the Cayley graph, and are not just about orders of elements. Group presentations are systems of generators and relators. Another key word is "free group"; every group is isomorphic to a quotient of a free group, and presentations formally define quotients of a free group.
The first presentations usually encountered are dihedral groups: $$ D_{2n}\cong\langle a, b\mid b^{-1}aba, a^n, b^2\rangle $$ Here, $\{a, b\}$ is my generating set, and $\{b^{-1}aba, a^n, b^2\}$ is my set of relators. A relator $W$ is a word over the generators and their inverses, and has the property that $W=1$ in the group. Note that the relator "$b^{-1}aba$" is not describing the order of an element, but instead says that $b^{-1}ab=a^{-1}$, i.e. that conjugation by the element $b$ inverts the element $a$. Another example of a group presentation would be $$ \mathbb{Z}_n\times\mathbb{Z}_2\cong\langle a, b\mid b^{-1}aba^{-1}, a^n, b^2\rangle $$ This is identical to the above presentation, except the relator "$b^{-1}aba$" has now been changed to "$b^{-1}aba^{-1}$". This is equivalent to $ab=ba$, and means that the elements $a$ and $b$ commute.
I wrote an answer here that puts presentations into the context of group extensions.
Presentations look very attractive, but they are difficult to work with. For example, it is in general undecidable (i.e. there is no general algorithm which will determine) if a given presentation defines a finite group. The above link to presentations in the context of group extensions discusses a similar problem: if a presentation looks like a group extension, then when can we conclude that it is indeed a group extension of the given groups? Furthermore, the "$\cong$" symbols I wrote above are correct, but hard to justify without further theory.
If you are interested in this topic, a good first step would be to read Magnus, Karrass and Solitar's book Combinatorial Group Theory, which has excellent exercises, or maybe Rotman's book, which is more friendly and modern, although I have not read it properly so can only half-recommend it.