Is there any body of literature on representing groups on pure sets, instead of necessarily vector spaces? It seems like a lot of the theory holds and has meaning. For example, if the action of a group $G$ is defined on a set $S$, then the set $S$ is invariant under $G$, and so can subsets of $S$ be as well. If $A \subseteq S$ is invariant under $G$, and $A$ has an invariant subset $B \subseteq A$, then $C = B^{c}$ relative to A is also invariant via the invertibility of group actions (if $C$ weren't invariant, $\exists g\in G, c \in C\ \text{s.t.} g*c \in B$, but then $g^{-1}*(g*c)\notin B$ contradicting the invariance of $B$). I can't say I see reason for such literature to exist given the power of representation theory on vector spaces, it just feels like several of the theorems have counterparts in set-only land.
1 Answer
What you are describing is called a group action. It is an old and extremely well-studied theory, and any algebra textbook will devote a fair amount of space on them.
Incidentally, this theory is included in linear representation theory: if $G$ acts on a set $X$, this induces a representation of $G$ on the free vector space on $X$ (basically, $V$ has a basis $(e_x)_{x\in X}$ indexed by $X$), simply by permuting the basis vectors according to the action of $G$ on $X$. This is called a permutation representation. An invariant subset will induce a subrepresentation, which is also a permutation representation, but not all subrepresentations are of this form. The "irreducible components" of a group action are the so-called orbits, but as a representation every such orbit will decompose further into irreducible subrepresentations.
You can also include both theories in a larger theory of groups acting on structures: group actions correspond to acting on sets, and group representations to acting on vector spaces. But you can also make a group act on a topological space, a graph, a ring, etc.
