Take the $C_{3v}$ point group for example:
Rot[θ_] := RotationMatrix[θ] E0 = IdentityMatrix[2]; C1 = Rot[2π/3]; C2 = C1.C1; σ1 = ReflectionMatrix[Rot[2π/3].{1, 0}]; σ2 = C1.σ1.Inverse[C1]; σ3 = C2.σ1.Inverse[C2]; C3v = {E0, C1, C2, σ1, σ2, σ3}; So $C_{3v}$ has the following 6 elements: $ \left\{ \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ \end{pmatrix}, \begin{pmatrix} -\frac{1}{2} & -\frac{\sqrt{3}}{2} \\ \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} -\frac{1}{2} & \frac{\sqrt{3}}{2} \\ -\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} \frac{1}{2} & \frac{\sqrt{3}}{2} \\ \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} \frac{1}{2} & -\frac{\sqrt{3}}{2} \\ -\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} -1 & 0 \\ 0 & 1 \\ \end{pmatrix} \right\} $
For given basis functions $\{f_n(\mathbf{r})\mid n=1\cdots N\}$, there is a representation $\{D(g)\mid g\in C_{3v}\}$, in which the $N\times N$ matrix $D(g)$ is defined as $$ f_i(g^{-1}\mathbf{r})=\sum_{j=1}^N f_j(\mathbf{r})D_{ji}(g)$$ where $\mathbf{r}=${x,y} .
If the basis functions are just monomials, e.g. $$f_1(\mathbf{r})=x^2,~~f_2(\mathbf{r})=y^2,~~f_3(\mathbf{r})=xy$$ I can compute the representation matrices using the Coefficient function
f1[{x_, y_}] := x^2 f2[{x_, y_}] := y^2 f3[{x_, y_}] := x y basis = {f1, f2, f3}; Rep[g_] := Block[{i, j, ig, x, y, r}, ig = Inverse[g]; r = {x, y}; Table[Coefficient[basis[[j]][ig.r], basis[[i]][r]], {i, 3}, {j, 3}] ] repC3v = Rep /@ C3v; MatrixForm /@ repC3v This gives the following results: $$ \left\{\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & \frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & -\frac{\sqrt{3}}{4} \\ -\frac{\sqrt{3}}{2} & \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & -\frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & \frac{\sqrt{3}}{4} \\ \frac{\sqrt{3}}{2} & -\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & \frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & -\frac{\sqrt{3}}{4} \\ \frac{\sqrt{3}}{2} & -\frac{\sqrt{3}}{2} & \frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & -\frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & \frac{\sqrt{3}}{4} \\ -\frac{\sqrt{3}}{2} & \frac{\sqrt{3}}{2} & \frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1 \\ \end{pmatrix}\right\} $$
However, if the basis functions are general functions, not monomials, how can I get the coefficient $a_i$ of function $f_i(\mathbf{r})$ in the transformed function $f_k(g^{-1}\mathbf{r})=\sum_i a_i f_i(\mathbf{r})$?
For example, if
F1[{x_, y_}] := (x + I y)^2 F2[{x_, y_}] := (x - I y)^2 F3[{x_, y_}] := (x + y)^2 How can I get the representation matrices under this basis?
NOTE: I mean a general method, not just a transformation from basis f1~f3 to F1~F3.
TeXForm[]? $\endgroup$