I have a question about the representation theory of semisimple Lie groups, motivated by concepts from quantum mechanics.
Let $G$ be a semisimple Lie group and let $(\pi, H)$ be a unitary representation of $G$ on a Hilbert space $H$.
Consider the representation of $G$ on the space of endomorphisms of $H$, which I'll denote $\text{End}(H)$, given by the action: $$ g \cdot A := \pi(g) A \pi(g^{-1}) \quad \text{for } g \in G, A \in \text{End(H)} $$
Crucially, I want to consider $\text{End}(H)$ not just as the space of bounded operators $B(H)$, but as the larger vector space of all linear operators defined on a common dense domain, such as the space of smooth vectors $H^\infty$.
Motivation from Quantum Mechanics:
My motivation for this question comes from quantum mechanics. In this context, physical observables (like momentum, angular momentum, etc.) are represented by self-adjoint operators on the Hilbert space $H$. A set of such observables spans a finite-dimensional vector space that is invariant under the action of a symmetry group $G$. The transformation rule for these operators is precisely the action $A \mapsto \pi(g) A \pi(g^{-1})$. Since many fundamental observables (like position and momentum) are represented by unbounded operators, I am interested in the general structure of such invariant subspaces, especially those composed of unbounded operators.
My Question:
I am looking for references that systematically study the classification of all finite-dimensional, irreducible, invariant subspaces within this very large space $\text{End}(H)$.
What I currently understand:
The Hilbert-Schmidt Case: I am aware that when this action is restricted to the space of Hilbert-Schmidt operators $HS(H)$, the representation is unitary and equivalent to the tensor product representation $\pi \otimes \bar{\pi}$. The study of its decomposition into irreducibles is a central and well-studied topic. The finite-dimensional subrepresentations here are realized by "well-behaved" operators.
An Example Beyond Hilbert-Schmidt Operators: I know of at least one class of examples of a finite-dimensional invariant subspace in $\text{End}(H)$ whose elements are typically unbounded. Let $\mathfrak{g}$ be the Lie algebra of $G$. The differentiated representation $d\pi$ maps $\mathfrak{g}$ to a space of (typically unbounded) skew-adjoint operators. The image $V = d\pi(\mathfrak{g})$ is a finite-dimensional vector space. This space $V$ is invariant under the action because of the identity: $$ \pi(g) (d\pi(X)) \pi(g^{-1}) = d\pi(\text{Ad}(g)X) $$ This shows that the representation on $V$ is equivalent to the adjoint representation of $G$ on its Lie algebra $\mathfrak{g}$. (Physically, the angular momentum operators form such a space for the rotation group $SO(3)$).
This example demonstrates that there are important finite-dimensional subrepresentations that exist outside of the standard $HS(H)$ framework.
So, my question is: Has the problem of classifying all finite-dimensional invariant subspaces of $\text{End}(H)$ (in this broad sense) been systematically studied? Are there known examples beyond the ones realized in $HS(H)$ and the adjoint representation living in $d\pi(\mathfrak{g})$?
I would be very grateful for any references (articles, books, lecture notes) that discuss this problem in this broader setting.
Thank you.
