Let $H$ be a hilbert space and let $\hat{A}$ be a linear operator on $H$.
My textbook states that $|\hat{A} \psi\rangle = \hat{A} |\psi\rangle$. My understanding of bra-kets is that $|\psi\rangle$ is a member of $H$ and that $\psi$ alone isn't defined to be anything, so $|\hat{A}\psi\rangle$ isn't defined.
Is $|\hat{A} \psi\rangle = \hat{A} |\psi\rangle$ just a notation or is there something deeper that I am missing?
This should be understood as a mere definition, i.e. a new label for the state you get when you apply the operator A to the ket psi.
I'm not 100% sure about this, but I believe this notation stems from the practice of treating $\psi$ as a wavefunction. In a typical introductory quantum mechanics class, the theory is expressed in terms of continuous functions $\psi(x)$ in the space $\mathbb{R}\to\mathbb{C}$, and operators that act on these functions to produce other functions. So intro-level QM students get used to thinking of $\psi(x)$, not $|\psi\rangle$, as the fundamental object in the theory.
Given this viewpoint, when Dirac notation is introduced, $|\psi\rangle$ looks like nothing more than a convenient notation for $\psi(x)$. In other words, a beginner to quantum mechanics interprets the thing inside the ket as a function, not a label. At this point in the class, the action of an operator on a function is well defined, whereas the action of an operator on the ket (an abstract object) is not. I would guess that your textbook is trying to define the action of an operator on a ket by relating it to the action of the operator on the function, something which can easily be understood by a student who is used to thinking in terms of wavefunctions.
To frame it more precisely: let $F$ be the (Hilbert) space of all possible wavefunctions $\psi:\mathbb{R}\to\mathbb{C}$, and $K$ be the (Hilbert) space of all possible kets $|\psi\rangle$. Then let $\mathcal{O}_F$ be the space of all operators acting on $F$, and let $\mathcal{O}_K$ be the space of all operators acting on $K$. At this point in the book, I'm guessing $F$ and $\mathcal{O}_F$ have already been discussed, $K$ is relatively new, and $\mathcal{O}_K$ has literally just been introduced. It's known from the given definition of the ket that every wavefunction corresponds to a ket, i.e. that there is a mapping $F\to K$ where the image of $\psi$ is written $|\psi\rangle$. But in order to work with kets the same way you work with wavefunctions, you also need a way to obtain the ket-operator corresponding to any given function-operator, i.e. a mapping $\mathcal{O}_F\to\mathcal{O}_K$. The book defines that mapping as follows: any given operator $\hat{A}_F\in\mathcal{O}_F$ maps to the operator $\hat{A}_K\in\mathcal{O}_K$ which satisfies $\hat{A}_K|\psi\rangle \equiv |\phi\rangle$, where $\phi = \hat{A}_F\psi$.
