I'm not exactly sure what you mean, but I'll try my best. $\newcommand{\ket}[1]{|#1\rangle}\newcommand{\bra}[1]{\langle#1|}$The way I would do this is by first noting that $$\ket{+} = \frac{1}{\sqrt{2}}(\ket{0} + \ket{1})$$ $$\ket{-} = \frac{1}{\sqrt{2}}(\ket{0} - \ket{1})$$ Basically, we can just substitute these identities into the expression: $$\begin{split} \frac{3}{8}\ket{+}\bra{+} + \frac{5}{8}\ket{-}\bra{-} &= \frac{3}{8}\cdot\frac{1}{2} (\ket{0} + \ket{1})(\bra{1} + \bra{0}) + \frac{5}{8}\cdot\frac{1}{2}(\ket{0} - \ket{1})(-\bra{1} + \bra{0})\\ &= \frac{3}{16}(\ket{0}\bra{0} + \ket{0}\bra{1} + \ket{1}\bra{0} + \ket{1}\bra{1}) \\ & \quad\qquad + \frac{5}{16}(\ket{0}\bra{0} - \ket{1}\bra{0} - \ket{0}\bra{1} + \ket{1}\bra{1}) \\ &= \frac{1}{2}\ket{0}\bra{0} - \frac{1}{8}\ket{0}\bra{1} - \frac{1}{8}\ket{1}\bra{0} + \frac{1}{2}\ket{1}\bra{1} \end{split}$$ which is $P$ in the $\{\ket{0}, \ket{1}\}$ basis.
To maybe clarify why what you were doing was wrong: first of all, in case you don't know, if we wanted to apply some gate $O$ to a density matrix like $P$, the new expression would be $$UPU^\dagger$$ This is because density matrices can be seen as a weighted sum of pure state vectors. To apply a gate to a statevector $\ket{\psi}$, we just do: $$\ket{\psi} \mapsto U\ket{\psi}$$ Then, the density matrix for $U\ket{\psi}$ would be $$U\ket{\psi}(U\ket{\psi})^\dagger = U\ket{\psi}\bra{\psi}U^\dagger$$ Then, since density matrices are just a sum of statevectors, we can apply $U$ like this to each statevector outer product in the sum, and then factor each $U$ on the left and on the right to get $UPU^\dagger$.
So, applying that gate you were using (which is the Hadamard gate, and is commonly denoted $H$), we get $$\begin{split} HPH &= \frac{3}{8}H\ket{+}\bra{+}H + \frac{5}{8}H\ket{-}\bra{-}H \\ &= \frac{3}{8}\ket{0}\bra{0} + \frac{5}{8}\ket{1}\bra{1} \end{split}$$ which is a different answer than earlier. We can still do what we did earlier where we substituted in expressions for $\ket{+}$ and $\ket{-}$, but now for $\ket{0}$ and $\ket{1}$. Doing that, we get that this is equal to $$\frac{1}{2}\ket{+}\bra{+} - \frac{1}{8}\ket{+}\bra{-} - \frac{1}{8}\ket{-}\bra{+} + \frac{1}{2}\ket{-}\bra{-}$$ which is parallel to earlier. To be honest, I'm pretty hazy on how change of basis works, but what I perceive to be the reason for this is that $H$ maps a vector in the $\{\ket{+}, \ket{-}\}$ basis to the coefficients it would have in the $\{\ket{0}, \ket{1}\}$ basis, but still stays in the $\{\ket{+}, \ket{-}\}$ basis. Normally you would do change of basis in matrix form, and I did it in bra-ket notation here, so I guess that's why it's weird?
Anyways, I hope that helps!
