How can one find matrix $X$ such that $X^\dagger Y X = D$ is a diagonal matrix. In particular for Y defined as
Y = ( { {a, b, 0, 0}, {a, b, 0, 0}, {0, 0, c, d}, {0, 0, a, d} } );
Where $a,b,c,d$ are in general complex, and $X^\dagger = ConjugateTranspose[X]$.
I'm not sure that matrix is unitarily diagonalizable without extra assumptions on the values that appear. Using Inverse instead of ConjugateTranspose we can do
Block[{Y = ({ {a, b, 0, 0}, {a, b, 0, 0}, {0, 0, c, d}, {0, 0, a, d} }), values, vectors, X }, {values, vectors} = Eigensystem[Y]; X = Transpose[vectors]; Inverse[X] . Y . X // FullSimplify ] Which provides the diagonal matrix $D$:
$$\left( \begin{array}{cccc} 0 & 0 & 0 & 0 \\ 0 & a+b & 0 & 0 \\ 0 & 0 & \frac{1}{2} \left(-\sqrt{4 a d+(c-d)^2}+c+d\right) & 0 \\ 0 & 0 & 0 & \frac{1}{2} \left(\sqrt{4 a d+(c-d)^2}+c+d\right) \\ \end{array} \right)$$
with $X$ given by:
$$\left( \begin{array}{cccc} -\frac{b}{a} & 1 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 0 & 0 & -\frac{\sqrt{4 a d+(c-d)^2}-c+d}{2 a} & \frac{\sqrt{4 a d+(c-d)^2}+c-d}{2 a} \\ 0 & 0 & 1 & 1 \\ \end{array} \right)$$
Edit: You can also use JordanDecomposition:
Block[ {Y = { {a, b, 0, 0}, {a, b, 0, 0}, {0, 0, c, d}, {0, 0, a, d}}, X, D }, {X, D} = JordanDecomposition[Y]; MatrixForm /@ {X, D} ] (Warning, don't actually call a symbol D.)
Reduce[matrix . Transpose[matrix] - Transpose[matrix] . matrix == 0, {a, b, c, d}]. That lead to (b == a && d == a) || (b == a && c == -a && d == -a) $\endgroup$ Reduce. That said the user did not specify whether the ConjugateTranspose should be its inverse. So the problem might be more general and maybe there is no constraint on Y $\endgroup$