For $i=1,2$, let $B_i\in\mathrm{SL}_n(\mathbb{Z})$ be two symmetric positive definite matrices. We define their automorphism groups as $$\mathrm{Aut}(B_i)=\{g\in\mathrm{GL}_n(\mathbb{Z})\mid\ gB_ig^{\top}=B_i\}.$$
In Charlap Exercise 6.4 of Chapter I, page 36, it is claimed that if $\mathrm{Aut}(B_1)$ is conjugate to $\mathrm{Aut}(B_2)$ within $\mathrm{GL}_n(\mathbb{Z})$, then $B_1$ and $B_2$ are congruent under $\mathrm{GL}_n(\mathbb{Z})$.
I have been able to find a proof for $n=2$, but I'm stuck for higher $n$. If one were to show that, in this context, $\mathrm{Aut}(B_i)$ always acts irreducibly on $\mathbb{C}^n$, the claim would follow. Unfortunately, I don't even see why $\mathrm{Aut}(B_i)$ would contain more than $\{\pm I_2\}$.
