You have to determine that problem is xfce releted or not.
So for this replace "startxfce4" with "xterm", "dwm" like then see the results.
Launct your remote Xorg from different vt if an Xorg session is already exists and active there.
Because of my REP and cant make comments.
Edit: Is Xorg allows more than one sessions(remote or not) active at the same time?