You have to determine that problem is xfce releted or not.
So for this replace "startxfce4" with "xterm", "dwm" like then see the results.
Launct your remote Xorg from different vt if an Xorg session is already exists and active there.
Because of my REP and cant make comments.
Edit: Is Xorg allows more than one sessions(remote or not) active at the same time?
If ok then xfce4 allows more than one sessions(remote or not) active at the same time?
Well, to fix my own fault above, strongly possibly that remote Xorg have no business with "vt" but not sure.