If there is no GUI session used, but the system still displays a GUI for login, only this GUI login part will use memory. Processes managing this are mostly waiting and thus mostly doing nothing. If swap if enabled (something to ponder if the only disks available are SSD that have to be preserved from wearing off, anyway this answer is not about deciding this), parts of the process(es) doing nothing will be swapped out when more memory becomes needed, thus further limiting the memory footprint of the GUI.
To answer the question: using SSH won't affect GUI parts currently in use. The comparison is about the currently loaded and running parts. On a typical Debian installation, choosing Gnome gets GDM (Gnome Display Manager) for GUI login prompt, choosing XFCE gets LightDM (Lightweight Display Manager) for GUI login prompt. I would tend to say that LightDM (and thus XFCE) will use less memory. For both cases, most of it (not all, nor active parts, such as displaying the time) would be swapped out if there is swap, but without it all of it will stay in physical RAM.
Testing in a Debian 12 amd64 VM without swap, accessed through SSH and doing nothing else than providing an SSH service and offering a GUI login prompt or none, the used memory measured with free -m on the VM right after echo 3 > /proc/sys/vm/drop_caches was along a few reboots:
GDM: ~ 494-506MB
LightDM: ~ 324-331MB
neither (console-only): ~ 196-214MB
I'm sure there can be variations, but overall LightDM, and thus meaning having chosen to install XFCE, appears to use less memory than GDM and thus having chosen to install Gnome.
Both were installed, then were switched or disabled as described below and then rebooted.
Now you can also use GDM to start XFCE or use LightDM to start Gnome to further murky this point, but I believe both might loose some parts of their integration to their default manager, such as issues when locking screen or switching user which would have to be further tinkered with. On Debian to switch between them (possibly not instantaneously but only for next start) if both are installed that would be either of:
dpkg-reconfigure gdm3 dpkg-reconfigure lightdm
to get prompted (a reboot or equivalent might be needed after this or some operations below).
The best way to not use such memory is to disable completely the start of the GUI: that's what is done on most server-only systems: even when they have a video card and are able still to display on their console output, they are usually set to not display any GUI, among other reasons to spare resources, especially memory. If that's what you intend, you can do it now, without uninstalling anything yet, and still change your mind later. On Debian, for GDM this is described on this Debian wiki: GDM - Controlling the GDM daemon:
systemctl set-default multi-user.target
This also applies to LightDM.
One can still reconsider and enable them back with either:
systemctl set-default graphical.target
Or start one of them only once without enabling them back at startup:
systemctl start gdm systemctl start lightdm
startx).