IE: why isn't Wayland vector scaling a vector image format, and what can I do about it? Surely the correct situation is that rendering should happen at the full capability of the display, independently of the scaling factor?
Wayland doesn't deal with fonts, or with any kind of rendering, at all. The only thing it receives is flat pixel buffers that are ready to be shown on screen. (Same with X11 in recent years, really – no app or toolkit written in the last decade or maybe two has used any of X11's very-limited text drawing facilities, always doing it client-side instead and giving Xorg finished pixmaps.)
The rest of the UI, too, is pre-rendered by the app and given to the display server as a bitmap rather than any form of vector.
So the only way it can work is that the Wayland server informs apps themselves to scale their rendering output by 2x or so (which for example GTK3/GTK4 will be able to do, drawing not just the vector fonts but everything else at 2x the size).
But if apps do not respond to the request, then the display server has only one option of stretching the final image buffer, as that's the only thing it has been given. To my knowledge, fractional scaling in Wayland uses a different "protocol" than the older integer scaling, so it's very likely that LibreOffice's "VCL" UI toolkit does not yet support it – or that it's actually still running in X11 mode through Xwayland.
LibreOffice's VCL is really a frontend to various toolkits (it can have the UI drawn by GTK3, by Qt5, by Windows native widgets...) so it seems that you would get better results if you told it to use the more modern Qt6 backend:
$ export SAL_USE_VCLPLUGIN=qt6 $ libreoffice --writer
…assuming the version in Ubuntu supports that, and assuming you have Qt6 installed. (GNOME's standard toolkit is GTK, which has gained some hacky form of fractional scaling support very recently in GTK 4.8, I believe, but LibreOffice still uses GTK 3.x.)
I've done the same thing on Windows 11 and it doesn't seem to have this unfortunate downside.
It does. Many Windows programs (older ones especially) still do not support scaling their UI; Windows tries to handle it at UI toolkit level for programs that use the basic GDI widgets (similar to how GTK or Qt do their own scaling), but often it doesn't go well and even Windows has to revert to bitmap scaling. (For example, we are still using an old accounting program where UI toolkit scaling results in a total mess of overlapping buttons and other elements, so we have to go to "Compatibility Options" and choose bitmap scaling instead.)
