You are right with your research and assumptions. MacOS doesn't provide scaling options for all external screens. RDM enables you to access unavailable resolutions. Though you won't get a perfect image, since you always want to use the full resolution, but different scaling. But if the native scaling is not good enough, and MacOS doesn't provide other scaling options, you will end up with a sub par image. Either not sharp or not properly scaled elements, as you described.
This is a problem many people had before: E.g. here, here and probably the best solution in here. To make it short: Apparently you can get the system to be scaled using this and set RDM to a high resolution. But I never tested this.
I understand that you want to buy a new monitor, that is capable to be sharp and scaled correctly using macOS out of the box. Makes sense. Unfortunately there is no reference or overview of scaling supported displays to my knowledge. It seems to be mostly trial and error. So I can share my experiences:
Helpful overview of display resolution standards
Displays that don't work properly on a Mac
- 27" 4KUHD: 3840 x 2160, 163.18ppi - which is almost every 4KUHD screen available
- 25" 2K (QHD)QHD - like this Benq 25", 117.49ppi (only native scaling available, therefore everything is quite small)
Apple Screens
- 13.3" MacBooks: 2560 × 1600, 227ppi
- 16" MBP: 3072 x 1920 (16:10), 226ppi
- iMac 21.5" (Full HD): 1920 x 1080, 102.5ppi
- iMac 21.5" (4K): 4096 x 2304, 219ppi
- LG Ultrafine 23.7": 3840 x 2160, 185.9ppi
- LG Ultrafine 27" and iMac (27"): 5120 x 2880, 218ppi
- Pro Display XDR (32"): 6016 x 3384 (16:9), 218ppi
Other screens that work properly on a Mac
- 24" Full HD, 91.79ppi
- ...more?
It would be great to hear others experiences with different screen sizes and resolutions, to help you with your decision.