I see this is done for an image field in the Manage Display options, but can't find this option for Entity Reference fields for Media (Image) with Rendered entity as the format. This affects Largest Contentful Paint, so I would love to be able to set this per Entity Reference field in a content type just like it's done with image fields. I can recreate the image tag through template files and preproccess hooks, but it seems like this should be possible through the UI.
entity reference field for media (image):
vs. regular image field showing loading option 