Since it hasn't been mentioned so far and it seems to be the right tool for exactly what you are trying to achieve: Aggregate PolygonsAggregate Polygons (see for ArcGIS Pro and for ArcGIS Desktop)
Set the minimum_hole_sizeminimum_hole_size to the minimum size of a polygon hole you want to keep. Smaller ones will be eliminated. In the example below holes smaller than 30 square meters are removed.
Also the optional minimum_areaminimum_area value in the same tool does the same for the polygon parts that are not holes.

