Looking at the page of hook_field_formatter_view, the explanation for $display is:
The display settings to use, as found in the 'display' entry of instance definitions. The array notably contains the following keys and values;
But I just found out that the $display variable does not hold accurate information.
This is what's inside my $instance (correct settings) 
This is what's inside my $display (default values for settings) 
I could overwrite $display from the $instance settings, but I can't figure out in which variable the current view mode is stored, so that I can grab the settings for the correct view mode.
As a little background info, this is a custom field with a custom formatter. Has anyone had similar issues with a custom field/formatter?