How get the view mode\third_party_settings of a field inside hook_preprocess_image_formatter(&$variables)

I'm on D9.

I've added a custom option to the image field formatter following Drupal guidelines

In the preprocess of the field I can get the settings I've created:

function my_module_preprocess_field(&$variables) {
  if ($variables["element"]["#formatter"] === 'image') {
    $entity = $variables['element']['#object'];
    $view_mode = $variables['element']['#view_mode'];
    $field_name = $variables['element']['#field_name'];

    $entity_display = EntityViewDisplay::collectRenderDisplay($entity, $view_mode);
    $field_display = $entity_display->getComponent($field_name);

    $variables['my_settings'] = isset($field_display["third_party_settings"]["my_module"]["my_settings"]) && $field_display["third_party_settings"]["my_module"]["my_settings"];
  }
}

However, the field.html.twig file is not good for my need: I have to pass this settings to the image-formatter.html.twig file, because I need to place a div right after the img tag and not outside the a.

Unfortunately, I can't get that information in the preprocess of the image formatter, because I can't find a way to get the view mode:

function my_module_preprocess_image_formatter(&$variables) {
  $item = $variables['item'];
  $entity = $item->getEntity();
  $field = $item->getFieldDefinition();

  //todo how get the view_mode\the third_party_settings?
  $entity_display = EntityViewDisplay::collectRenderDisplay($entity, $missing_view_mode);
}

Is that possible?

EDIT:

Thanks 4k4 answer, what I've done:

1. Added inside the preprocess field function:

    if (!empty($variables['items'])) {
      foreach ($variables['items'] as &$item) {
        $item['content']['#item_attributes']['my_settings'] = $my_settings;
      }
    }

And then added a image preprocess function in order to have that info as variable and not attribute - more a preference than a strict necessity.

function my_module_preprocess_image(&$variables) {
  $variables['my_settings'] = $variables["attributes"]["my_settings"] ?? FALSE;
  unset($variables["attributes"]["my_settings"]);
}