How to invalidate config or service cache

I have a service wich is loading a config at construct time:

class MyService {
  public function __construct(ConfigFactoryInterface $config_factory) {
    $config = $config_factory->get('my.config');
    ...
  }
...
}

There is a config form to fill in the my.config's values.

In the submit function of this form, I obviously save the config but after that, I would like to use MyService. Unfortunately, the config loaded by MyService doesn't contains the last changes encoded in the config form.

Not sure it is useful to know, but MyService is also loaded in buildForm()

Here is the submit function in which I am trying to invalidate my.config

public function submitForm(array &$form, FormStateInterface $form_state) {
  ...
  $config->save();
  \Drupal::cache()->invalidate('my.config');
  \Drupal::cache()->invalidate('my.service');
  $my_service = Drupal::service('my.service');
  $my_service->useMyService(); // it will use the old values
}