Add simple config object as a cacheable dependency?

I am building a custom breadcrumb that has some admin settings from a module in Drupal 9. When that settings form is saved, I want any breadcrumb cache that was built with this builder to be invalidated.

I tried passing my config object into the builder as such:

$breadcrumb->addCacheableDependency($this->config);

Only when I save the form, the breadcrumb serves the cached version until I clear it. Then it shows the right breadcrumb. My constructor has:

  public function __construct(
    AdminContext $admin_context,
    AliasManagerInterface $alias_manager,
    EntityTypeManagerInterface $entity_type_manager,
    ConfigFactoryInterface $config_factory
  ) {
    $this->adminContext = $admin_context;
    $this->aliasManager = $alias_manager;
    $this->entityTypeManager = $entity_type_manager;
    $this->config = $config_factory->get('mymodule.settings');
  }

To answer the question. Yes, this is the correct way.

You can check whether the cache tag config:* is passed to the block variables:

function mymodule_preprocess_block__system_breadcrumb_block(&$variables) {
  var_dump($variables['content']['#cache']);
}

Then it can only be a problem with the template not rendering the entire content of the content variable.