How do I set an item definition class for a configuration field?

I am not able to set an item definition class for a configuration field.

I tried implementing hook_entity_bundle_field_info_alter() , but it doesn't work.

function hook_entity_bundle_field_info_alter(&$fields, \Drupal\Core\Entity\EntityTypeInterface $entity_type, $bundle) {
  if ($entity_type->id() === 'my_entity_type_id') {
    foreach ($fields as $field) {
      if ($field->getType() === 'file') {
        $field->getItemDefinition()->setClass(AnswerFileItem::class);
      }
    }
  }
}

It seems Drupal is ignoring it and constructs item using the base class.

My file item class:

use Drupal\file\Plugin\Field\FieldType\FileItem;

class AnswerFileItem extends FileItem {

  /**
   * {@inheritDoc}
   */
  public function getUploadLocation($data = []) {
    $data += ['my_entity_type_id' => $this->getEntity()];
    return parent::getUploadLocation($data);
  }

}

Basically I want to add additional token data to all configuration fields on given entity_type_id.

Entity class is not important, it could be node.

I think problem is in FieldItemList because it uses wrong field item class.