Why doesn't the entity access handler I wrote work when the entity is referenced?

I have set a handler for the access of my entity but when I reference it in another entity, the access is not checked. I want the reference and the entity to have the same access check. Any idea ? The entity is a taxonomy term and I reference it in a basic node

The handler below, I return forbidden to test

/**
 * Defines the access control handler for the taxonomy term entity type.
 *
 * @see \Drupal\taxonomy\Entity\Term
 */
class MyAccessControlHandler extends TermAccessControlHandler {

  /**
   * {@inheritdoc}
   */
  protected function checkAccess(EntityInterface $entity, $operation, AccountInterface $account) {
    return AccessResult::forbidden();
  }

}

Edit : Until I found a better solution, I use view and form alter hook to check the referenced entity access by getting the said entity and using it's access() method