Disable the "Add Link" button in entity.menu.edit_form for certain roles

I'd like to hide the "Add Link" button in menu.edit_form for certain roles. Running dpm($form) for this form I can't find that button in there. In core there is menu_ui.links.action.yml with this snippet:

  entity.menu.add_link_form:
  route_name: entity.menu.add_link_form
  title: 'Add link'
  class: \Drupal\menu_ui\Plugin\Menu\LocalAction\MenuLinkAdd
  appears_on:
  - entity.menu.edit_form

I don't get it how to hide this button within Hook menu_edit_form_alter. Any hints?

After Clives hint below I've added a routingsubcriber but it the User without administrator role does still have access to the form. This works fine at other places e.g. Webform but not at admin/structure/menu/manage/{menu}/add I tried it with different route from the menu Module too but still no chance.

    <?php

namespace Drupal\hw_menu\Routing;

use Drupal\Core\Routing\RouteSubscriberBase;
use Symfony\Component\Routing\RouteCollection;

/**
 * Class RouteSubscriber
 *
 * hiding menu editing components for non-admins
 *
 * @package Drupal\hw_menu\Routing
 */
class RouteSubscriber extends RouteSubscriberBase {

  /**
   * {@inheritdoc}
   */
  protected function alterRoutes(RouteCollection $collection) {

    if ($route = $collection->get('entity.menu.add_link_form')) {
      dpm($route->getRequirements());
      $route->setRequirement('_role', 'administrator');
      dpm($route->getRequirements());
    }

  }

}