Disabling Flag link access on view

I have a custom entity displayed in a view. If some field is of a certain value, the Flag link should not be available and replaced by some text.

function my_module_preprocess_flag(&$vars){
  $f = \Drupal::service('flag.link_builder');
  if ($vars["flaggable"]->field_my_field->getValue() != null) {
    if ($vars["flaggable"]->field_my_field->getValue()[0]["value"] == "0") {
      $link = $f->build('flag', $vars['flag']->id(), 'buy');
      $link["#access"] = FALSE;

    }
  }
}

Nothing seems to happen.

What could be the solution here?

I managed to disable the link by changing my function.

function my_module_preprocess_flag(&$vars){
  if ($vars["flaggable"]->field_my_field->getValue() != null) {
    if ($vars["flaggable"]->field_my_field->getValue()[0]["value"] == "0") {
      $vars["title"]["#markup"] = "New text";
      unset($vars["attributes"]["href"]);

    }
  }
}