How to alter Layout Builder block visiblity?

Is there a hook to alter block access for blocks placed using Layout Builder? I see there is hook_block_access; but sadly this is not triggered when the block is placed using Layout Builder.

There is also work going on here: https://www.drupal.org/project/drupal/issues/2916876#comment-14279293 to allow standard block visibility settings to be made available to Layout Builder blocks; but this only allows for the defined visibilify plugins. Is this my only option, to create a custom block visiblity plugin and use it with that patch?

Is this my only option, to create a custom block visiblity plugin and use it with that patch?

No, configuring visibility of a block instance is not the only option. You can also control access on plugin level. If you add BlockPluginTrait::blockAccess to the plugin class you control all block instances of this plugin.

However, most common in Layout Builder are inline custom blocks. The plugin for these blocks implements already this method, checking access of the underlying block content entity. So in this case you can use a hook, but for block_content:

use Drupal\Core\Access\AccessResult;
use Drupal\Core\Entity\EntityInterface;
use Drupal\Core\Session\AccountInterface;

/**
 * Implements hook_ENTITY_TYPE_access().
 */
function mymodule_block_content_access(EntityInterface $entity, $operation, AccountInterface $account) {
  if ($operation == 'view') {
    if ($entity->id() == 123) {
      return AccessResult::forbidden();
    }
  }
  return AccessResult::neutral();
}