Notice: Undefined index: provider in *_preprocess_block()

I am in the process of upgrading a site to Drupal 9. I finally have 9.2.0 in place, but I started seeing a very large number of notices, that all fit the pattern:

Notice: Undefined index: provider in block_theme_suggestions_block() (line 203 of core/modules/block/block.module).

This isn't limited to just the block module, it seems to generate a notice in every implementation of hook_preprocess_block() (which is a lot).

It very well may be one of the custom blocks on this site, but I've compared those to the D9 documentation, and I don't see anything amiss. I've also tried clearing the twig cache.

Update: I dropped a breakpoint in the spot where the error. The element in the variables array. The #plugin_id value is just "broken" and the content is a TranslatableMarkup with the value "This block is broken or missing. You may be missing content or you might need to enable the original module."

So I'm going to try the advice in How to fix a broken or missing block?

I figured it out. I was using the twig_tweak module, and using drupal_block() to place blocks in templates manually. As a part of the updateto the twig_tweak module, you need to replace those with calls to drupal_entity().

So for example, if you called:

drupal_block('footer_meta_menu')

Then that becomes:

drupal_entity('block', 'footer_meta_menu')

Additionally, a common technique in Drupal 8 was to place blocks in the disabled region, which allowed you to render them in templates. If that is what you are doing then you need to pass an additional parameter for access, e.g.

drupal_entity('block', 'footer_meta_menu', check_access=false)

I thought I had gotten them all, but I overlooked one, which was only visible in a popover element, and not readily on page load.