Score:1

How can I fix "Allowed memory size exhausted" on batch finish?

cn flag

The batch processed all the items, but instead of showing the finish message, I see the error "Allowed memory size of 536870912 bytes exhausted".

When I debug the code, I notice that Drupal loads each proccessed block on batch finish (ContentEntityBase->__construct). I can't figure out why Drupal is doing that.

Structure of the code:

This is a custom module with form class, and batch functions in the custom_module.module file. On form submit, the module calls the create batch function:

public function submitForm(array &$form, FormStateInterface $form_state) {
  custom_module_make_batch();
}

The custom_module_build_batch function getting ids of custom blocks (4000 or more) and generating the batch:

function custom_module_make_batch()
{

  $batch = [];

  $items = get_blocks_ids();

  $batch = custom_module_generate_batch($items);
  batch_set($batch);
}

function custom_module_generate_batch($items)
{

  $operations = [];

  $operations_groups = array_chunk($items, 50);

  foreach ($operations_groups as $key => $operations_group) {
    $operations[] = [
      'custom_module_batch_op',
      [$operations_group],
    ];
  }

  $batch = [
    'operations' => $operations,
    'finished' => 'custom_module_batch_finished',
    'title' => 'Custom batch',
    'init_message' => 'Batch is starting.',
    'progress_message' => 'Processed @current out of @total parts.',
    'error_message' => 'Batch has encountered an error.',
  ];
  return $batch;
}

function custom_module_batch_op($operations_group, &$context) {

  foreach ($operations_group as $key => $bid) {
  
    $block = \Drupal::service('entity.repository')->loadEntityByUuid('block_content', $bid);

    $block->field_name = $new_value;
    
    $block->save();
  }

}

function custom_module_batch_finished($success, $results, $operations)
{

  $messenger = \Drupal::messenger();
  if ($success) {
    // Here we could do something meaningful with the results.
    // We just display the number of nodes we processed...

    if ($total) {
      $messenger->addMessage(t('@count results processed.', ['@count' => $total]));
    } else {
      $messenger->addMessage(t('There no items for the migration'));
    }

  } else {
    // An error occurred.
    // $operations contains the operations that remained unprocessed.
    $error_operation = reset($operations);
    $messenger->addMessage(
      t(
        'An error occurred while processing @operation with arguments : @args',
        [
          '@operation' => $error_operation[0],
          '@args' => print_r($error_operation[0], TRUE),
        ]
      )
    );
  }
}
leymannx avatar
ne flag
`$new_value` is undefined. And better use `$block->set('field_MYFIELD', $new_value)` to set the value or `$block->set('field_MYFIELD', [])` to empty it.
Egor Elkin avatar
cn flag
@leymannx thanks, ok, i will use $block->set() $new_value - it's just an example, it can be array: [ 'value' => 'Some tex', 'format' => 'plain_text', ];
apaderno avatar
us flag
It seems that `get_blocks_ids()` is loading all the items, which would explain the error message. A batch callback executes the query to get the necessary items. Getting all the items and handing them in batches is not how a batch operation is supposed to work, since batch operations are done to avoid to use all the available memory and avoid time outs caused from PHP taking more time to handle the request than the assigned time.
Score:5
us flag

Batch operations are used when the number of items to handle is unknown and there could possibly be so much items that either loading all of them would use all the memory PHP make available, or handling them would take more time than the time PHP gives to the script to run. When data is loaded from a database table, even entity data, it's the batch operation callback that loads the data, like this code does, for example.

$batch_builder = (new BatchBuilder())
  ->setTitle(t('Deleting database rows'))
  ->setFinishCallback('mymodule_finished_callback')
  ->addOperation('mymodule_delete_rows', []);

batch_set($batch_builder->toArray());

function mymodule_delete_rows(&$context) {
  $connection = \Drupal::database();

  if (empty($context['sandbox'])) {
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_id'] = 0;
    $context['sandbox']['max'] = $connection
      ->query('SELECT COUNT(DISTINCT [id]) FROM {example}')
      ->fetchField();
    $context['results'] = 0;
  }

  $limit = 5;
  $result = $connection
    ->select('example')
    ->condition('id', $context['sandbox']['current_id'], '>')
    ->orderBy('id')
    ->range(0, $limit)
    ->execute();

  foreach ($result as $row) {
    $context['sandbox']['progress']++;
    $context['results']++;
    $context['sandbox']['current_id'] = $row->id;
    
    if (!empty($row->title) && is_numeric($row->title[0])) {
      $connection->delete('example')
        ->condition('id', $row->id)
        ->execute()
    }
  }
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
  }
}

function mymodule_finished_callback($success, $results, $operations, $elapsed) {
  if ($success) {
    $message = \Drupal::translation()->formatPlural($results, '@count row deleted.', '@count rows deleted.');
  }
  else {
    $message = t('Finished with an error.');
  }
  \Drupal::messenger()->addMessage($message);
}

The relevant values set in the batch operations callback are the following ones:

  • $context['finished'] tells Drupal to stop calling the batch operations callback, when its value is 1 or $context['finished'] isn't set
  • $context['results'] is passed to the callback invoked when the batch operations are done, as its second parameter
  • $context['message'] is a text message displayed in the progress page

I would not build an array containing a value for each handled item, as that would use all the available memory, when the handled items are much.

Reference

Egor Elkin avatar
cn flag
"The code seems to use all the allowed memory because it first loads all the entities and then passes their IDs to the operation callback. That's not how batch operations should be implemented." - no, the get_blocks_ids() function getting block id's from DB, in order to optimize the speed and memory usage - Please read the topic again, i'm getting "Allowed memory size of 536870912 bytes exhausted" on batch finish, when the script processed all the items Anyway thanks for your thoughts @apaderno
apaderno avatar
us flag
Still, the memory gets exhausted also because you are loading the entity IDs and creating an operation every 50 entities; with 4000 entities, that means an array of 80 batch operations instead of a single batch operation.
Egor Elkin avatar
cn flag
Same with single batch operation, i tested this variant. And again: the batch process all the items successfully, the error apearch on batch finish page load.
apaderno avatar
us flag
If you are creating an array that contains a value for each handled item, that would increase the used memory.
Egor Elkin avatar
cn flag
Correct. This doesn't solve my problem but anyway, reduce memory usage - thanks, and speed up batch loading
apaderno avatar
us flag
Unfortunately, the *memory exhausted* error appears when the code tries to allocate the last byte left of available memory, but the code consuming most of the memory could be code from a different module, including a Drupal core module. Given the code shown in the question, the only possible answer is *The code implements a batch operation callback in the wrong way.* It cannot list all the possible causes of exhausted memory.
sonfd avatar
in flag
Note that operation and finished callbacks cannot be in a `mymodule.install` file (which you may try to do because you're generating your batch from `hook_install`).
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.