Score:0

How to handle large queue process

cn flag

I'm using the QueueWorker plugin to do some update/create node process in the background. On local there isn't an issue, it completes the whole queue process. However on the aws server it usually stops at some point. I'm assuming because of the resource consumption on the server. Whats the ideal way to optimize my QueueWorker?

Here is my code:

$offset = 0;

while (TRUE) {
  $nodes= \Drupal::entityQuery('node')->condition('type', 'article')->range($offset, $limit)->execute();

  $offset = $offset + $limit;
  if (empty($nodes)) {
    break;
  }

  // Initialize QUEUE
  $queue_manager = \Drupal::service('plugin.manager.queue_worker');
  $queue_worker = $queue_manager->createInstance('ex_queue');
  $queue = $this->queueFactory->get('ex_queue');
  // Create QUEUE items
  foreach ($nodes as $node) {
    $item = new \stdClass();
    $item->content = $node;
    $queue->createItem($item);
  }
  // Execute QUEUE items
  while ($item = $queue->claimItem()) {
    try {
      $queue_worker->processItem($item->data);
      $queue->deleteItem($item);
    }
    catch (RequeueException $e) {
      $queue->releaseItem($item);
      \Drupal::logger('system')->warning('RequeueException');
    }
    catch (SuspendQueueException $e) {
      $queue->releaseItem($item);
      \Drupal::logger('system')->error('SuspendQueueException');
    }
    catch (\Exception $e) {
      $queue->releaseItem($item);
      \Drupal::logger('system')->error('Exception');
    }
  }
}

and my QueueWorker

/**
 * @QueueWorker(
 *   id = "ex_queue",
 *   title = @Translation("Ex Processor"),
 *   cron = {"time" = 3600}
 * )
 */
class ExQueueProcessor extends QueueWorkerBase implements ContainerFactoryPluginInterface {

  protected $configuration;

  /**
   * {@inheritdoc}
   */
  public function __construct(array $configuration) {
    $this->configuration = $configuration;
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
    return new static(
      $configuration
    );
  }
       /**
       * {@inheritdoc}
       */
      public function processItem($item) {
        // do things
      }

Lets say, the total count of $nodes is 17k items, and it stops at around 15k. Is there anyway to optimize this more to make it handle large data?

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.