How to execute migration in queue through cron run?

Ahmad

6/9/23, 2:33 PM

I am migrating some nodes from an API.

The API only accepts 60 requests per minute.

So it means if 60 items are imported in a minute, all other items will not be imported because the API will return 429 error code in that case.

So on the next cron, the same 60 items will be processed and will be updated. All the other items will never be imported.

    function mymodule_cron() {
      /** @var \Drupal\migrate\Plugin\MigrationInterface $migration */
          $migration = $this->migrationPluginManager->createInstance($migration_id);

          $migration->getIdMap()->prepareUpdate();
          $executable = new MigrateExecutable($migration, new MigrateMessage());
          $executable->import();
    }

So what should I do in this case to import all the items?

Is there any queue functionality with the cron which executes migration? So that I can tell to execute 60 items per queue and execute one queue per minute, something like that?

Update:

$request = $this->httpClient->get($url, [
 'headers' => [
   'Content-Type' => 'application/json',
   'Accept' => 'application/json',
   'Authorization' => 'Bearer ' . $this->token,
 ],
]);
$news_item = json_decode($request->getBody(), TRUE);
$source = $migration->getSourceConfiguration();
$source['data_rows'] = $news_item;
$migration->set('source', $source);

They are 300+ items.

0 + 0

migrations

cron

queue

Joseph

6/9/23, 2:56 PM

How big of a response is this API returning? How many items is it? Is it JSON? CSV? How big is the response size (MB)?

Ahmad

6/10/23, 5:46 AM

The response is in JSON. Instead of using the URL, I sent a request to the URL and then use the `embedded_data `plugin to avoid multiple requests for a single node, because I am also migrating the images and taxonomy from that response. Please check the update section of the question.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How to execute migration in queue through cron run?

TH: จะดำเนินการย้ายข้อมูลในคิวผ่านการรัน cron ได้อย่างไร

RO: Cum se execută migrarea în coadă prin cron run?

RU: Как выполнить миграцию в очереди через запуск cron?

VI: Làm cách nào để thực hiện di chuyển trong hàng đợi thông qua chạy cron?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.