Join Log in
Score:0
avatar Drupal

Drupal migrate - "--limit" option works only for first run?

in flag
MilanG

I have custom source plugin (imports from MongoDB) and when I run migration for the first time like:

drush mim article --limit=10

it works well: 10 articles are being imported and process stops.

But if I repeat that command then only 1 article is imported.

And if I have like 10 imported articles and run again with limit=20 then 10 new ones are going to be imported.

And if I set limit smaller than current number of imported ones then only 1 new row will be imported.

Should command I specified import always 10 new rows (if there are available, unprocessed)?

Any hint is welcome!

(using drush 11.0.7 and Drupal 9.3.12) (edited)

Update:

So it looks like that code which is counting number of processed rows and checking if limit parameter is reached is not working well, that is, it counts also previously processed rows which should not be the case.

34
0 + 0
in flag
MilanG
Looks like my source plugin is responsible for skipping processed row (answered by @mikelutz on slack). So looks like in my case processed rows are not skipped, but counted again. But I didn't override next() method. Any example on how to do this?
0
Reply
in flag
MilanG
next() method (which I didn't override) of SourcePluginBase (which I'm using as a base class) should do the checking if row is processed already or not. But looks like it fails. I had an issue before with condition like that, when previously processed are always been processed again and it was resolved by updating drush.
0
Reply
Score:0
avatar Drupal
in flag
MilanG

I solved it this way:

Original next() of SourcePluginBase has checking if row processing is needed or not this way:

  if (!$row->getIdMap() || $row->needsUpdate() || $this->aboveHighWater($row) || $this->rowChanged($row)) {
    $this->currentRow = $row->freezeSource();
  }

Now, in my class that inherits that SourcePluginBase I added to it prepareRow override:

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    if (!$row->getIdMap() || $row->needsUpdate() || $this->aboveHighWater($row)) {
      return parent::prepareRow($row);
    }
    return FALSE;
  }

I noticed that for previously imported rows condition $this->rowChanged($row) is true and for new rows that really needs to be processed $this->rowChanged($row) is true. So I removed first one and kept second one.

I'm aware that this is kind of workaround and that there is probably some incompatibility between drush and drupal, but this is the best solution I could do. If someone has better solution/explanation I will be glad to hear it.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.