Migrate accessing source fields by name instead of by selector?

MilanG

3/3/23, 12:50 PM

I'm working on the MongoDB source plugin and I'm able to do the migration, but I have to make strange compromises. When defining the source fields, I have to set name the same as selector, like this.

fields:
  -
    name: _id
    label: 'Unique ID'
    selector: _id

That works, but if I decide to change field name and keep selector, the migration fails.

fields:
  -
    name: node_id
    label: 'Unique ID'
    selector: _id

I get this error.

'node_id' is defined as a source ID but has no value.

Why is that? Why is the source array accessed by name instead of by selector?

0 + 0

migrations

Joseph

3/3/23, 1:51 PM

This highly depends on the source plugin implementation. For instance, in `migrate_plus`'s JSON source, the `selector` is xpath to the data and `name` is the shorthand. However, for `migrage_source_csv`, the `name` must be the heading value. It also has `label` but no `selector`. So you'll have to dig into the source's implementation to see what properties it needs and what it does.

Score:1

Drupal

Hudri

3/3/23, 6:06 PM

You can change the name of source fields, but you must not change the name of source fields used as source IDs.

source:
  fields:
    -
      name: _id
      label: 'Unique ID'
      selector: _id
  ids:
    _id:   <-- this is the problem, you can not rename source fields used here
      type: integer

On the first migration the module creates a mapping table migration_map_MIGRATION_ID for each remote source row <=> local entity. After that, you no longer can change the name of source fields used as source IDs.

The only way around this is a complete rollback, then manually deleting those migration tables (including table migration_message_MIGRATION_ID ) and then you can make a fresh start.

0 + 0

MilanG

3/4/23, 8:48 AM

@hundri thanks for explaining this. However source parameter is not working at all, not even for other fields. From what Joseph said and from other source I think that source plugin it self has to implement that functionality. But problem is I can find and instructions/tutorial on how to do that.

Score:0

Drupal

MilanG

3/9/23, 3:30 PM

Selectors are handled by data processor plugins. Initially I tried to use iterator object provided by MongoDB collection object. That works, but selectors are not supported. Then I saw how custom data processor is made in Migrate plus module, for parsing json and made one based on that. Basically I'm using collection iterator to get full json and then my custom data processor works with that json the similar way as Migrate Plus json data processor does.

0 + 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Migrate accessing source fields by name instead of by selector?

TH: Migrate accessing source fields by name instead of by selector?

RO: Migrate accessing source fields by name instead of by selector?

RU: Migrate accessing source fields by name instead of by selector?

VI: Migrate accessing source fields by name instead of by selector?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.