Score:0

Migrate Json data with non-numeric unique identifier (from facebook...)

cm flag

I am trying to fetch posts form facebook, using migrate.

My question is:

in my migration conf.yml how should I set the source:ids key to handle non-numeric IDs, like the ones provided by facebook that look like that 1333842536759100_23306506123438


Some details:

I fetch a json from FB that look like that:

{
   "data": [
      {
         "permalink_url": "URL",
         "created_time": "2021-11-06T16:59:16+0000",
         "is_popular": false,
         "message": "xxxxx",
         "id": "1333842536759100_23306506123438"
      }, {...}
   ]
}

In my source YML i've got:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  track_changes: false
  urls:
    - 'XXX'
  item_selector: data
  fields:
    -
      name: id
      label: 'Unique post identifier'
      selector: id
   [...]
  ids:
    id:
      type: string

But when I try to import, I get:

Numeric value out of range: 1264 Out  
   of range value for column 'sourceid1' at row 1: INSE  
  RT INTO "migrate_map_facebook_posts" ("source_ids_has  
  h", "sourceid1", "source_row_status", "rollback_actio  
  n", "hash") VALUES (:db_insert_placeholder_0, :db_ins  
  ert_placeholder_1, :db_insert_placeholder_2, :db_inse  
  rt_placeholder_3, :db_insert_placeholder_4); Array     
  (                                                      
      [:db_insert_placeholder_0] => c21c5df349e2f4591c8  
  34b5342c35d69fca6f9aed586b75227057791d41b097b          
      [:db_insert_placeholder_1] => 1333842536759100_23306506123438                                         
      [:db_insert_placeholder_2] => 3                    
      [:db_insert_placeholder_3] => 0                    
      [:db_insert_placeholder_4] =>                      
  ) 

               

Despite the ID type being set as "string", and I see in my DB the column being int(13)...

How can I handle this non-numeric facebook ID ?

Score:0
cm flag

It turns out I was looking in the wrong place: my yml was fine at this point, but before, I did import the migration conf with ids as integer.

The problem was that the database tables created by migrate (migrate_map_YOURMIGRATIONNAME and migrate_message_YOURMIGRATIONNAME ) won't get updated if you merely change and reimport your migration configuration.

In the end, I had to drop them manually for them to get recreated with the right structure.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.