Score:3

Migrating content from live to development environment with Docker

id flag

I'm trying to create a localhost development server with a dockerized drupal 8 environment and need to transfer the content from the live site to localhost.

I have followed the instructions for dockerization here: https://www.drupal.org/docs/develop/local-server-setup/docker-with-solr-integration/docker-configuration

And now I have to migrate the content from the live site into the dev server, and I'm confused about the process.

Any information would be highly appreciated!

Edit:

One way to import into my local environment was to create a SQL dump from the live database and then upload that into my docker-compose file.

version: '3'
services:
  db:
    image: mariadb:10.3
    environment:
      MYSQL_DATABASE: drupal
      MYSQL_ROOT_PASSWORD: MyGreatPassword
    volumes:
      - db_data:/var/lib/mysql
    restart: always

  phpmyadmin:
    depends_on:
      - db
    image: phpmyadmin/phpmyadmin
    restart: always
    ports:
      - "8080:80"
    environment:
      PMA_HOST: db
      MYSQL_ROOT_PASSWORD: MyGreatPassword
      UPLOAD_LIMIT: '500M'

  drupal:
    depends_on:
      - db
    build: .
    ports:
      - "2345:80"
      - "2443:443"
    volumes:
      - ./:/var/www/html
    restart: always

  solr:
    image: solr:8
    ports:
      - "8983:8983"
    volumes:
      - ./mycores/collection1:/mycores/collection1
    entrypoint:
      - docker-entrypoint.sh
      - solr-precreate
      - collection1
      - /mycores/collection1
volumes:
  db_data:

Now, I'm having this error.

In Statement.php line 59                                                                              
  SQLSTATE[42S22]: Column not found: 1054 Unknown column 'tags' in 'field list'

In Connection.php line 701:
                                                                                                                                             
  SQLSTATE[42S22]: Column not found: 1054 Unknown column 'tags' in 'field list': INSERT INTO {cache_bootstrap} (cid, expire, created, tags,  
   checksum, data, serialized) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder  
  _3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6) ON DUPLICATE KEY UPDATE cid = VALUES(cid), expire = VAL  
  UES(expire), created = VALUES(created), tags = VALUES(tags), checksum = VALUES(checksum), data = VALUES(data), serialized = VALUES(serial  
  ized); Array                                                                                                                               
  (                                                                                                                                          
      [:db_insert_placeholder_0] => hook_info                                                                                                
      [:db_insert_placeholder_1] => -1                                                                                                       
      [:db_insert_placeholder_2] => 1626559961.843                                                                                           
      [:db_insert_placeholder_3] =>                                                                                                          
      [:db_insert_placeholder_4] => 0                                                                                                        
      [:db_insert_placeholder_5] => a:39:{s:10:"token_info";a:1:{s:5:"group";s:6:"tokens";}s:16:"token_info_alter";a:1:{s:5:"group";s:6:"to  
  kens";}s:6:"tokens";a:1:{s:5:"group";s:6:"tokens";}s:12:"tokens_alter";a:1:{s:5:"group";s:6:"tokens";}s:10:"views_data";a:1:{s:5:"group";  
  s:5:"views";}s:16:"views_data_alter";a:1:{s:5:"group";s:5:"views";}s:13:"views_analyze";a:1:{s:5:"group";s:5:"views";}s:22:"views_invalid  
  ate_cache";a:1:{s:5:"group";s:5:"views";}s:26:"views_plugins_access_alter";a:1:{s:5:"group";s:5:"views";}s:24:"views_plugins_area_alter";  
  a:1:{s:5:"group";s:5:"views";}s:28:"views_plugins_argument_alter";a:1:{s:5:"group";s:5:"views";}s:36:"views_plugins_argument_default_alte  
  r";a:1:{s:5:"group";s:5:"views";}s:38:"views_plugins_argument_validator_alter";a:1:{s:5:"group";s:5:"views";}s:25:"views_plugins_cache_al  
  ter";a:1:{s:5:"group";s:5:"views";}s:36:"views_plugins_display_extender_alter";a:1:{s:5:"group";s:5:"views";}s:27:"views_plugins_display_  
  alter";a:1:{s:5:"group";s:5:"views";}s:32:"views_plugins_exposed_form_alter";a:1:{s:5:"group";s:5:"views";}s:25:"views_plugins_field_alte  
  r";a:1:{s:5:"group";s:5:"views";}s:26:"views_plugins_filter_alter";a:1:{s:5:"group";s:5:"views";}s:24:"views_plugins_join_alter";a:1:{s:5  
  :"group";s:5:"views";}s:25:"views_plugins_pager_alter";a:1:{s:5:"group";s:5:"views";}s:25:"views_plugins_query_alter";a:1:{s:5:"group";s:  
  5:"views";}s:32:"views_plugins_relationship_alter";a:1:{s:5:"group";s:5:"views";}s:23:"views_plugins_row_alter";a:1:{s:5:"group";s:5:"vie  
  ws";}s:24:"views_plugins_sort_alter";a:1:{s:5:"group";s:5:"views";}s:25:"views_plugins_style_alter";a:1:{s:5:"group";s:5:"views";}s:26:"v  
  iews_plugins_wizard_alter";a:1:{s:5:"group";s:5:"views";}s:25:"views_query_substitutions";a:1:{s:5:"group";s:15:"views_execution";}s:24:"  
  views_form_substitutions";a:1:{s:5:"group";s:15:"views_execution";}s:14:"views_pre_view";a:1:{s:5:"group";s:15:"views_execution";}s:15:"v  
  iews_pre_build";a:1:{s:5:"group";s:15:"views_execution";}s:16:"views_post_build";a:1:{s:5:"group";s:15:"views_execution";}s:17:"views_pre  
  _execute";a:1:{s:5:"group";s:15:"views_execution";}s:18:"views_post_execute";a:1:{s:5:"group";s:15:"views_execution";}s:16:"views_pre_ren  
  der";a:1:{s:5:"group";s:15:"views_execution";}s:17:"views_post_render";a:1:{s:5:"group";s:15:"views_execution";}s:17:"views_query_alter";  
  a:1:{s:5:"group";s:15:"views_execution";}s:16:"field_views_data";a:1:{s:5:"group";s:5:"views";}s:22:"field_views_data_alter";a:1:{s:5:"gr  
  oup";s:5:"views";}}                                                                                                                        
      [:db_insert_placeholder_6] => 1                                                                                                        
  ) 

I've read some solutions to update the entities, but the drush command is no longer been maintained.

Run drush updb

Run drush entup
Score:3
nr flag

There are two parts to the migration: the database and the static files.

The most important part is the database. You'll need a database backup (aka sqldump) of the production database, and you'll need to import that into your local database. There are several ways to accomplish this, depending on your site hosting and whether or not you can use drush.

The local site should be somewhat functional after a database import, but it may look odd if it is missing static assets such as image files. Depending on how the production site is configured, you can probably find most of these in the public files directory. The most common location for these files is ./web/sites/default/files but the path may differ. You'll need to copy these files from the production web host to your local environment using a tool like rsync, scp, or sftp.

If you have installed Drush (via composer require drush/drush) then you may be able to use drush sql:sync to synchronize databases.

Before you do that, you might want to set up Drush aliases for local and remote environments.

Keep in mind, also, that drush migrate is used for a different purpose -- migrating site contents from another non-Drupal platform into your Drupal site.

As a thought experiment, let's say you had three copies of your Drupal site -- Dev, Stage, and Prod -- on one server (using three different Apache2 VirtualHosts with DocumentRoot set to /var/www/dev, /var/www/stg, and /var/www/prd).

In that case, you could use the local_sync function defined below to sync the database and files from a higher environment to a lower environment:

#!/bin/bash
​
function local_drush () {
  if [[ $# -eq 0 ]]; then
    echo "local_drush (set environment and execute project-local drush)"
    echo "Usage: local_drush [dev stg prd] [drush-args]"
    echo "Sets environment to dev stg or prod, then passes argv to local drush."
  fi
  if [[ $1 == 'dev' ]] || [[ $1 == 'stg' ]] || [[ $1 == 'prd' ]]; then
    RETURN_DIR=${PWD}
    cd /var/www/$1
    ./vendor/bin/drush ${@:2}
    cd ${RETURN_DIR}
  else
    echo "Invalid usage: first arg must be dev, stg, or prd."
  fi
}
​
function local_sync () {
    if [[ $# -lt 1 ]] || [[ $# -gt 2 ]]; then
    echo "local_sync (migrate db and files to a lower environment)"
    echo "Usage: local_sync <SOURCE> <TARGET>"
    echo "Set source and target environments, then sync db and files."
  fi
  if [[ $1 == 'prd' ]] || [[ $1 == 'stg' ]]; then
    SOURCE=$1
    if [[ $2 == 'stg' ]] || [[ $2 == 'dev' ]]; then
      TARGET=$2
      if [[ ( $1 = 'prd' && $2 == 'stg' ) || ( $1 = 'prd' && $2 == 'dev' ) || ( $1 = 'stg' && $2 == 'dev' ) ]]; then
        RETURN_DIR=${PWD}
        BACKUP_DIR=$HOME/sqldumps/${SOURCE}
        BACKUP_FILE=${BACKUP_DIR}/${SOURCE}_$(date -I).sql
        mkdir -p ${BACKUP_DIR}
        SOURCE_FILES=/var/www/${SOURCE}/web/sites/default/files/
        TARGET_FILES=/var/www/${TARGET}/web/sites/default/files/
        EXCLUDE_CSS=${SOURCE_FILES}css/
        EXCLUDE_JS=${SOURCE_FILES}js/
        EXCLUDE_PHP=${SOURCE_FILES}php/
​
        echo "Creating temporary database backup of ${SOURCE} environment ..."
        local_drush ${SOURCE} sql:dump > ${BACKUP_FILE}
​
        if [[ -s "${BACKUP_FILE}" ]]; then
          echo "Importing database backup into ${TARGET} environment ..."
          local_drush ${TARGET} sqlc < ${BACKUP_FILE}
          echo "Deploying staged configuration for ${TARGET} environment ..."
          local_drush ${TARGET} deploy
          echo "Copying public files from ${SOURCE} to ${TARGET} ..."
          rsync --exclude ${EXCLUDE_CSS} \
          --exclude ${EXCLUDE_JS} \
          --exclude ${EXCLUDE_PHP} \
          --delete --progress -av ${SOURCE_FILES} ${TARGET_FILES}
          echo "Rebuilding Drupal caches for ${TARGET} environment ..."
          local_drush ${TARGET} cache:rebuild
          echo "Import from ${SOURCE} to ${TARGET} completed successfully."
          echo "Removing database backup of ${SOURCE} environment ..."
          rm ${BACKUP_FILE}
          cd ${RETURN_DIR}
        else
          echo "Error: ${BACKUP_FILE} not found, cannot continue."
        fi
      else
        echo "Invalid usage: cannot copy from ${SOURCE} to ${TARGET}."
      fi
    else
      echo "Invalid usage: second arg must be stg or dev."
    fi
  else
    if [[ $1 == 'dev' ]]; then
      echo "Invalid usage: cannot migrate from dev to higher environment."
    else
      echo "Invalid usage: first arg must be prd or stg."
    fi
  fi
}

Note that the local_sync command depends on the local_drush command, which in turn depends on the local and remote projects containing a site-local Drush in ./vendor/bin/drush, which is the recommended method of installing Drush. The shell script is non-standard, but it has the advantage of working whether or not you have configured Drush aliases properly.

I'll leave it to you as an exercise to figure out how to get the files from your remote server into your local environment. There's nothing wrong with doing all these steps manually, or with using Filezilla to transfer files via SFTP over SSH, but consider that the rsync command (on lines 49-52) can be used with a different ${SOURCE} which can be on a remote server. The only other change you'd need to make would be to also rsync the ${BACKUP_FILE} to your local (around line 42).

Ha! I see that you edited your question while I was writing my answer.

Your new question is a duplicate, answered here by Clive.

From the comments on that answer:

The official way is for modules relying on the old way to explicitly write the code to make the updates happen in hook_update_N. All the gritty details are here: https://drupal.org/node/3034742

The workaround is to install the Devel Entity Updates module, which makes the old drush entup command available.

You're not supposed to use this in production, but it's OK to use in your local development environment as long as you don't promote your local DB to the production environment.

It sounds like you are only having this problem in your local, not in production, so this option may work for you.

Good luck!

Renato Francia avatar
id flag
Thank you! Appreciate this feedback
hotwebmatter avatar
nr flag
Glad to help! If my answer solves your problem, be sure to accept it by clicking the green check mark.
Score:2
id flag

Ok, I was able to do it quite easily with drush dumps.

First, I did a dump on the live server with:

drush sql:dump > db.sql

And then in my docker environment.

First I made sure I drop the tables from the database from Phpmyadmin.

Then I entered the docker image.

docker container exec -it <docker-container-id> bash

And then run drush with the name of the database I dumped from live(I use filezilla to transfer it to my local dev).

drush sql:cli < db.sql

Make sure that you download the public files with the images and add them to the files folder in local.

I first created the site and settings with an empty site. Then, added only the images since replacing the generated css, js and php in files created problems.

Hope this helps.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.