Join Log in
Score:3
avatar Drupal

File can not be copied during migration of media entity

in flag
MilanG

I'm migrating pictures into Drupal (9) media entities (image). System is using AWS S3 for local storage. I have 2 migrations:

First migrates external images into drupal's files. It copies files from S3 bucket where export is to S3 bucket used as drupal local storage and registers the files.

Second is parsing same data source but creating image media entities from those previously migrated image files.

With second migration, for most of the files it works well, but for some I get message like:

[error]  File 'private://images/2.0_138.jpg' could not be copied because a file by that name already exists in the destination directory ('').

And migration stops. If I roll back that row and try to migrate it

Stack trace when that happens looks like:

#0 /var/www/web/core/lib/Drupal/Core/File/FileSystem.php(298): Drupal\Core\File\FileSystem->prepareDestination('...', false, 2)
#1 /var/www/web/modules/contrib/s3fs/src/S3fsFileService.php(244): Drupal\Core\File\FileSystem->copy('...', false, 2)
#2 /var/www/web/modules/contrib/imagemagick/src/EventSubscriber/ImagemagickEventSubscriber.php(129): Drupal\s3fs\S3fsFileService->copy('...', '...', 2)
#3 /var/www/web/modules/contrib/imagemagick/src/EventSubscriber/ImagemagickEventSubscriber.php(217): Drupal\imagemagick\EventSubscriber\ImagemagickEventSubscriber->doEnsureSourceLocalPath(Object(Drupal\imagemagick\ImagemagickExecArguments))
#4 [internal function]: Drupal\imagemagick\EventSubscriber\ImagemagickEventSubscriber->ensureSourceLocalPath(Object(Drupal\imagemagick\Event\ImagemagickExecutionEvent), '...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#5 /var/www/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php(142): call_user_func(Array, Object(Drupal\imagemagick\Event\ImagemagickExecutionEvent), '...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#6 /var/www/web/modules/contrib/imagemagick/src/Plugin/FileMetadata/ImagemagickIdentify.php(242): Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object(Drupal\imagemagick\Event\ImagemagickExecutionEvent), '...')
#7 /var/www/web/modules/contrib/imagemagick/src/Plugin/FileMetadata/ImagemagickIdentify.php(114): Drupal\imagemagick\Plugin\FileMetadata\ImagemagickIdentify->identify()
#8 /var/www/web/modules/contrib/file_mdm/src/Plugin/FileMetadata/FileMetadataPluginBase.php(292): Drupal\imagemagick\Plugin\FileMetadata\ImagemagickIdentify->doGetMetadataFromFile()
#9 /var/www/web/modules/contrib/file_mdm/src/Plugin/FileMetadata/FileMetadataPluginBase.php(389): Drupal\file_mdm\Plugin\FileMetadata\FileMetadataPluginBase->loadMetadataFromFile()
#10 /var/www/web/modules/contrib/file_mdm/src/FileMetadata.php(187): Drupal\file_mdm\Plugin\FileMetadata\FileMetadataPluginBase->getMetadata(NULL)
#11 /var/www/web/modules/contrib/imagemagick/src/Plugin/ImageToolkit/ImagemagickToolkit.php(743): Drupal\file_mdm\FileMetadata->getMetadata('...')
#12 /var/www/web/core/lib/Drupal/Core/Image/Image.php(53): Drupal\imagemagick\Plugin\ImageToolkit\ImagemagickToolkit->parseFile()
#13 /var/www/web/core/lib/Drupal/Core/Image/ImageFactory.php(85): Drupal\Core\Image\Image->__construct(Object(Drupal\imagemagick\Plugin\ImageToolkit\ImagemagickToolkit), '...')
#14 /var/www/web/core/modules/image/src/Plugin/Field/FieldType/ImageItem.php(316): Drupal\Core\Image\ImageFactory->get('...')
#15 /var/www/web/core/lib/Drupal/Core/Field/FieldItemList.php(233): Drupal\image\Plugin\Field\FieldType\ImageItem->preSave()
#16 /var/www/web/core/lib/Drupal/Core/Field/FieldItemList.php(191): Drupal\Core\Field\FieldItemList->delegateMethod('...')
#17 /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php(941): Drupal\Core\Field\FieldItemList->preSave()
#18 /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php(891): Drupal\Core\Entity\ContentEntityStorageBase->invokeFieldMethod('...', Object(Drupal\media\Entity\Media))
#19 /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php(563): Drupal\Core\Entity\ContentEntityStorageBase->invokeHook('...', Object(Drupal\media\Entity\Media))
#20 /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php(756): Drupal\Core\Entity\EntityStorageBase->doPreSave(Object(Drupal\media\Entity\Media))
#21 /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php(517): Drupal\Core\Entity\ContentEntityStorageBase->doPreSave(Object(Drupal\media\Entity\Media))
#22 /var/www/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php(804): Drupal\Core\Entity\EntityStorageBase->save(Object(Drupal\media\Entity\Media))
#23 /var/www/web/core/modules/media/src/MediaStorage.php(29): Drupal\Core\Entity\Sql\SqlContentEntityStorage->save(Object(Drupal\media\Entity\Media))
#24 /var/www/web/core/lib/Drupal/Core/Entity/EntityBase.php(339): Drupal\media\MediaStorage->save(Object(Drupal\media\Entity\Media))
#25 /var/www/web/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php(240): Drupal\Core\Entity\EntityBase->save()
#26 /var/www/web/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php(179): Drupal\migrate\Plugin\migrate\destination\EntityContentBase->save(Object(Drupal\media\Entity\Media), Array)
#27 /var/www/web/core/modules/migrate/src/MigrateExecutable.php(248): Drupal\migrate\Plugin\migrate\destination\EntityContentBase->import(Object(Drupal\migrate\Row), Array)
#28 /var/www/vendor/drush/drush/includes/drush.inc(122): Drupal\migrate\MigrateExecutable->import()
#29 /var/www/vendor/drush/drush/includes/drush.inc(113): drush_call_user_func_array(Array, Array)
#30 /var/www/vendor/drush/drush/src/Drupal/Commands/core/MigrateRunnerCommands.php(455): drush_op(Array)
#31 [internal function]: Drush\Drupal\Commands\core\MigrateRunnerCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), '...', Array)
#32 /var/www/vendor/drush/drush/src/Drupal/Commands/core/MigrateRunnerCommands.php(405): array_walk(Array, Array, Array)
#33 [internal function]: Drush\Drupal\Commands\core\MigrateRunnerCommands->import('...', Array)
#34 /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#35 /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#36 /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#37 /var/www/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(350): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#38 /var/www/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#39 /var/www/vendor/symfony/console/Application.php(1039): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#40 /var/www/vendor/symfony/console/Application.php(275): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#41 /var/www/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#42 /var/www/vendor/drush/drush/src/Runtime/Runtime.php(121): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#43 /var/www/vendor/drush/drush/src/Runtime/Runtime.php(51): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#44 /var/www/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#45 /var/www/vendor/drush/drush/drush(4): require('...')
#46 /var/www/vendor/bin/drush(120): include('...')

So what happens (how I understood stack trace) is that when image media is created some meta data are checked (image dimensions) and ImageMagick is used for that. It can not work with remote files so it's doEnsureSourceLocalPath is called. That method figures out that image is not local and tries to copy it. It calls copy from S3fsFileService class, but it assumes that file is not on S3 so calls copy from FileSystem. That one tries to copy file but I guess again S3 wrapper class is called (?) to copy file over it self and that fails.

Now I don't understand why that happens when majority of images are processed well and image media entities are created. I don't see anything specific about those which fail.

Any idea why this happens?

Update: Place where exception is thrown (web/core/lib/Drupal/Core/File/FileSystem.php prepareDestination() method) looks like:

// Determine whether we can perform this operation based on overwrite rules.
$destination = $this->getDestinationFilename($destination, $replace);
echo "Destination: $destination\n";
if ($destination === FALSE) {
  $this->logger->error("File '%original_source' could not be copied because a file by that name already exists in the destination directory ('%destination').", [
    '%original_source' => $original_source,
    '%destination' => $destination,
  ]);
  throw new FileExistsException("File '$original_source' could not be copied because a file by that name already exists in the destination directory ('$destination').");
}

I added that echo to see what proper destination looks like and it's something like: temporary://imagemagick_C3os7P.jpg But for rows which are failing it's empty string.

88
0 + 4
9
in flag
MilanG
I'm not sure even if ImageMagick is failing copying file to temp dir or from temp to some permanent place.
0
Reply
in flag
milkovsky
Can you paste your migration config? Looks like you need to tell to replace your file if it exists. If you are using the `file_copy` plugin, you need to specify `exists: replace`.
0
Reply
in flag
MilanG
Migration that reports that error is not copying any file. Image media entity generation triggers that copy operation.
0
Reply
in flag
milkovsky
According to the error it is clear, that there are multiple parties involved: imagemagick, s3, media image entity, file permissions, etc. The problem could be in any of them. 1) The migration config yaml content would still be useful to investigate the problem. 2) I would also strongly recommend to use xdebug for troubleshooting of any problems in migration.
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.