Score:0

Creating file from s3fs (S3 bucket) breaks migration

in flag

I have s3fs module installed and configured, so when I upload some file it goes to S3 bucket. Then I'm making processing plugin which should copy file from another S3 bucket to Drupal's bucket and register that file. Copying bucket to bucket works well, so after that part is done I have file in Drupal's bucket at private/images/image.jpg. Inside private/images folder I also have other images uploaded from the back-end so that part should be ok.

Then my transform() function of my class that extends Drupal\migrate\Plugin\migrate\process\FileCopy should create a file and return it's id. I'm doing it like:

 $file = File::create([
        'uri' => $final_destination,
        'uid' => $uid,
        'status' => FILE_STATUS_PERMANENT,
      ]);

      $file->save();
      return $file->id();

$final_destination has the value: private://images/image.jpg, file is created, id is returned.

Now the problems:

  • migration fails and when I list messages I get: "[file]: uri.0.value=This value should be of the correct primitive type. "
  • File does exist in the back-end and it's path looks ok, but when I try to open it I get the message that file doesn't exist. When comparing with some file uploaded from the back-end path looks ok and also location in the bucket is correct.
  • What I notice is missing is record in s3fs_file table for migrated file, while manually uploaded ones have the record there.
  • Tried manually adding the record there and after that when trying to open file instead of file is missing I get message that I don't have permission to see the file (logged in as master admin).

What I'm doing wrong? Should I create file somehow differently so it's also processed by s3fs module?

Update: I added following rows after $file->save();

  $uri = $file->getFileUri();
  $stream = new S3fsStream();
  $stream->writeUriToCache($uri);

  if ($stream->waitUntilFileExists($uri)) {
    return $file->id();
  }
  throw new MigrateSkipProcessException("File could not be retrieved from S3://$uri.");

And now Instead of "file doesn't exist" I'm getting "access denied" when trying to open file, i.e. in drupal's log I see message:

Path: /system/files/images/image.jpg. Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException: in Drupal\system\FileDownloadController->download() (line 80 of /var/www/web/core/modules/system/src/FileDownloadController.php).

And there request is identical (except the file name and date stumps), but header is different, for working file uploaded in back-end:

Array
(
    [Content-Type] => image/png
    [Content-Length] => 744920
    [Cache-Control] => private
)

And for problematic one:

Array
(
    [0] => -1
)
Score:0
in flag

Problem was what plugin returns. It should return file uri object, and not file id:

return $file->getFileUri();

Also, if files are private and not referenced accessing them is not allowed.

I sit in a Tesla and translated this thread with Ai:

TH:
RO:
RU:
VI:
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.