Join Log in
Score:0
Ben Ogorek avatar Server

Why does an S3 to S3 copy care about VPCs? Related to error: "VPC endpoints do not support cross-region requests."

ml flag
Ben Ogorek

Goal: Get files

  • from Bucket 1 in ca-central-1 in Account A
  • to Bucket 2 in us-east-1 in Account B
  • using the AWS CLI from a third machine using an the IAM role with correct S3 read and write permissions (assume unless that's unlikely)

I got the error:

ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: VPC endpoints do not support cross-region requests

and this broke my mental model about how S3 works. I thought that S3 was not behind any VPC and that VPC endpoints were just about an alternate routing pathway (other than the internet) for a machine within a private subnet.

But if you're using the CLI and asking to transfer files from one S3 bucket to another, why would a VPC come into play at all?

2742
1 + 0
Score:1
Tim avatar Server
gp flag
Tim

I assume since you say "using the AWS CLI from a third machine using an the IAM role" that your "third machine" (why third?) is an EC2 instance inside a VPC.

EC2 instances are inside VPCs. If that VPC has an S3 VPC endpoint then the EC2 instance will use it, as use of endpoints is based on DNS. You've found that VPC endpoints don't support cross-region copying.

Options:

  • Use built in S3 replication to copy the bucket / folder, but this is probably not flexible enough to do individual files
  • Remove your VPC endpoints
  • Use the --endpoint-url of the S3 CLI to manually specify the URL of a public S3 endpoint. I'm not 100% sure this will work but it's worth a try
  • Write a lambda function to do the copy, make sure it runs outside the VPC - this is the default unless you configure it to run inside the VPC
+ 3
Ben Ogorek avatar
ml flag
Ben Ogorek
I was wondering if you were going to bring up the "third machine" (third bullet point, just a machine). I've sent the S3 CLI operations from both an EC2 instance and just a Mac, both with the same ~/.aws files. I guess that's why I was confused: the S3 file transfer wouldn't be going through the machine sending the CLI commands, would it?
0
Reply
Tim avatar
gp flag
Tim
The copy operation could download / upload, or it could be an internal transfer, I don't know.
1
Reply
Ben Ogorek avatar
ml flag
Ben Ogorek
Thank you for the insight, @Tim. I was thinking that the S3 buckets were somehow in VPCs but now I understand that the VPC endpoints themselves were getting in the way.
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.