Join Log in
Score:0
avatar Server

Download from URL into GCP bucket

cn flag
FelixHJ

Given a URL that points to a large file, is there a shortcut to getting that file into a bucket, without downloading it locally and then uploading?

Is there a simpler way to do it, other than building a docker image that does the task and run that in GCP somehow?

97
0 + 0
Score:1
Divyani Yadav avatar Server
ng flag
Divyani Yadav

As mentioned in the link

1: You can do it using curl command by streaming the data directly into the bucket.

curl http://speedtest.tele2.net/10GB.zip | gsutil cp -gs://YOUR_BUCKET_NAME/10GB.zip

2: Files smaller than 4.6 GB.

Original approach: This works as long as your download is less than ~ 4.6 GB. Launch the Cloud Shell (first icon on your top right after you login to your project in GCP) and use wget to download the file you want. For instance, to download 7-Zip type: wget https://www.7-zip.org/a/7z1805-x64.exe Now with the file in your Cloud Shell user home you can copy it to a Google Cloud Storage bucket using the gsutil command:

gsutil cp ./7z1805-x64.exe gs://your_bucket_name/

3: Files bigger than 4.6 GB.

Create a directory in your Cloud Shell user home mkdir ~/mybucket

Now mount your bucket in that directory using gcsfuse:

gcsfuse bucket_name ~/mybucket

Change the current directory to mount point directory: cd mybucket (if you want to run "df -h ." to see how much space you got in that mount point) Now use wget to get the file directly into your bucket (sample using 10GB file off the web):

wget https://speed.hetzner.de/10GB.bin

For more information, you can refer to the documentation on storage transfer services and creating a URL list.

0 + 0
Score:0
John Hanley avatar Server
cn flag
John Hanley

Google Cloud Storage does not provide the ability to load an object from a URL.

You must write your own code, use the CLI or a managed service such as Storage Transfer Service

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.