Join Log in
Score:0
avatar Server

Bucket mounted using gcsfuse doesn't match what's in my cloud storage bucket

us flag
smontanaro

I have a Google Cloud Storage bucket and am experimenting with creating a website in Google Cloud Platform (using Python+Flask FWIW). For my initial experiment, I'm using gcsfuse to mount my GCS bucket as a filesystem:

https://github.com/GoogleCloudPlatform/gcsfuse

Following the directions I got it installed in my cloud shell and mounted it like so:

mkdir -p ~/mnt

gcsfuse <my-bucket> ~/mnt

This works, but gives me a strange (as in wildly inaccurate) view of my bucket. It shows some YYYY-MM directory structure (all empty) and a bunch of files at the top level (though far from all of them), but it doesn't look like what's really in my bucket. I can verify my actual data is still available by visiting my existing GCS-homed static website and wandering through the archived YYYY-MM directories by date and thread, viewing the messages they contain:

http://www.smontanaro.net/

(I know, it's ugly. That's one reason to use something more modern to produce a decent website.)

I am new to gcsfuse and Google Cloud Platform, so it's quite likely I'm doing something wrong. Any suggestions about where to look for clues would be much appreciated.

Edit: I upload files from my laptop using gsutil:

gsutil -m cp -I gs://<my-bucket>/path...

(It's not really big data stuff, less than 10gb. GCS was just an easy place to toss it and make it available through the web.)

Edit: I created a new bucket and tossed in a few files. I can gcsfuse mount it from a Cloud shell (for either my GCP project or GCS project) and see the files. I've been so far unsuccessful trying to mount it from my Linux laptop, getting:

SetUpBucket: OpenBucket: Bad credentials for bucket "<my-bucket>": permission denied

This when specifying a JSON key file I created for my GCP project.

104
0 + 0
RJC avatar
us flag
RJC
Please avoid exposing your bucket in public, as it may be access by other community members to try resolve or reproduce your problem and it may cause sudden spike in your API traffic.
0
Reply
RJC avatar
us flag
RJC
How did you created and populated your bucket? Besides the official documentation, are there any guides you've followed in creating your bucket? Also, can you verify or try to create a new bucket, and mount in the same drive that you mounted your current bucket, and see if you would get the same behavior. Can you try to create a new folder on your existing bucket, and see if that newly created folder would get the same problem?
0
Reply
us flag
smontanaro
Thanks @RJC. I tweaked my examples and detailed how I populate the bucket. I will see about creating a new bucket and mounting it later, once I'm fully caffeinated, then report back.
0
Reply
RJC avatar
us flag
RJC
Hello @smontanaro, `-I` uses stdin to specify a list of files or objects to copy. Could you try [rsync](https://cloud.google.com/storage/docs/gsutil/commands/rsync#description) to copy files to your bucket instead? In addition, the new error appears like a different problem. I suggest posting a new question about that one to isolate it.
0
Reply
us flag
smontanaro
Thanks, yes, I understand what -I does. My file upload process finds a list of files to upload and feeds them into gsutil which expects the list of files on stdin. I think at this point I am down to (mostly) bucket/object permission issues. I will create a new question.
0
Reply
us flag
smontanaro
Final comment... Here's my new question: https://serverfault.com/questions/1083965/confused-about-how-to-manipulate-gcs-bucket-object-permissions
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.