How do I extract a list of all folders of my AWS S3 storage bucket or directories?

sam23

9/2/23, 7:40 AM

Does Amazon provide an easy way extract a list of all folders that have files greater than 500 MB from a s3 bucket? want to limit the scope to the '/files/ftp_upload/' directories also This is so I can calculate my costs, etc.

I had tried this but doesn't get so much help

aws s3 ls s3://YOUR_BUCKET/YOUR_FOLDER/ --recursive --human-readable --summarize

what is the best approach here ?

0 + 0

amazon-s3

amazon-ec2

amazon-web-services

aws-cli

Score:0

Server

Tim P

9/4/23, 3:13 AM

S3 does not have a concept of "folders", the console only presents the data like folders in the console by splitting object keys on the forward slash ("/"). So summarizing data by "folder" would require parsing the key of each object.

You could pull ALL of the data for objects that match your key prefix and then perform some logic & math to sum it together, but that is a lot of work.

If you don't need the data in real-time, S3 Inventory may provide a good solution. Basically you get a list of the objects in the specified S3 bucket output in one of three formats: CSV, Apache ORC or Apache Parquet. You can then do much easier computations based on the data.

The downside to S3 Inventory is that it takes a day or so to get the report and it is not real-time.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

-- If you are looking at an easier way to manage costs you may want to break the use case up using one S3 bucket for each. Then you could utilize Cost Allocation Tags at the Bucket level.

Also S3 utilization reporting is helpful for determining if you should be using a different storage class.

0 + 0

Illya Moskvin

8/24/23, 7:00 PM

Belated, but for anyone else reading this: you can add the `Size` optional field to your S3 inventory, and in your inventory parsing script, check the value of that field.

sam23

9/4/23, 6:29 AM

I just want to filter out the files which are greater than 500 MB from my bucket upload folder ? is there any advice on that?

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How do I extract a list of all folders of my AWS S3 storage bucket or directories?

TH: ฉันจะแยกรายการโฟลเดอร์ทั้งหมดของบัคเก็ตหรือไดเร็กทอรีพื้นที่จัดเก็บ AWS S3 ของฉันได้อย่างไร

RO: Cum extrag o listă cu toate folderele din compartimentul sau directoarele mele de stocare AWS S3?

RU: Как извлечь список всех папок корзины или каталогов хранилища AWS S3?

VI: Làm cách nào để trích xuất danh sách tất cả các thư mục trong bộ chứa hoặc thư mục lưu trữ AWS S3 của tôi?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.