Shell script to copy multiple files

I'm trying to do a shell script that would copy a bunch of files but struggle with setting up a loop to read through the files:

aws s3 cp s3://noaa-bdp-pds/gdas.YYYYMMDD/00/atmos/hdas.t00z.sfcf000.nc s3://s3internal/raw/HDAS/hdas.YYYYMMDD_00Z.nc

Here YYYY, MM, DD are numbers that I need to loop through.

I need to loop through all years, all months, and then dates to save all files. Let me know if this can be done?

Thanks

You can achieve this using aws s3 sync with wildcard and --dryrun which produces the output:

$ aws s3 sync s3://noaa-bdp-pds . \
  --exclude "*" --include "gdas.*/00/atmos/hdas.t00z.sfcf000.nc"

(dryrun) download s3://noaa-bdp-pds/gdas.20210001/00/atmos/hdas.t00z.sfcf000.nc to noaa-bdp-pds /gdas.20210001/00/atmos/hdas.t00z.sfcf000.nc
 ...
 ...

^{Remember to use an empty directory, or it may interfere with the output.}

Now, you can use this to construct a loop:

#!/bin/bash
for line in $( \
    aws s3 sync s3://noaa-bdp-pds . \
    --exclude "*" --include "gdas.*/00/atmos/hdas.t00z.sfcf000.nc" | awk '/s3:\/\//{print $3}' --dryrun \
); do
    [[ $line =~ .*/gdas.(.*)/00/.* ]] && \
    echo aws s3 cp ${BASH_REMATCH[0]} s3://s3internal/raw/HDAS/hdas.${BASH_REMATCH[1]}_00Z.nc
done

^{When you're satisfied with the result, remove echo to copy the files.}