How to download EODATA files using s3cmd on WEkEO Elasticity

Establishing connection

At first, please verify your connection with your EODATA buckets by using the list command.

eouser@vm01:~$ s3cmd ls

You should obtain this output.

2017-11-15 10:40  s3://DIASac
2017-11-15 10:40  s3://EOCLOUD
2017-11-15 10:40  s3://HRSI

If you cannot see these buckets, please complete all the steps from the following tutorial: /eodata/How-to-access-EODATA-and-Object-Storage-using-s3cmd-on-Linux-on-WEkEO-Elasticity.

Downloading an EOData product

An entire EOData product is available to download as a whole directory, not just a single file. Therefore, you have to use –recursive parameter in order to avoid skipping of any files.

Example of downloading a Sentinel-2 product:

s3cmd get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/

The above command will download the S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE product to your local folder from which you are issuing the command.

Files can be also redirected to a specific local folder. For our purposes it would be a downloads directory. Add the path to your local destination folder to the command.

eouser@vm01:~$ mkdir downloads
eouser@vm01:~$ s3cmd -d get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/ downloads/

Troubleshooting

If nothing happens or you could not download a product properly, please use -d switch to see a debug output for further diagnosis:

s3cmd -d get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/

Alternative solution

For syncing, you can use the s3cmd sync command. In synchronizes a local directory tree with S3.

Firstly we can create a new directory for our product and download that product using the s3cmd sync command on that directory.

Now we will present the sync method in practice.

Go to your target directory and list the files.

For our purposes we will remove the measurement directory.

In order to sync the data with your local folder, you have to put a dot (. → location of the current directory) as a last character.

After invoking the s3cmd sync command all the missing files will be found and restored, as presented below.

Done. Downloaded 5347560568 bytes in 23.7 seconds, 214.86 MB/s.
eouser@vm01:~/example-product$ ls
S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE-report-20181002T024920.pdf
annotation
manifest.safe
measurement
preview
support