How to access or list EODATA using boto3 on WEkEO Elasticity

Warning

We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is available here: How to install Python virtualenv or virtualenvwrapper on WEkEO Elasticity

If virtualenv is activated:

(myvenv) eouser@vm01:~$ pip3 install boto3

If we want to install the package globally:

eouser@vm01:~$ sudo pip3 install boto3

Two examples for listing products are presented below. You will need EODATA access key and secret key to execute that code. Obtain them by following this article: How to get credentials used for accessing EODATA on a cloud VM on WEkEO Elasticity. In each example below, replace the contents of the access_key variable with your access key and the contents of the secret_key variable with your secret key, respectively.

Client

  • low-level service access

  • generated from service description

  • exposes botocore client to the developer

  • typically maps 1:1 with the service API

  • snake-cased method names (e.g. ListBuckets API => list_buckets method)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://data.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2020/11/09/',MaxKeys=30000)['CommonPrefixes']:
   print(i)

Resource:

  • higher-level, object-oriented API

  • generated from resource description

  • uses identifiers and attributes

  • has actions (operations on resources)

  • exposes subresources and collections

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://data.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
   print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
   print(bucket_object.key)

Save your file with the .py extension and run it with the ‘python3 <filename.py>’ command in your terminal. For example:

(boto3) eouser@vm01:~$ python3 boto_list.py