How to mount eodata using s3fs in Linux on WEkEO Elasticity
In this tutorial, you will access EO DATA resources using a Linux VM running on WEkEO Elasticity WAW3-1 cloud using s3fs, which is a tool for mounting S3 buckets.
What We Are Going To Cover
Accessing eodata using s3fs on new VMs using preinstalled images
Accessing eodata using s3fs on older VMs or VMs using custom images
Prerequisites
No. 1 Account
You need a WEkEO Elasticity hosting account with access to the Horizon interface: https://horizon.cloudferro.com.
No. 2 Linux VM
You must operate an Ubuntu, Debian or CentOS virtual machine running on the WEkEO Elasticity cloud. It must be connected to the eodata network.
You can create the VM using either one of the default images or uploading your own disk image of the operating system. See these articles:
How to create a Linux VM and access it from Linux command line on WEkEO Elasticity
How to create a Linux VM and access it from Windows desktop on WEkEO Elasticity.
How to upload your custom image using OpenStack CLI on WEkEO Elasticity.
No. 3 Credentials
This Prerequisite only applies if you are using an operating system image which does not come with access to EODATA repository via s3fs. It could be for example a custom image uploaded by you.
You need to obtain appropriate credentials to be able to configure s3fs. Follow this article to learn how to do that: How to get credentials used for accessing EODATA on a cloud VM on WEkEO Elasticity.
Accessing eodata using s3fs on new VMs Configured Using Default Images
New virtual machines running on WEkEO Elasticity WAW3-1 cloud that are configured using Ubuntu or CentOS default images have access to the eodata repository. It is mounted automatically at startup in the /eodata folder. The service eodata.mount is used for that purpose. To navigate to the eodata repository simply execute the following command:
cd /eodata
Now you can list its contents using the following command:
ls
You should see output similar to this:
Attention
The rest of this article applies only to VMs using operating systems which are configured with custom images.
Accessing eodata Using s3fs on Older VMs or VMs Configured Using Custom Images
Older VMs running on WEkEO Elasticity WAW3-1 cloud might not have s3fs configured. The same applies to VMs configured with custom images which are uploaded by the user. If that is the case, s3fs needs to be installed and configured.
Step 1: Install s3fs
Perform the next steps depending on the operating system you are using:
Ubuntu or Debian
Access your virtual machine using SSH or web console. Update the packages on your system and install s3fs:
sudo apt update && sudo apt upgrade
sudo apt install s3fs
CentOS
The CentOS repositiories do not have s3fs. Therefore, you will need to compile it yourself.
First, using your local computer, visit the following website (it contains the releases of s3fs): https://github.com/s3fs-fuse/s3fs-fuse/releases.
Then, in the section with the most recent release find the part Assets. From there, find the link to the zip version of the Source code.
Memorize or write it somewhere down. You will later use it as a parameter for the wget command for download to your virtual machine.
Access your VM on the WEkEO Elasticity cloud using the web console or SSH.
Update your packages:
sudo yum update
Install the prerequisites including fuse, the C++ compiler and make:
sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel wget unzip
Now, use wget to download the source code. Replace https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip with the link to the source code you found previously:
wget https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip
Use the ls command to verify that the zip archive has been downloaded:
ls
Unzip the archive (replace v1.91.zip with the name of the archive you downloaded):
unzip v1.91.zip
Use the ls command to find the name of the folder you just extracted:
ls
Now, navigate to that folder (replace s3fs-fuse-1.91 with the name of the folder you just extracted):
cd s3fs-fuse-1.91
Perform the compilation by executing the following commands in order:
./autogen.sh
./configure
make
sudo make install
s3fs should now be installed in /usr/local/bin/s3fs.
Step 2: Create a file which will store the s3fs key pair
Create a file that will be used to store the key pair used to access the eodata repository somewhere in your home directory. Give it a descriptive name, for example .passwd-eodata.
Put credentials (access key and secret key) you obtained by following Prerequisite No. 3, separated by colon, in that file. For example, if your access key is CLOUDFERRO and your secret key is PUBLIC, your file should have the following content:
CLOUDFERRO:PUBLIC
Change the permissions of this file to 600. For example, if your file is stored in your home directory and is called .passwd-eodata, this is the command you should execute:
chmod 600 ~/.passwd-eodata
Step 3 Configure mounting of the eodata repository
Open the file /etc/fstab using your favorite command line text editor for editing. You will need sudo privileges for that. For example, if you want to use nano, execute this command:
sudo nano /etc/fstab
Proceed with one of the methods below depending on whether you wish to have the eodata repository automatically mounted at system startup:
Method 1: Mount the repository automatically on system startup
Add the following line to the /etc/fstab file:
/usr/bin/s3fs#DIAS /eodata fuse passwd_file=/home/eouser/.passwd-eodata,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=data.cloudferro.com 0 0
In the command above:
Replace /usr/bin/s3fs with the location of your s3fs binary. If you installed it using apt on Debian or Ubuntu, you do not have to change anything here. If you are using a self-compiled version of s3fs created on CentOS as explained in this article, that location is /usr/local/bin/s3fs.
Replace /home/eouser/.passwd-eodata with the location of the file which contains the key pair used for mounting the eodata repository. If you followed this article, you created that file in Step 2.
Method 2: Do NOT mount the repository automatically on system startup
Add the following line to the /etc/fstab file:
/usr/bin/s3fs#DIAS /eodata fuse noauto,passwd_file=/home/eouser/.passwd-eodata,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0
In the command above:
Replace /usr/bin/s3fs with the location of your s3fs binary. If you installed it using apt on Debian or Ubuntu, you do not have to change anything here. If you are using a self-compiled version of s3fs created on CentOS as explained in this article, that location is /usr/local/bin/s3fs.
Replace /home/eouser/.passwd-eodata with the location of the file which contains the key pair used for mounting the eodata repository. If you followed this article, you created that file in Step 2.
The difference between this code and the code mentioned in Method 1 is the addition of the option noauto.
Step 4: Mount the eodata repository
Mount /eodata using s3fs:
sudo mount /eodata
Please note that if you chose Method 2 in previous step, you will need to execute this command after each system startup to access the eodata repository.
You can now navigate to the /eodata directory:
cd /eodata
To list its contents, enter the following command:
ls
You should see output similar to this:
Gathering Additional Logs for s3fs
For troubleshooting and testing, you can collect additional logs by mounting eodata with additional options for debugging. Here are the instructions used both for default and custom images.
Debugging Options for Default images
Execute the following command to perform all operations as root:
sudo -i
If you have eodata mounted automatically, unmount it:
umount /eodata
Create a file that will be used to store the key pair used to access the eodata. In this example, we will store this file in your home directory and its name will be .passwd-s3fs.
Put credentials (access key and secret key) you obtained by following Prerequisite No. 3, separated by colon, in that file. For example, if your access key is CLOUDFERRO and your secret key is PUBLIC, your file should have the following content:
CLOUDFERRO:PUBLIC
Change permissions for that file to 600 using chmod, for example:
chmod 600 ~/.passwd-s3fs
Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.
mkdir testing_dir
After that, mount eodata there with additional options for gathering more logs:
s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d
Navigate to that directory and list its contents:
cd testing_dir
ls
The output should be similar to what is seen on the screenshot below:
To try it out, copy a file from it to your local directory using the cp command, for example:
cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/
These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the WEkEO Elasticity customer support at their request.
Debugging Options for Custom images
Execute the following command to perform all commands as root:
sudo -i
If you have the eodata repository mounted, unmount it (replace /eodata with the location in which you have it mounted):
umount /eodata
Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.
mkdir testing_dir
After that, mount eodata there with additional options for gathering more logs:
s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d
Navigate to that directory and list its contents:
cd testing_dir
ls
To try it out, copy a file from it to your local directory using the cp command. For example:
cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/
These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the WEkEO Elasticity customer support at their request.