How to mount EODATA as a filesystem using Goofys in Linux on WEkEO Elasticity
In this article, you will mount the EODATA repository using goofys on your Linux virtual machine.
The EODATA repository contains dozens of petabytes of Earth observation data from missions such as Sentinel-2A or Sentinel-5P.
Goofys is a piece of software used for mounting S3 containers. You will use it for mounting the bucket containing the EODATA repository on your Linux VM.
On WEkEO Elasticity WAW3-1 cloud the virtual machines come configured mounting of the EODATA repository using s3fs. The following article contains more information: How to mount eodata using s3fs in Linux on WEkEO Elasticity. This article, on the other hand, is for users who want to use goofys.
Prerequisites
No. 1 Account
You need a WEkEO Elasticity hosting account with access to the Horizon interface: https://horizon.cloudferro.com.
No 2. Linux VM
You will need a Linux VM on the WEkEO Elasticity WAW3-1 cloud with access to the EODATA network. You can create it by following one of the articles:
This article was created for Ubuntu 20.04. If you are using a different Linux distribution, adjust the commands accordingly.
No 3. Basic knowledge of Linux command line
Basic knowledge of Linux command line is required.
What We Are Going To Cover
Installing goofys
Providing credentials to configure goofys
Mounting the EODATA repository
Configuring automatic mounting of EODATA
Stopping the automatic mounting of EODATA repository using goofys
Step 1: Install goofys
Login to the eouser account of your virtual machine using SSH.
Install wget to be able to download the goofys binary directly to your virtual machine:
sudo apt update && sudo apt upgrade && sudo apt install wget
Now, navigate to your home directory:
cd
Use wget to download the goofys binary:
wget https://github.com/kahing/goofys/releases/latest/download/goofys
Make the goofys binary executable:
chmod +x goofys
To update goofys in the future:
make sure that the data in the EODATA repository is not actively used by any software on your virtual machine,
remove the goofys binary from your home directory,
execute the above commands (those starting with wget and chmod) from your home directory again and
reboot your virtual machine.
Step 2: Provide credentials to configure goofys
Make a folder to store your credentials:
mkdir ~/.aws/
Create file ~/.aws/credentials using your favorite text editor (for example nano or vim). Add the following contents to it:
[s3]
aws_access_key_id=CLOUDFERRO
aws_secret_access_key=PUBLIC
Save the file and exit the text editor.
Step 3: Mount EODATA
Create the directory which will be the mount point for EODATA, for example:
mkdir ~/eo
Mount EODATA using the following command:
./goofys --region RegionOne --profile s3 --endpoint http://data.cloudferro.com DIAS ~/eo
In this command,
./goofys is the location of the goofys binary you downloaded
~/eo is the location of the folder in which you want to mount the EODATA filesystem
DIAS is the name of the bucket which contains the EODATA resources.
You can replace the first two values listed above as needed.
In order to test whether the mount was successful, navigate to the directory in which you mounted the EODATA repository, for example:
cd ~/eo
Use the ls command to list its content. You should see the output similar to this:
The EODATA repository has now been mounted using goofys.
After each reboot, you will need to execute the mounting command above again.
Step 4 Mounting EODATA on startup
Here is how to automatically mount the EODATA repository using goofys at system startup.
If you still have the EODATA repository mounted using the method described in Step 3, reboot your virtual machine.
Create directory in /root folder in which you will store the credentials:
sudo mkdir /root/.aws
Copy the credentials you created in your local directory to the .aws directory in the /root folder:
sudo cp ~/.aws/credentials /root/.aws/
Now, add the line below to the file /etc/fstab. In order to do that, you need to start your text editor with elevated privileges (for example using sudo).
/home/eouser/goofys#DIAS /home/eouser/eo fuse _netdev,allow_other,–dir-mode=0777,–file-mode=0666,–region=RegionOne,–profile=s3,–endpoint=http://data.cloudferro.com 0 0
In the above command,
/home/eouser/goofys is the goofys binary you downloaded and
/home/eouser/eo is the location in which you wish to mount EODATA.
Once you have added that line to your /etc/fstab file, reboot the VM. After the system has restarted, check whether the EODATA repository is mounted in the directory specified by you.
Stopping the automatic mounting of EODATA repository using goofys
If you want to stop automatic mounting of the EODATA repository, remove the line you added in Step 3 from /etc/fstab file. You can also comment it out by adding # character in front of that line.a
After that, reboot the VM. Optionally, you can also remove the goofys binary and the credentials if you no longer want to use goofys.
What To Do Next
You can also try other methods of accessing EODATA, such as:
How to mount eodata using s3fs in Linux on WEkEO Elasticity
How to download EODATA files using s3cmd on WEkEO Elasticity
If you want to access EODATA on a Windows VM, you can do it by following this article:
How to mount EODATA on Windows virtual machine on WEkEO Elasticity hosting
If you find that the amount of disk space on your VM is insufficient for your EODATA processing, you can create and attach a volume to it. You can do so by following one of these articles: