How to access object storage using OpenStack CLI on WEkEO Elasticity

Cloud computing offers the ability to handle large chunks of data, directly on the remote server. OpenStack module Swift was created expressly to enable access to unstructured data that can grow without bounds, with the following design goals in mind :

  • durability,

  • scalability,

  • concurrency across the entire data set,

  • all while keeping the API simple.

Swift is installed as an independent module but on the syntax level, it is used through the parameters of openstack command.

What We Are Going To Cover

  • How to install Swift

  • How to connect Swift to OpenStack cloud

  • Basic openstack operations with containers

  • Basic openstack operations with objects

Prerequisites

No. 1 Hosting

You need a WEkEO Elasticity hosting account, available at https://wekeoelasticity.cloudferro.com/. If you want to follow up with articles about object storage on Horizon, you will this link too: https://horizon.cloudferro.com.

No. 2 Install or activate openstack command

To be able to connect to the cloud, openstack command must be operational. If not installed already, use article How to install OpenStackClient for Linux on WEkEO Elasticity

No. 3 Authenticate to OpenStack using application credentials

Then you have to authenticate your account to the cloud. The usual way is to activate openstack command using an RC file for on- or two-factor authentication. That will not work in case of Swift module. It is authenticated with application credentials, as explained in article

How to generate or use Application Credentials via CLI on WEkEO Elasticity.

No. 4 Familiarity with object storage on WEkEO Elasticity OpenStack

This article is explaining the basics, using the Horizon interface:

How to use Object Storage on WEkEO Elasticity.

Swift can be understood as the CLI tool for accessing object storage under OpenStack.

No. 5 Python installed

The following articles contain sections on how to install Python:

On Ubuntu, you can also use virtualenvwrapper: How to install Python virtualenv or virtualenvwrapper on WEkEO Elasticity

How to install Swift

Here is how to install Swift on Ubuntu 22.04:

sudo apt-get install python3-pip # if not already installed, install pip on Ubuntu 22.04
pip install python-swiftclient # install python openstack client

You would then use Prerequisite No. 3 to authenticate Swift to WEkEO Elasticity.

Basic openstack commands for object data

OpenStack allows large quantities of data to be placed into so-called

  • “containers”, which are similar to folders on a desktop computer. Containers can further contain

  • “files”, which are here called “objects”.

Therefore, the commands to work with data containers and objects look like this:

  • openstack container and

  • openstack object

Use --help option to show the parameters, for instance:

openstack container --help
../_images/openstack_container_create_help.png

Basic openstack CLI operations with containers

Container name cannot contain spaces and special characters.

Correct

Incorrect

backupvelero

backup velero

backup-velero

Create container

openstack container command can create, list and delete containers:

openstack container create backup09
openstack container list
openstack container delete backup09

Here is how you can

  • create a new container, called backup09,

  • list it and thus show that it has been created, then

  • delete it and again

  • use list,

this time to show that it has been deleted:

../_images/combination_swift_commands.png

Basic openstack CLI operations with objects

Upload an Object

Similarly, it is possible to create, save, list and delete objects (which can think of as “files” in the usual sense of the word). This would be a general form of one such command:

openstack object create <container_name> <object_name>

For instance:

../_images/object_create_container_name.png

Let us list the contents of the container and verify that file test.txt is there:

../_images/object_list_backup09.png

To save file locally, add --file to denote the name of the local file:

openstack object save backup09 test.txt --file test32.txt

To delete, use:

openstack object delete <container_name> <object_name>

Set Metadata

Metadata can be used to tag and categorize objects so that later on, large amounts of data can be categorized. If you are creating some kind of media storage system, you can tag images, videos, pure text and PDF files differently. You can also add creation date as a meta tag.

This is a general command to add metadata to the object:

openstack object set --property <key=value> <container_name> <object_name>

And here is a more concrete example of

  • uploading an image to the container and

  • categorizing it as an image via a meta tag:

# First create a new container, solar_system, to contain images of solar system
openstack container create solar_system
# Upload an image and add metadata to categorize it
openstack object create solar_system image1.jpg
# Tag it as an image and with the date of creation
openstack object set \
--property type=image \
--property created=2024-06-12 solar_system image1.jpg

If you wanted to keep track of versions, you could tag the image like this:

openstack object set \
--property version=2 \
--property modified=2024-06-12 solar_system image1.jpg

What To Do Next

You can combine standard bash commands with basic object and container commands to

  • perform backups,

  • copy and/or migrate data to other containers,

  • search using metadata you entered earlier

and so on.

This type of bash programming is, however, out of scope of this article.

Also of interest:

How to access object storage from WEkEO Elasticity using boto3

How to access object storage from WEkEO Elasticity using s3cmd