VOSpace backend

Introduction

This repository hosts the code of the VOSpace backend. This is a dockerized version including all the VOSpace components which can be directly executed on your laptop.

The VOSpace implementation is composed by several parts. Each of these parts is represented by one of the following repositories.

For a production-like demo, please refer to the vospace-demo repository or simply visit this page and try it out.

For more information about the VOSpace specification, please refer to:

• IVOA Documents & Standards
• VOSpace standard v2.1
• Universal Worker Service Pattern v1.1

Further documentation on the VOSpace implementation can be found here.

Main features

• Recursive scan, checksum calculation and .tar generation for data provided by users
• Database interaction for storing and retrieving information about VOSpace nodes, jobs, storage points and users
• Simple FCFS (First Come First Served) scheduling mechanism for jobs based on Redis lists
• Set of command line tools to simplify the administrator interaction with the backend architecture.

Getting started

First of all clone the repository on your local Linux machine, open a terminal and place yourself within the vospace-transfer-service folder. You can launch the whole environment by running the following commands (as user):

docker-compose pull
docker-compose up

The web interface will be available on your browser at http://localhost:8080/ once all the containers are up and running.

To stop the environment and perform a cleanup, launch the following commands from another shell:

docker-compose down
docker system prune -a
docker volume prune

Components

• Client (container_name: client), is used to provide to the user some useful command line tools to interact with the backend
• Transfer service (container_name: transfer_service) is the core of the backend architecture
• RabbitMQ (container_name: rabbitmq), is a AMQP broker used to deliver messages containing requests from the user command line tools and from the VOSpace REST APIs
• Redis (container_name: job_cache), is used as a cache for job queues
• File catalog (container_name: file_catalog), now available here, is a posgresql database used to store information on VOSpace nodes, but also on storage locations, jobs and users.

Client

The client container provides the following command line tools:

• vos_data: launches a job to automatically store data provided by the user on a given storage point (hot or cold)
• vos_import: imports VOSpace nodes on the file catalog for data already stored on a given storage point
• vos_job: provides information about jobs
• vos_storage: allows to add/list/remove storage points.

You can launch each of these commands without any argument to see their help page.

Transfer service

The transfer service is the core of the VOSpace backend architecture.

You can access the transfer_service container with:

docker exec -it transfer_service bash

On this container, hosted on the so-called transfer node, each user will find his/her own home folder and two subfolders representing respectively the entry point and exit point for the user data:

• /home/name.surname/store
• /home/name.surname/retrieve

The user will copy the data to be stored within the store folder, while he/she will find the requested data within the retrieve folder. This use case was implemented in order to try to offer support to users providing huge amounts of data in the order of terabytes.

RabbitMQ

You can access the RabbitMQ web interface via browser in two steps.

1. Find the IP address of the RabbitMQ broker:

docker network inspect vospace-transfer-service_backend_net | grep -i -A 3 rabbitmq

2. Open your browser and point it to http://IPv4Address:15672 (user: guest, password: guest)

Redis

You can access the Redis server from the client container by following the steps here below.

1. Execute an interactive bash shell on the client container:

docker exec -it client bash

2. Use redis-cli command to connect to Redis:

redis-cli -h job_cache

3. You can obtain some info about the jobs by searching them on the following queues:
   • For write operations the queues are write_pending, write_ready and write_terminated
   • For read operations the queues are read_pending, read_ready and read_terminated.

Example: list the first six elements of the write_ready queue

redis:6379> lrange write_ready 0 5

File catalog

You can access the file catalog from the client container by following the steps here below.

1. Execute an interactive bash shell on the client container:

docker exec -it client bash

2. Access the database via psql client:

psql -h file_catalog -d vospace_testdb -U postgres

3. You can now perform a query, for example show all the tuples of the node table displaying some fields:

vospace_testdb=# SELECT node_id, path, name, parent_path, type, owner_id, content_md5, async_trans, sticky FROM node;

You can perform whatever query also on the other tables: deleted_node, storage, location, job and users.