Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "HOWTO09 How to use Federated Cloud Storage"

From EGIWiki
Jump to navigation Jump to search
(20 intermediate revisions by 3 users not shown)
Line 5: Line 5:
{{TOC_right}}
{{TOC_right}}


{{Under_construction}}
{{DeprecatedAndMovedTo|new_location=https://docs.egi.eu/users/online-storage/}}
 
= EGI Federated Cloud Storage Tutorial  =
 
This page aims to give a brief description of the storage services provided by the EGI Federated Cloud and a basic tutorial on how to use and integrate them into your application.
 
The guide is intended for application developers and system administrator to select the best Federated Cloud storage solution for their application needs and to understand how to integrate it into their own applications.
 
== Storage solutions overview ==
 
If you are in need of more storage than the one provided within the VM OS image disk, you can can use the EGI Federated Cloud storage services.
 
There are two kind of services, the Block Storage and the Object Storage. Both of them have their own set of advantages and disadvantages
 
Block storage is a capability of the Federated Cloud Infrastructure-as-a-Service (IaaS). It provides additional storage blocks who can be attached to a virtual machine. A storage block is a virtual disk of a given size, which may be exposed as a virtual device in the VM. You can think of this can of devices as a USB stick that can be plugged into the VMs and can be used as a normal drive. You can format it with any file system you want and mount it in your VM. Block devices are persistent, thus they keep all the data after VM shutdown and need to be explicitly destroyed when data is not needed anymore. Block storage disks can be accessed only from within a VM, and only from VMs running at the same site where the block storage is located. Also, they can be accessed by only one VM at the same time. As part of the IaaS service, block storage is managed via OCCI (or OpenStack native interface). There is a limit on the number of block storage devices you can attach on a VM and there is a limit to the maximum size of such virtual disks. These values will depend on the particular Federated Cloud site. Moreover, the disk space is accounted for the entire block storage device, regardless how much of it is actually used.
 
Object storage is a standalone service of the EGI Federated Cloud, usually referred also as Federated Cloud STorage-as-a-Service (STaaS). Object storage stores data as set of individual objects, which can have different types (e.g. files, images, documents) and are organized within containers (e.g. folders). Each object/file has is own URL, which can be used to access the resource, share the file with other people, setup custom metadata and access control lists. These objects are accessed and managed via a REST API. The STaaS interface for the EGI Federated Cloud is provided via to the CDMI standard and/or OpenStack SWIFT interface. Differently from the block storage, there is virtually no limit to the amount of data you can store, only the space used is accounted, you can access the data from any location (from any VM running at any EGI site or even from other cloud providers or from your own laptop/browser), you can expose the data via external portals (using HTTP as transport protocols), you can set access control lists per container and even make the data publicly available. On the other hand, data is accessed via a API requests, thus integration with existing applications may require a change to the application logic.
 
A summary of the main differences between Block and Object Storage is reported in the following table.
 
{| cellspacing="5" cellpadding="5" border="0" class="wikitable"
|-
|
| '''Access'''
| '''Sharing'''
| '''Accounting'''
| '''Management'''
| '''Integration'''
|-
| '''Block Storage'''
| only from within a VM
only at the same site the VM is located
| not possible
| for the entire block, regardless how much of it is actively used in the VM
| via OCCI interface (or native OpenStack for provider enabling it)
| POSIX access, easy with any application capable to write/read file from a local disk
|-
| '''Object Storage'''
| from any device connected to the internet.
| possible (data can be kept private or public)
| only for the data stored
| via CDMI interface (or native OpenStack for providers enabling it)
| files are accessed via requests to the server, requires a client to be integrated within the application
|}
 
According to your application needs, you may select one technology over the other. In general, block storage is a good and simple solution for temporary data and data which you do not need to share beside the single application running on a single VM. If you need to have your data exposed within portals or shared between different steps of your processing workflow, it is usually best to use the object storage.
 
== Block Storage ==
 
=== Managing Block Storage ===
 
Block storage is created and managed via requests to specific APIs. Once the storage is attached to a VM, is managed as a regular block device that can be managed from within the VM as any other block device.
 
==== OCCI ====
 
The EGI Federated Cloud block storage can be managed via the OCCI interface, so you can use the rOCCI command line client as shown in the examples below. The [[HOWTO11#Installation|installation guide for rOCCI client]] can help you to get the client ready.
 
To use a block storage device, you need first to create it. You can do so issuing a "create storage" OCCI command, which via rOCCI is:
 
<pre>
occi -e <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a create -r storage \
    -t occi.storage.size='num(<storage_site_in_gb>)',occi.core.title=<storage_resource_name>
</pre>
 
where:
* ''<site_occi_endpoint>'' is the OCCI endpoint of your site.
* ''<proxy_certificate>'' is a X509 proxy certificate for authentication.
* ''<storage_site_in_gb>'' is the size of your block storage device in GB. You will be accounted for the entire disk size, regardless how much space you are using from it. Consider also that this is the raw size of the disk. Actual available file space will depend on the file system. The minimum size is 1 (1 GB), while the maximum size depends on the site, but is usually no more than 2-5TB.
* ''<storage_resource_name>'' is a mnemonic name for the resource. You can use this parameter internally to discriminate between disks.
 
That command will return the ID of the newly created storage resource (typically in the form <code><nowiki>https://<site_occi_endpoint>/storage/<some-id></nowiki></code>). This ID will be used to identify the resource in subsequent commands.
 
You can list your available volumes with the list command:
 
<pre>
occi -e <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a list -r storage
</pre>
 
And get detailed information on any of the available volumes:
 
<pre>
occi -e <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a describe -r <storage_resource_id>
</pre>
 
 
After the successful creation of the storage resource, you can attach it to a VM. You can do it on an already existing VM, via the "link" command:
 
<pre>
occi -e <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a link -r <vm_id> \
    -j <storage_resource_id>
</pre>
 
You can also attach the storage directly to a VM on creation (and this be able to use it during contextualization). Just add the <code>--link</code> (or the equivalent <code>-j</code> command to the "compute create" command:
 
<pre>
occi -e <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a create -r compute \
    [...other VM creation parameters...] \
    -j <storage_resource_id>
</pre>
 
 
Please note that you can attach a storage to only one VM at the time. Any attempt to attach it to more than one VM will fail.
 
If a block storage is attached correctly to a VM, it will be listed as <code>storagelink</code> a when described, e.g:
 
<pre>
occi -e  <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a describe -r <vm_id>
[...]
  Links:
    [[ http://schemas.ogf.org/occi/infrastructure#storagelink ]]
    >> location: /storage/link/f9e5b73d-71ac-4abb-96c9-ce7649734ae1
    occi.core.source = /compute/2f6d70c6-fb75-4372-9917-ac688b1391ee
    occi.core.target = /storage/7cfba655-f692-406f-a659-79b0224290cc
    occi.core.id = /storage/link/f9e5b73d-71ac-4abb-96c9-ce7649734ae1
    occi.storagelink.deviceid = /dev/vdb
[...]
</pre>
 
The <code>occi.storagelink.deviceid</code> shows the device on the VM where the disk is found.  Block storage will be persistent, so it will not be destroyed if the VM is destroyed. It is also possible to detach it and reattach it to a different VM. Detaching is performed with this command (<code>storage_link_id</code> in the example above is <code>/storage/link/f9e5b73d-71ac-4abb-96c9-ce7649734ae1</code>):
 
<pre>
occi -e  <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a unlink -r <storage_link_id>
</pre>
 
Once you do not need the storage anymore, you can delete it issuing a command like this:
 
<pre>
occi -e  <site_occi_endpoint> -n x509 -x <proxy_certificate> -X \
    -a delete -r <storage_resource_id>
</pre>
 
==== OpenStack ====
 
=== How to integrate the EGI Block Storage into your application ===
 
The easiest way to integrate block storage into your application is to do so at contextualization time. If you are using linux, you can use the following script as a sample for your contextualization/deployment script (NOTE: It may not work for all the OSes), which will format and mount the available storage resources into the /mnt/available_disks/ folder:
 
#!/bin/bash
i=0
for d in xvdb xvdc xvdd vdb vdc vdd; do
  if <nowiki>[[ -e /dev/$d ]]</nowiki>; then
    if [[ ! -e /dev/${d}1 ]]; then
      echo -e "o\nn\np\n1\n\n\nw" | fdisk /dev/$d
      sleep 1
      mkfs.ext4 /dev/${d}1
    elif [[ -e /dev/${d}2 ]]; then
      continue
    fi
    mkdir -p /mnt/additional_storage/$i
    mount -t ext4 /dev/${d}1 /mnt/additional_storage/$i
    i=$(( $i + 1 ))
  fi
done
 
You can then setup your application in its deployment script to write the application files into the block storage device. Applications will not see any difference between a block storage device and a normal hardware disk, thus no major changes should be required in the application logic.
 
Note that some OS, like [https://appdb.egi.eu/store/vappliance/cernvm/ CERNVM], will automatically detect all the attached block storage and add it to the root virtual file system.
 
== Object Storage ==
 
EGI currently offers two APIs for accessing Object Storage:
* [http://www.snia.org/cloud SNIA Cloud Data Management Interface (CDMI)], which defines a RESTful open standard for operations on storage objects, and
* [https://wiki.openstack.org/wiki/Swift SWIFT] is the OpenStack Object Store project, also providing a RESTful API for managing and accessing the objects.
 
=== CDMI ===
 
<span style="color:#ff0000">NOTE: CDMI STORAGE IS CURRENTLY IN PRE-PRODUCTION MODE IN THE EGI FEDERATED CLOUD. ACCESS TO STORAGE RESOURCES IS AVAILABLE ON REQUEST THROUGH THE EGI USER COMMUNITY SUPPORT TEAM: UCST@EGI.EU </span>
 
Most of the CDMI operations are performed via simple HTTP calls, thus it is not strictly required to have a particular client for CDMI and generic HTTP clients like curl or wget may be used. Anyway, to access all the CDMI capabilities and manage authorization, it is recommended to use a simple CDMI client, such as [https://github.com/EGI-FCTF/bCDMI bCDMI].
 
Access to object storage is performed via a different set of operations. Here we will try to list the most important ones and give some usage samples:
 
* List the content of a container:
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ list /
marica-container/
 
* Create a container:
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ mkdir test
{
  "completionStatus": "Complete",
  "objectName": "test/",
  "capabilitiesURI": "/cdmi/AUTH_113d9a9a671944648722e890ecb94d36/cdmi_capabilities/container/",
  "parentURI": "/cdmi/AUTH_113d9a9a671944648722e890ecb94d36/",
  "objectType": "application/cdmi-container",
  "metadata": {}
}
 
* Upload a file (object):
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ put -T testfile test/test.txt
 
* Download a file (object):
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ get test/test.txt -o testfile
 
* Delete a file (object):
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ delete test/test.txt
 
* Delete a container (must be empty):
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ delete test/
 
* Delete a container recursively:
$ ./bcdmi -e https://prisma-swift.ba.infn.it:8080/ delete -r test/
 
=== OpenStack SWIFT ===
 
Similarly to CDMI, SWIFT offers a RESTful API to manage your storage. There are two main clients for this platform: the [http://docs.openstack.org/cli-reference/openstack.html OpenStack command line client] and the [http://docs.openstack.org/cli-reference/swift.html swift client]. For the OpenStack client we provide an extension to authenticate with the current EGI AAI. The [[Federated_Cloud_APIs_and_SDKs#CLI_2|API and SDK guide]] contains the details on how to install the client.
 
Available resources can be gathered in  [https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=org.openstack.swift&selectItemserviceType=org.openstack.swift&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= swift services in GOCDB]. For accessing the endpoint check the <code>URL</code> of the specific provider, e.g. for [https://goc.egi.eu/portal/index.php?Page_Type=Service&id=8473 server4-epsh.unizar.es] the URL is <code><nowiki>https://server4-epsh.unizar.es:5000/v2.0/</nowiki></code>. With your openstack client, first get which projects are you allowed in:
 
<pre>
$ keystone_tenants --os-auth-url https://server4-epsh.unizar.es:5000/v2.0/ \
                  --os-auth-type v2voms --os-x509-user-proxy $X509_USER_PROXY
Tenant id: fffd98393bae4bf0acf66237c8f292ad
Tenant name: egi
Enabled: True
Description: egi fedcloud
</pre>
 
With the tenant id, you can use the openstack client directly:
 
<pre>
$ openstack --os-auth-url https://server4-epsh.unizar.es:5000/v2.0/ \
            --os-auth-type v2voms os-x509-user-proxy $X509_USER_PROXY \
            --os-project-id fffd98393bae4bf0acf66237c8f292ad \
            <commands>
</pre>
 
For convenience you can also set the parameters as environment variables, so you don't have to include in every other command:
<pre>
export OS_AUTH_URL=https://server4-epsh.unizar.es:5000/v2.0/
export OS_AUTH_TYPE=v2voms
export OS_X509_USER_PROXY=$X509_USER_PROXY
export OS_PROJECT_ID=fffd98393bae4bf0acf66237c8f292ad
</pre>
 
Here follows some common operations:
* List containers:
<pre>
$ openstack container list
+-----------------+
| Name            |
+-----------------+
| Cloudflow      |
| my-new-bucket23 |
+-----------------+
</pre>
 
* Create a container
<pre>
$ openstack container create test
+---------+-----------+------------+
| account | container | x-trans-id |
+---------+-----------+------------+
| v1      | test      | None      |
+---------+-----------+------------+
</pre>
 
* Create an object on a container:
<pre>
$ openstack object create test test.txt
+----------+-----------+----------------------------------+
| object  | container | etag                            |
+----------+-----------+----------------------------------+
| test.txt | test      | 3fc8eaba542609681ac900797e67ac98 |
+----------+-----------+----------------------------------+
</pre>
 
* List objects on a container:
<pre>
$ openstack object list test
+----------+
| Name    |
+----------+
| test.txt |
+----------+
</pre>
 
* Download object:
<pre>
$ openstack object save test test.txt
</pre>
 
* Delete object:
<pre>
$ openstack object delete test test.txt
</pre>
 
* Delete container (must be empty):
<pre>
openstack container delete test
</pre>
 
* Delete recursively container:
<pre>
openstack container delete -r test
</pre>
 
==== Swift client ====
 
Although the swift client does not integrate directly with the EGI AAI, it does include some useful features that are missing in the OpenStack command-line client that may be relevant for your needs. Nevertheless it's quite simple to use the swift client following this instructions:
 
First get a token with the OpenStack client (adapt arguments to your endpoint):
<pre>
$ openstack --os-auth-url https://fsd-cloud.zam.kfa-juelich.de:5000/v2.0 \
            --os-auth-type v2voms --os-x509-user-proxy $X509_USER_PROXY \
            --os-project-id df37f5b1ebc94604964c2854b9c0551f \
            token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-05-27T12:21:14Z            |
| id        | 70c8706f59bb4986bef3e463b9169477 |
| project_id | df37f5b1ebc94604964c2854b9c0551f |
| user_id    | 54ab086a6c1949dab782f7addb2689da |
+------------+----------------------------------+
</pre>
 
Get the URL of the SWIFT endpoint (again adapt arguments to your endpoint):
<pre>
$ openstack --os-auth-url https://fsd-cloud.zam.kfa-juelich.de:5000/v2.0 \
            --os-auth-type v2voms --os-x509-user-proxy $X509_USER_PROXY \
            --os-project-id df37f5b1ebc94604964c2854b9c0551f \
            catalog show swift
+-----------+-----------------------------------------------------------------------------------------------+
| Field    | Value                                                                                        |
+-----------+-----------------------------------------------------------------------------------------------+
| endpoints | FSDCloud                                                                                      |
|          |  publicURL: https://swift.zam.kfa-juelich.de:8888/v1/AUTH_df37f5b1ebc94604964c2854b9c0551f  |
|          |  internalURL: https://swift.zam.kfa-juelich.de:8888/v1/AUTH_df37f5b1ebc94604964c2854b9c0551f |
|          |  adminURL: https://swift.zam.kfa-juelich.de:8888/v1                                          |
|          |                                                                                              |
| name      | swift                                                                                        |
| type      | object-store                                                                                  |
+-----------+-----------------------------------------------------------------------------------------------+
</pre>
 
Now you can use the token and the public URL to access with the swift client the resources, for example the stat command:
<pre>
swift --os-auth-token 70c8706f59bb4986bef3e463b9169477 \
      --os-storage-url https://swift.zam.kfa-juelich.de:8888/v1/AUTH_df37f5b1ebc94604964c2854b9c0551f \
      stat
                        Account: AUTH_df37f5b1ebc94604964c2854b9c0551f
                    Containers: 19
                        Objects: 141
                          Bytes: 26977607038
Containers in policy "policy-0": 19
  Objects in policy "policy-0": 141
    Bytes in policy "policy-0": 26977607038
                            Via: 1.1 swift.zam.kfa-juelich.de:8888
    X-Account-Project-Domain-Id: default
                    Connection: close
                        Server: Apache
                    X-Timestamp: 1354717393.97315
                    X-Trans-Id: txff3a507662cf484990e61-0057482f0a
                  Content-Type: text/plain; charset=utf-8
                  Accept-Ranges: bytes
</pre>
 
===== Large files =====
 
Swift client will manage the upload of large files automatically for you and split into segments as required to meet the size limits of the server. The <code>[http://docs.openstack.org/cli-reference/swift.html#swift-upload swift upload]</code> command will do that for you, just use it as usual:
 
<pre>
swift upload <container> <file>
</pre>
 
The upload options can be further refined as described in the [http://docs.openstack.org/cli-reference/swift.html#swift-upload swift upload documentation] and [http://docs.openstack.org/developer/swift/overview_large_objects.html Large Objects overview of OpenStack]. When you download the object, you will get the whole file independently of how the upload was splitted.
 
===== Setting ACLs =====
 
One of the interesting features of Object Storage is the possibility to set ACLs on the containers. You can easily do this with the <code>[http://docs.openstack.org/cli-reference/swift.html swift post]</code> command. For example to set a container as publicly readable:
 
<pre>
swift post -r ".r:*" <container>
</pre>
 
To remove public access from a container
<pre>
swift post -r "" <container>
</pre>
 
Check the OpenStack documentation for more information.
 
=== How to integrate the EGI Object Storage into your application ===
 
Integration of the object storage within your application will require a client (check the sections above for how to use them),  libraries/SDKs that implement CDMI or SWIFT access or direct coding of the API operations into your application (these are HTTP RESTful and relatively simple to implement).
 
You must also consider the need for authentication: read operations may be performed in containers if they are set as public (see above for how to set ACLs), so you can access them just with the URL. Other operations (e.g. write) require authentication. FedCloud currently uses X.509 proxy certificates with VOMS extensions that depending on the provider are used in different ways. The [Federated_Cloud_APIs_and_SDKs#Authentication|authentication section of the APIs and SDKs page] contains more details on how to authenticate against the resources.
 
If you have a legacy application that is only able to access files on a filesystem, you can always download/upload your objects with an independent client that is executed before/after your application.

Revision as of 09:20, 16 October 2020