Difference between revisions of "Online Storage"

From EGIWiki
Jump to: navigation, search
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page collects guide lines on the way online storage can be accessed. Still under construction.  
+
= Overview =
 +
Online Storage allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance.
 +
Online Storage gives you complete control over the data you share and with whom.
 +
Main features:
 +
*Assign global identifiers to files
 +
*Access highly-scalable storage from anywhere
 +
*Control the data you share
 +
*Organise your data using a flexible hierarchical structure
 +
 
 +
Online Storage is based on [[#GridFTP|GridFTP]] and [[#SRM|SRM]] technology.
 +
 
 +
=Getting Started =
  
 
Prerequirements needed for GridFTP and SRM are:  
 
Prerequirements needed for GridFTP and SRM are:  
  
*A personal X509 certificate, signed by your organization or institute, issued by one of the grid certificate authorities (CAs) that are member of the IGTF or their affiliated registration authorities (RAs) is required for data transfers. Please ask your organisation or institute in order to acquire such a personal X.509 certificate from a Certification Authority and configure it properly on your machine.
+
*A personal X509 certificate, signed by your organization or institute, issued by one of the grid certificate authorities (CAs) that are member of the IGTF or their affiliated registration authorities (RAs) is required for data transfers. Please ask your organisation or institute in order to acquire such a personal X.509 certificate from a Certification Authority and configure it properly on your machine. You can start with https://www.eugridpma.org/members/worldmap/
  
 
*The distinguished name (DN) of your X.509 certificate. After receiving your personal X.509 certificate you need to forward the certificate's DN to the HLRS personnel in order to activate access to our GridFTP servers. To determine the DN, please run:
 
*The distinguished name (DN) of your X.509 certificate. After receiving your personal X.509 certificate you need to forward the certificate's DN to the HLRS personnel in order to activate access to our GridFTP servers. To determine the DN, please run:
Line 9: Line 20:
 
  $ openssl x509 -noout -subject -in <YOUR_PERSONAL_X509_CERTIFICATE_FILE>
 
  $ openssl x509 -noout -subject -in <YOUR_PERSONAL_X509_CERTIFICATE_FILE>
  
*a Linux machine with GridFTP clients installed (a User Interface installable through UMD can make the purpose)
+
*a Linux machine with [http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/user/index.html#gridftp-user-quickstart-building GridFTP clients] installed (a User Interface installable through UMD can make the purpose)
  
 
See also grid-proxy-init and voms-proxy-init to get proper X509 credentials to perform the transfer.
 
See also grid-proxy-init and voms-proxy-init to get proper X509 credentials to perform the transfer.
  
= GridFTP  =
+
== GridFTP  ==
 +
GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. Here we introduce two important GridFTP interface: globus-url-copy and uberftp. For more information about GridFTP, please refer to [http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/index.html GridFTP user guide].
  
== globus-url-copy ==
+
=== globus-url-copy ===
  
 
globus-url-copy performs a file transfer from a source to a destination endpoint. The usage is:  
 
globus-url-copy performs a file transfer from a source to a destination endpoint. The usage is:  
Line 29: Line 41:
 
  globus-url-copy gsiftp://host1/source gsiftp://host2/destination
 
  globus-url-copy gsiftp://host1/source gsiftp://host2/destination
  
== uberftp ==
+
=== uberftp ===
UberFTP is a GridFTP client. Usage is:  
+
UberFTP is an interactive client for GridFTP. Usage is:  
  
 
  uberftp <URL>
 
  uberftp <URL>
Line 36: Line 48:
 
returning a prompt, where commands can be performed interactively (ls, get, put...). Please refer to the uberftp man for further details.
 
returning a prompt, where commands can be performed interactively (ls, get, put...). Please refer to the uberftp man for further details.
  
= SRM =
+
== SRM ==
 
SRM is a hierarchical file access and transfer protocol to access a Storage Element just like GridFTP, but more powerful. The implementation of the SRM client to interact with differen storage elements could differ.  
 
SRM is a hierarchical file access and transfer protocol to access a Storage Element just like GridFTP, but more powerful. The implementation of the SRM client to interact with differen storage elements could differ.  
  
Line 45: Line 57:
 
*StoRM client guide http://italiangrid.github.io/storm/documentation/clientsrm-guide/
 
*StoRM client guide http://italiangrid.github.io/storm/documentation/clientsrm-guide/
  
= Cloud storage =
+
== Cloud storage ==
  
 
[https://wiki.egi.eu/wiki/HOWTO09 EGI HOWTO on cloud storage] is available. Official reference for using OpenStack Swift is available on the [https://docs.openstack.org/cli-reference/swift.html OpenStack documentation].  
 
[https://wiki.egi.eu/wiki/HOWTO09 EGI HOWTO on cloud storage] is available. Official reference for using OpenStack Swift is available on the [https://docs.openstack.org/cli-reference/swift.html OpenStack documentation].  
  
 
Finally, oneData client is documented on the official [https://onedata.org/docs/index.html OneData documentation reference]
 
Finally, oneData client is documented on the official [https://onedata.org/docs/index.html OneData documentation reference]

Latest revision as of 13:51, 23 August 2019

Overview

Online Storage allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom. Main features:

  • Assign global identifiers to files
  • Access highly-scalable storage from anywhere
  • Control the data you share
  • Organise your data using a flexible hierarchical structure

Online Storage is based on GridFTP and SRM technology.

Getting Started

Prerequirements needed for GridFTP and SRM are:

  • A personal X509 certificate, signed by your organization or institute, issued by one of the grid certificate authorities (CAs) that are member of the IGTF or their affiliated registration authorities (RAs) is required for data transfers. Please ask your organisation or institute in order to acquire such a personal X.509 certificate from a Certification Authority and configure it properly on your machine. You can start with https://www.eugridpma.org/members/worldmap/
  • The distinguished name (DN) of your X.509 certificate. After receiving your personal X.509 certificate you need to forward the certificate's DN to the HLRS personnel in order to activate access to our GridFTP servers. To determine the DN, please run:
$ openssl x509 -noout -subject -in <YOUR_PERSONAL_X509_CERTIFICATE_FILE>
  • a Linux machine with GridFTP clients installed (a User Interface installable through UMD can make the purpose)

See also grid-proxy-init and voms-proxy-init to get proper X509 credentials to perform the transfer.

GridFTP

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. Here we introduce two important GridFTP interface: globus-url-copy and uberftp. For more information about GridFTP, please refer to GridFTP user guide.

globus-url-copy

globus-url-copy performs a file transfer from a source to a destination endpoint. The usage is:

globus-url-copy [options] <sourceURL> <destURL>

See globus-url-copy -help for more details. An example:

$ globus-url-copy gsiftp://exampleurl.eu/home/users/username/myfile file:///localdir/username/myfile

if the sites allow that, it should be possible to make a third party transfer, namely host client triggering a transfer between two other gridftp servers. Both URIs would use gsiftp: as the protocol:

globus-url-copy gsiftp://host1/source gsiftp://host2/destination

uberftp

UberFTP is an interactive client for GridFTP. Usage is:

uberftp <URL>

returning a prompt, where commands can be performed interactively (ls, get, put...). Please refer to the uberftp man for further details.

SRM

SRM is a hierarchical file access and transfer protocol to access a Storage Element just like GridFTP, but more powerful. The implementation of the SRM client to interact with differen storage elements could differ.

Please refer to:

Cloud storage

EGI HOWTO on cloud storage is available. Official reference for using OpenStack Swift is available on the OpenStack documentation.

Finally, oneData client is documented on the official OneData documentation reference