Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "HOWTO13 EUDAT B2STAGE cookbook for EGI VOs"

From EGIWiki
Jump to navigation Jump to search
 
(7 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}
[[Category:Operations Manuals]]
[[Category:Deprecated]]
{| style="border:1px solid black; background-color:lightgrey; color: black; padding:5px; font-size:140%; width: 90%; margin: auto;"
| style="padding-right: 15px; padding-left: 15px;" |
|[[File:Alert.png]] This page is '''Deprecated''' and should no longer be used
|}
= About =
= About =
This cookbook provides the information about how to move data between EGI resources and EUDAT resources using the B2STAGE service. The page was prepared for research communities who would like to process data on EGI resources using input data from an EUDAT storage site, or pushing output data to an EUDAT storage site. The cookbook explains how to
# copy data from an EUDAT site onto an EGI site for processing;
# copy data from an EGI site to an EUDAT site for long-term storage;
Pre-requirements for this guide are:
* You are member of an EGI Virtual Organisation (so you have access to the EGI sites that support this Virtual Organisation)
* You have access to an EUDAT site


[http://eudat.eu/b2stage B2STAGE service] from EUDAT supports all known GridFTP command line and graphical or web based user clients (globus-url-copy, uberftp, WebFTS, globusonline, XSEDE-EUDAT File Manager).
[http://eudat.eu/b2stage B2STAGE service] from EUDAT supports all known GridFTP command line and graphical or web based user clients (globus-url-copy, uberftp, WebFTS, globusonline, XSEDE-EUDAT File Manager).
Line 5: Line 20:
DSS (Data Staging Script) is a command line tool which is recommended to be used if user would like to benefit from PIDs (Persistent Identification of the files which are stored on a dedicated server named EPIC server).
DSS (Data Staging Script) is a command line tool which is recommended to be used if user would like to benefit from PIDs (Persistent Identification of the files which are stored on a dedicated server named EPIC server).


This cookbook provides the information about how EGI communities can benefit from the B2STAGE service offered by [http://eudat.eu EUDAT], particularly how members of an EGI Virtual Organisation can use B2STAGE to copy large amount of data from an EUDAT Storage site onto an EGI Compute site for processing.


= EUDAT B2STAGE and EGI =
= EUDAT B2STAGE and EGI =

Latest revision as of 13:48, 10 September 2021

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Alert.png This page is Deprecated and should no longer be used

About

This cookbook provides the information about how to move data between EGI resources and EUDAT resources using the B2STAGE service. The page was prepared for research communities who would like to process data on EGI resources using input data from an EUDAT storage site, or pushing output data to an EUDAT storage site. The cookbook explains how to

  1. copy data from an EUDAT site onto an EGI site for processing;
  2. copy data from an EGI site to an EUDAT site for long-term storage;

Pre-requirements for this guide are:

  • You are member of an EGI Virtual Organisation (so you have access to the EGI sites that support this Virtual Organisation)
  • You have access to an EUDAT site

B2STAGE service from EUDAT supports all known GridFTP command line and graphical or web based user clients (globus-url-copy, uberftp, WebFTS, globusonline, XSEDE-EUDAT File Manager).

DSS (Data Staging Script) is a command line tool which is recommended to be used if user would like to benefit from PIDs (Persistent Identification of the files which are stored on a dedicated server named EPIC server).


EUDAT B2STAGE and EGI

EUDAT B2STAGE cookbook for EGI VOs.png

List of EUDAT B2STAGE service endpoints.


[1] User documentation about B2STAGE service. (EUDAT website)

[2] Site Administrator documentation about B2STAGE service. (EUDAT website)

EGI VOs

Installing Data Staging Script

1. Install Python pip

ekar@ekar-VirtualBox:~# sudo apt-get install python-pip

2. Install Globus Online API

ekar@ekar-VirtualBox:~# sudo pip install globusonline­-transfer­-api­-client

3. Install dependencies

ekar@ekar-VirtualBox:~# sudo apt-get install swig python-dev 
ekar@ekar-VirtualBox:~# sudo pip install m2crypto

4. Download Data Staging Script from GitHub

ekar@ekar-VirtualBox:~# mkdir DSS
ekar@ekar-VirtualBox:~# sudo cd DSS
ekar@ekar-VirtualBox:~/DSS# git clone https://github.com/EUDAT-B2STAGE/B2STAGE-DataStaging-script.git
ekar@ekar-VirtualBox:~/DSS# cd B2STAGE-DataStaging-script
ekar@ekar-VirtualBox:~/DSS/B2STAGE-DataStaging-script# sudo cp m2.py /usr/local/lib/python2.7/dist-packages/globusonline/transfer/api_client/x509_proxy/m2.py


5. Deploy Globus Connect if you would like to use your laptop or any other endpoint you manage as a GridFTP endpoint.

Configuring Data Staging Script

Always consult the user documentation of B2STAGE service for latest updates.

1. Ask for access to one of B2STAGE service provided endpoints.

2. datastager.cfg


[Defaults]
user =  ekarolis (Your GlobusOnline.eu account name)
cert =  ~/.globus/usercert.pem
key  =  ~/.globus/userkey.pem
certdir=~/.globus/certificates
rmode=  DSSfile
#rmode=  icommands
ipath=  /home/jack/CINECA/GridTools/iRODS/iRODS/clients/icommands/bin/

src_site = None
src_dir  = ~/
dst_site = None
dst_dir  = ~/

GClocalhost=mylaptop (Your GlobusOnline.eu Connect endpoint)
DSSfilePath=/CINECA01/home/EUDAT_STAFF/keigelis/.DSSfile (Path of the EUDAT storage server to .DSSfile)
dssfiledefserver=B2STAGE (Default GO endpoint for DSSfile)

Note: If you are using MyProxy Tool GUI you can retrieve your uploaded VOMS proxy and put it to current directory of DSS to a file named: credential-<Your GlobusOnline.eu account name>.pem. Example:


ekar@ekar-VirtualBox:~# myproxy-get-delegation -l ekarolis -s px.grid.sara.nl --voms biomed
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis/CN=467317771/CN=1594137829/CN=2054790710/CN=456155345/CN=1233761789
A credential has been received for user ekarolis in /tmp/x509up_u1000.

ekar@ekar-VirtualBox:~# cp /tmp/x509up_u1000 credential-ekarolis.pem
ekar@ekar-VirtualBox:~# chmod 600 credential-ekarolis.pem

Using Data Staging Script

Deploy Globus Connect if you would like to use your laptop or any other endpoint you manage as a GridFTP endpoint.

1. Generating random file to be used for the transfers.


ekar@ekar-VirtualBox:~# dd if=/dev/urandom of=1M.rand bs=1M count=1

2.1 Upload randomly generated file from your laptop to EUDAT endpoint (storage endpoint) using GlobusConnect client and DSS.


ekar@ekar-VirtualBox:~/DSS/B2STAGE-DataStaging-script# ./datastager.py in issue -p 1M.rand --ss mylaptop --sd /home/ekar/DSS/B2STAGE-DataStaging-script/data --ds B2STAGE --dd /CINECA01/home/EUDAT_STAFF/keigelis/

Hello, welcome to data staging!

Loading.... /You are staging in so save the taskID in order to know the PID(s).

=== Before transfer ===
=== Activate endpoints ===
Please enter your myproxy username ('none' if you do not have one).
ekarolis
==Activating mylaptop ==
==Try autoactivation==
==Activating B2STAGE ==
=== Prepare transfer ===
=== Submit transfer ===
=== Checking completion ===
Task 1fbd6982-290a-11e4-b5ce-12313940394d complete!
 Task 1fbd6982-290a-11e4-b5ce-12313940394d is still under process 
=== Exiting ===
The task ID for this operation is: 1fbd6982-290a-11e4-b5ce-12313940394d

Its status is SUCCEEDED

2.1.1 Getting the PID for the uploaded file.


ekar@ekar-VirtualBox:~/DSS/B2STAGE-DataStaging-script# ./datastager.py --verbose in pid -t 1fbd6982-290a-11e4-b5ce-12313940394d

Namespace(action='pid', cert='~/.globus/usercert.pem', certdir='~/.globus/certificates', direction='in', dssfilepath='/CINECA01/home/EUDAT_STAFF/keigelis/.DSSfile', dst_dir='~/', dst_site='None', example=False, gclocalhost='mylaptop', ipath='/home/jack/CINECA/GridTools/iRODS/iRODS/clients/icommands/bin/', key='~/.globus/userkey.pem', rmode='DSSfile', src_dir='~/', src_site='None', taskid='1fbd6982-290a-11e4-b5ce-12313940394d', user='ekarolis', verbose=True)

credential-ekarolis.pem exist
Proxy found!

Loading.... |The list of the corresponding PID is going to be saved in DSSfile.
Retrieving /CINECA01/home/EUDAT_STAFF/keigelis/.DSSfile via GridFTP
from data.repo.cineca.it that is B2STAGE
Transfer parameters: 
ekarolis
B2STAGE
mylaptop
/home/ekar/DSS/B2STAGE-DataStaging-script
/

=== Before transfer ===
=== Activate endpoints ===
Please enter your myproxy username ('none' if you do not have one).
/-ka\o|/s
==Activating B2STAGE ==
/==Activating mylaptop ==
/=== Prepare transfer ===
=== Submit transfer ===
=== Checking completion ===
|Task 5718d718-290a-11e4-b5ce-12313940394d complete!
=== Exiting ===
The task ID for this operation is: 5718d718-290a-11e4-b5ce-12313940394d

Its status is SUCCEEDED

2.1.2 Checking received PID for the uploaded file.

ekar@ekar-VirtualBox:~/DSS/B2STAGE-DataStaging-script# cat .DSSfile
irods://data.repo.cineca.it:1247/CINECA01/home/EUDAT_STAFF/keigelis/1M.rand,11100/afd7d0b8-221e-11e4-89e0-e41f13eb41b2


2.2 Upload randomly generated file from your laptop to EGI endpoint (storage endpoint) using GlobusConnect client and DSS.


ekar@ekar-VirtualBox:~/DSS/B2STAGE-DataStaging-script# ./datastager.py in issue -p 1M.rand --ss mylaptop --sd /home/ekar/DSS/B2STAGE-DataStaging-script/data --ds PUC --dd /dpm/fis.puc.cl/home/biomed/

Hello, welcome to data staging!

You are staging in so save the taskID in order to know the PID(s).
 Loading.... /

=== Before transfer ===
=== Activate endpoints ===
Please enter your myproxy username ('none' if you do not have one).
ekarolis
==Activating mylaptop ==
==Activating PUC ==
=== Prepare transfer ===
=== Submit transfer ===
=== Checking completion ===
Task b51a250c-26b1-11e4-b5ca-12313940394d complete!
 Task b51a250c-26b1-11e4-b5ca-12313940394d is still under process 
=== Exiting ===
The task ID for this operation is: b51a250c-26b1-11e4-b5ca-12313940394d

Its status is SUCCEEDED

3. Staging-in 1M.rand file to EUDAT server from EGI storage endpoint (PUC) (third party transfer).


ekar@ekar-VirtualBox:~# ./datastager.py in issue -p 1M.rand --ss PUC --sd /dpm/fis.puc.cl/home/biomed/ --ds B2STAGE --dd /CINECA01/home/EUDAT_STAFF/keigelis/

Hello, welcome to data staging!

Loading.... /You are staging in so save the taskID in order to know the PID(s).

=== Before transfer ===
=== Activate endpoints ===
Please enter your myproxy username ('none' if you do not have one).
ekarolis
==Activating PUC ==
==Activating B2STAGE ==
=== Prepare transfer ===
=== Submit transfer ===
=== Checking completion ===
Task f00a7bb4-2452-11e4-b5c1-12313940394d complete!
 Task f00a7bb4-2452-11e4-b5c1-12313940394d is still under process 
=== Exiting ===
The task ID for this operation is: f00a7bb4-2452-11e4-b5c1-12313940394d

Its status is SUCCEEDED

4. Staging-out 1M.rand file from EUDAT server using PID to EGI storage endpoint (PUC) (third party transfer).


ekar@ekar-VirtualBox:~# ./datastager.py --verbose out issue --ds PUC --dd /dpm/fis.puc.cl/home/biomed/ pid -P 11100/afd7d0b8-221e-11e4-89e0-e41f13eb41b

Namespace(action='issue', cert='~/.globus/usercert.pem', certdir='~/.globus/certificates', direction='out', dssfilepath='/CINECA01/home/EUDAT_STAFF/keigelis/.DSSfile', dst_dir='/dpm/fis.puc.cl/home/biomed/', dst_site='PUC', example=False, gclocalhost='mylaptop', ipath='/home/jack/CINECA/GridTools/iRODS/iRODS/clients/icommands/bin/', key='~/.globus/userkey.pem', pid='11100/afd7d0b8-221e-11e4-89e0-e41f13eb41b', pidfile=None, rmode='DSSfile', src_dir='~/', src_site='None', sub_action='pid', user='ekarolis', verbose=True)

credential-ekarolis.pem exist
Proxy found!

Loading....  /Using .DSSfile
DSSfileURLfromPID -> argument: 11100/afd7d0b8-221e-11e4-89e0-e41f13eb41b
The URL is in json_file
Element 1 of 1
lista:  ['Output', 'irods://data.repo.cineca.it:1247/CINECA01/home/EUDAT_STAFF/keigelis/1M.rand']
/


=== Before transfer ===
=== Activate endpoints ===
Please enter your myproxy username ('none' if you do not have one).
ekarolis
==Activating ekarolis#B2STAGE ==
==Activating PUC ==
=== Prepare transfer ===
=== Submit transfer ===
=== Checking completion ===
Task 4f9f19fa-2911-11e4-b5ce-12313940394d complete!
=== Exiting ===
The task ID for this operation is: 4f9f19fa-2911-11e4-b5ce-12313940394d

Its status is SUCCEEDED 

Setting up GlobusOnline.eu

Please consult Globus Online cookbook for EGI VOs. Information about how to configure EGI storage endpoints and other relevant information is briefly described. If your VO Manager does not provide you with public storage endpoints at GlobusOnline.eu, you can follow the procedure to set them up by yourself.