Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

HOWTO12 Globus Online cookbook for EGI VOs

From EGIWiki
Jump to navigation Jump to search


The Globus Online cookbook describes how the 'SRM type' storage services that are federated into EGI can be used as endpoints of file transfers managed by Globus Online. The "biomed" VO is used as an example in the Cookbook to demonstrate EGI storage usage, but the steps are the same for other EGI VOs as well.

The cookbook was prepared as a guide for EGI Virtual Organisations (VOs) on how to use the Globus Online service that is available at http://www.globusonline.eu. GlobusOnline.eu provides robust and easy to use file transfer capabilities for EGI users. The service manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status.

The Cookbook consists of two parts:

  1. For VO Managers : The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as transfer endpoints for VO members. This registration could be performed by any member of a VO, however for most VOs the VO Manager is the most suitable person to complete this step because the VO Manager has sufficient knowledge on storage sites that support the VO and about the BDII information system where detailed information about the storages is recorded.
  2. For VO Members (researchers) : The second part provides step-by-step instructions for VO members on how to use VO storage endpoints in Globus Online. This part is relevant for any meber of any EGI VO. The list of EGI VOs and information on joining these VOs is available at http://operations-portal.egi.eu/vo.

This cookbook has been prepared by the EGI.eu User Community Support Team in consultation with representatives of EGI Operations, storage technology and information system developer groups. Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.

VO Managers

1. Get all storage managers (Glue 2.0):


-bash-3.2$ ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=glue '(&(objectclass=GLUE2StorageService))' \
-bash-3.2$ |perl -p00e 's/\r?\n //g'|grep ^dn:|cut -d" " -f2 > GLUE2StorageServiceDN

2. Filter out only storage managers for biomed VO (Glue 2.0):


-bash-3.2$ for i in `cat GLUE2StorageServiceDN`;do \
-bash-3.2$ ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" "(&(objectClass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:biomed))" GLUE2PolicyRule \
-bash-3.2$ | grep -q GLUE2PolicyRule && echo $i;done > GLUE2AccessPolicyDN


3. Get all GridFTP endpoints for biomed VO (Glue 2.0):


-bash-3.2$ for i in `cat GLUE2AccessPolicyDN`; do ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" \
-bash-3.2$ "(&(objectClass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp))" GLUE2EndpointURL \
-bash-3.2$ | grep ^GLUE |tail -n 1; echo StorageServiceID: $i|cut -d"," -f1|sed 's/GLUE2ServiceID=//g';echo "-------";done

GLUE2EndpointURL: gsiftp://gaeds020.ciemat.es:2811
StorageServiceID: glue:srm.ciemat.es/data
-------
GLUE2EndpointURL: gsiftp://se001.ipp.acad.bg:2811
StorageServiceID: glue:se001.ipp.acad.bg/data
-------
GLUE2EndpointURL: gsiftp://juliet.zih.tu-dresden.de:2811
StorageServiceID: glue:ophelia.zih.tu-dresden.de/data
-------
GLUE2EndpointURL: gsiftp://sedsk58.grid.hep.ph.ic.ac.uk:2811
StorageServiceID: glue:gfe02.grid.hep.ph.ic.ac.uk/data
-------

ATTENTION: you may not get all the GridFTP endpoints of your VO with the above query because at the moment (June 2013) only dCache type storage elements publish information in the EGI BDII with Glue 2. If your VO includes DPM and/or StoRM type storage elements too, then please complete the Additional steps for finding the GridFTP endpoints for DPM and StoRM (Glue 1.3):. (The DPM community is working on a new feature to publish Gridftp endpoints with Glue 2 (REF: https://its.cern.ch/jira/browse/LCGDM-1083).)

4. Find myproxy server for biomed VO:


-bash-3.2$ lcg-infosites --is lcg-bdii.cern.ch --vo biomed myproxy
myproxy://px.grid.sara.nl:7512/
myproxy.cern.ch:7512
myproxy.usatlas.bnl.gov:7512
myproxy://cluster6.knu.ac.kr:7512/
myproxy://grid-mypx.feit.ukim.edu.mk:7512/
myproxy://grid-px0.desy.de:7512/
myproxy://grid153.kfki.hu:7512/
myproxy://gridpx01.ifca.es:7512/
myproxy://ii.biomed.kiev.ua:7512/
myproxy://kek2-px.cc.kek.jp:7512/
myproxy://lcg-px01.icepp.jp:7512/
myproxy://lcg2proxy.ific.uv.es:7512/
myproxy://lcgpx01.jinr.ru:7512/
myproxy://lcgrbp01.gridpp.rl.ac.uk:7512/
<...>

If your VO does not have a dedicated MyPRoxy server, then the catch-all MyProxy server of EGI can be used. This catch-all service is available from CESNET at myproxy.egi.eu.

Additional steps for finding the GridFTP endpoints for DPM and StoRM (Glue 1.3):

Extra step 1. Example script to obtain GridFTP endpoint information with Glue 1.3. (Command line tools from emi-ui-3.0.0-1.el6.x86_64):

#!/bin/bash

bdii="lcg-bdii.cern.ch";

lcg-infosites --is $bdii --vo $1 se |awk '{print $4}'|sort|uniq > $1

for host in `cat $1`; do

ServiceEndpoint=`lcg-info --list-service --bdii ldap://lcg-bdii.cern.ch:2170 --vo $1 --query "ServiceType=SRM" --attrs "ServiceEndpoint"|grep $host|cut -d" " -f3|head -1;`
   
VOInfoPath=`lcg-info --list-se --bdii ldap://$bdii:2170 --vo $1 --query "SE=$host" --attrs "VOInfoPath"|grep VOInfoPath|awk '{print $3}'`;

gsiftp=`ldapsearch -LLL -x -H ldap://$bdii:2170 -x -b o=grid "(&(GlueSEAccessProtocolType=gsiftp)(GlueChunkKey=GlueSEUniqueID=$host))" GlueSEAccessProtocolEndpoint | grep ^Glue |awk '{print $2}'| tail -1`;

details=`ldapsearch -LLL -h lcg-bdii.cern.ch -p 2170 -x -b o=grid "(&(objectClass=GlueSE)(GlueSEUniqueID=$host))" GlueSEImplementationName GlueSEImplementationVersion|grep ^Glue`;

echo -e "Host: $host\nSRMServiceEndpoint: $ServiceEndpoint\nVOInfoPath: $VOInfoPath\nGlueSEAccessProtocolEndpoint: $gsiftp\n$details\n";

done;


-bash-3.2$ ./go.sh biomed

Host: dcache-se-desy.desy.de
SRMServiceEndpoint: httpg://dcache-se-desy.desy.de:8443/srm/managerv2
VOInfoPath: /pnfs/desy.de/biomed
GlueSEAccessProtocolEndpoint: gsiftp://dcache-door-desy09.desy.de:2811
GlueSEImplementationVersion: 1.9.12-12 (ns=Chimera)
GlueSEImplementationName: dCache

Host: dc2-grid-64.brunel.ac.uk
SRMServiceEndpoint: httpg://dc2-grid-64.brunel.ac.uk:8446/srm/managerv2
VOInfoPath: /dpm/brunel.ac.uk/home/biomed
GlueSEAccessProtocolEndpoint: 
GlueSEImplementationVersion: 1.8.6
GlueSEImplementationName: DPM

Host: grid2.fe.infn.it
SRMServiceEndpoint: httpg://grid2.fe.infn.it:8444/srm/managerv2
VOInfoPath: /biomed
GlueSEAccessProtocolEndpoint: 
GlueSEImplementationVersion: 1.10.0
GlueSEImplementationName: StoRM

<...>

Extra step 2. Testing GridFTP endpoint for DPM or StoRM:


-bash-3.2$ touch test
-bash-3.2$ lcg-cp file://$PWD/test srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
-bash-3.2$ lcg-gt srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test gsiftp
gsiftp://dc2-grid-pool-a4-02.brunel.ac.uk/dc2-grid-pool-a4-02.brunel.ac.uk:/data2/dpmfs/biomed/2013-05-23/test.29428758.0
d4f44efc-b0c2-4ab2-ba0d-1c80f758e25b
-bash-3.2$ lcg-del -l srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
 

Testing GridFTP endpoint:


-bash-3.2$ uberftp dc2-grid-pool-a4-02.brunel.ac.uk "ls /dpm/brunel.ac.uk/home/biomed"


5. Register the endpoints in Globus Online:

a) Go to http://www.globusonline.eu/signup, create an account with your VO name e.g. biomed. (Note that usernames can include only letters and numbers and "_".)
(Using the VO name as an account name will ensure that VO members can easily find the endpoints that are available for them.)

b) Upload your ssh public key via globusonline.eu -> "manage identities"

c) Now you can manage your account using globusonline.eu client


-bash-3.2$ ssh biomed@cli.globusonline.eu "help"

d) Register the endpoints in the Globus Online service.

Endpoints can be registered through the command line interface or the graphical portal interface of Globus Online. The command line interface allows the association of a default directory with the endpoint and this simplifies the use of the endpoint by VO members. The below example therefore shows the command line tool for endpoint registration. The MyProxy server becomes the default MyProxy, and can be replaced with other MyProxy by VO members during the activation of the endpoint.


-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-add BRUNEL_UK -p gsiftp://dc2-grid-64.brunel.ac.uk:2811"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --public BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --myproxy-server=px.grid.sara.nl BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --default-directory=/dpm/brunel.ac.uk/home/biomed/ BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-list -v BRUNEL_UK"
Name                    : biomed#BRUNEL_UK
Host(s)                 : gsiftp://dc2-grid-64.brunel.ac.uk:2811
Subject(s)              : 
Target Endpoint         : n/a
Default Directory       : /dpm/brunel.ac.uk/home/biomed/
Force Encrypted Transfer: No
Disable Verify          : No
MyProxy Server          : px.grid.sara.nl
MyProxy DN              : n/a
MyProxy OAuth Server    : n/a
Credential Status       : EXPIRED
Credential Expires      : 
Credential Subject      : 


VO Members

1. Generate a VOMS proxy and upload it into a MyProxy server. There are two ways to do this:

a). With a graphical tool, such as the GSISSH-Term tool.

OR

b). From the command line tools of an EGI grid User Interface.

The usage of GSISSH-Term to manage a VOMS proxy is detailed on a dedicated page. A usage of the command line tools is detailed below.

Generating a VOMS proxy with the command line tools (You should have these installed on the User Interface machine of your VO):

-bash-3.2$ voms-proxy-init --voms biomed

Enter GRID pass phrase for this identity:
Contacting cclcgvomsli01.in2p3.fr:15000 [/O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr] "biomed"...
Remote VOMS server contacted succesfully.

Created proxy in /tmp/x509up_u507.

Your proxy is valid until Fri May 24 04:12:03 CEST 2013

VOMS proxy lifetime: by default voms proxy extension is generated for 12hours, some voms servers within EGI allow to have lifetime for 1 week (168hours), some allow up to 24hours, please consult your VO manager.


Upload VOMS proxy to a MyProxy server with the command line tool. If your VO does not have any MyProxy server, then you can use the EGI catch-all MyProxy server. Further information about the EGI catch-all MyProxy server.


-bash-3.2$ myproxy-init -l <CHOOSE ANY USERNAME> -s px.grid.sara.nl
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Enter GRID pass phrase for this identity:
Creating proxy ........................................................................ Done
Proxy Verify OK
Your proxy is valid until: Thu May 30 16:15:47 2013
Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
Verifying - Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
A proxy valid for 168 hours (7.0 days) for user <YOUR USERNAME SPECIFIED WITH -l WITHIN THE COMMAND> now exists on px.grid.sara.nl.

ATTENTION: The password entered for MyProxy will be used to authenticate the user via GlobusOnline.eu and the user name is the one which is chosen by you and provided with "-l" argument.

2. Perform file transfers

a) Go to http://www.globusonline.eu/signup and create an account. (The account can be later associated with your EGI Single Sign-On account (EGI SSO). Attention: login with the EGI Single Sign On account is possible only if you have a valid VOMS proxy in the EGI catch-all MyProxy server.)

b) Go to "Start Transfer"

c) Find the transfer endpoints that are available for your VO by searching for your VO name in the endpoint field e.g. biomed

Gobiomed.png


IMPORTANT TO KNOW: you may use field "Credential Lifetime (hours)" and enter 168hours - 1 week of proxy lifetime. (Default is 12hours). What is important to understand is that GlobusOnline.eu will retrieve the instance of your proxy from MyProxy server and will activate the endpoint for 168hours - within the GlobusOnline.eu interface you will see that endpoint is activated for 168hours, but this might not be true because your voms proxy lifetime depends on your VOMS server, where you should consult your VO Manager. Some EGI voms servers allow up to 24hours only of the voms proxy lifetime to be.

e) Click "Authenticate"

What you can do with the endpoints?

  • Transfer files from your laptop using GlobusConnect client to a endpoint using Globus Online.
  • Transfer files from an endpoint to your laptop using GlobusConnect client and Globus Online.
  • Transfer files from an endpoint to another endpoint using Globus Online.

Please consult with the Globus Online documentations that are available at http://www.globusonline.eu.

Additional materials

The tests that have been carried out while creating this Cookbook are available at GO_testing