HOWTO12 Globus Online cookbook for EGI VOs

From EGIWiki
Revision as of 10:52, 11 July 2013 by Sarac (talk | contribs)
Jump to: navigation, search


The Globus Online cookbook describes how the 'SRM type' storage services that are federated into EGI can be used as endpoints of file transfers managed by Globus Online. The "biomed" VO is used as an example in the Cookbook to demonstrate EGI storage usage, but the steps are the same for other EGI VOs as well.

The cookbook was prepared as a guide for EGI Virtual Organisations (VOs) on how to use the Globus Online service that is available at http://www.globusonline.eu. GlobusOnline.eu provides robust and easy to use file transfer capabilities for EGI users. The service manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status.

The Cookbook consists of two parts:

  1. For VO Managers : The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as transfer endpoints for VO members. This registration could be performed by any member of a VO, however for most VOs the VO Manager is the most suitable person to complete this step because the VO Manager has sufficient knowledge on storage sites that support the VO and about the BDII information system where detailed information about the storages is recorded.
  2. For VO Members (researchers) : The second part provides step-by-step instructions for VO members on how to use VO storage endpoints in Globus Online. This part is relevant for any meber of any EGI VO. The list of EGI VOs and information on joining these VOs is available at http://operations-portal.egi.eu/vo.

This cookbook has been prepared by the EGI.eu User Community Support Team in consultation with representatives of EGI Operations, storage technology and information system developer groups. Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.

VO Managers

1. Get all storage managers (Glue 2.0):


-bash-3.2$ ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=glue '(&(objectclass=GLUE2StorageService))' \
-bash-3.2$ |perl -p00e 's/\r?\n //g'|grep ^dn:|cut -d" " -f2 > GLUE2StorageServiceDN

2. Filter out only storage managers for biomed VO (Glue 2.0):


-bash-3.2$ for i in `cat GLUE2StorageServiceDN`;do \
-bash-3.2$ ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" "(&(objectClass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:biomed))" GLUE2PolicyRule \
-bash-3.2$ | grep -q GLUE2PolicyRule && echo $i;done > GLUE2AccessPolicyDN


3. Get all GridFTP endpoints for biomed VO (Glue 2.0):


-bash-3.2$ for i in `cat GLUE2AccessPolicyDN`; do ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" \
-bash-3.2$ "(&(objectClass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp))" GLUE2EndpointURL \
-bash-3.2$ | grep ^GLUE |tail -n 1; echo StorageServiceID: $i|cut -d"," -f1|sed 's/GLUE2ServiceID=//g';echo "-------";done

GLUE2EndpointURL: gsiftp://gaeds020.ciemat.es:2811
StorageServiceID: glue:srm.ciemat.es/data
-------
GLUE2EndpointURL: gsiftp://se001.ipp.acad.bg:2811
StorageServiceID: glue:se001.ipp.acad.bg/data
-------
GLUE2EndpointURL: gsiftp://juliet.zih.tu-dresden.de:2811
StorageServiceID: glue:ophelia.zih.tu-dresden.de/data
-------
GLUE2EndpointURL: gsiftp://sedsk58.grid.hep.ph.ic.ac.uk:2811
StorageServiceID: glue:gfe02.grid.hep.ph.ic.ac.uk/data
-------

ATTENTION: you may not get all the GridFTP endpoints of your VO with the above query because at the moment (June 2013) only dCache type storage elements publish information in the EGI BDII with Glue 2. If your VO includes DPM and/or StoRM type storage elements too, then please complete the Additional steps for finding the GridFTP endpoints for DPM and StoRM (Glue 1.3):. (The DPM community is working on a new feature to publish Gridftp endpoints with Glue 2 (REF: https://its.cern.ch/jira/browse/LCGDM-1083).)

4. Find myproxy server for biomed VO:


-bash-3.2$ lcg-infosites --is lcg-bdii.cern.ch --vo biomed myproxy
myproxy://px.grid.sara.nl:7512/
myproxy.cern.ch:7512
myproxy.usatlas.bnl.gov:7512
myproxy://cluster6.knu.ac.kr:7512/
myproxy://grid-mypx.feit.ukim.edu.mk:7512/
myproxy://grid-px0.desy.de:7512/
myproxy://grid153.kfki.hu:7512/
myproxy://gridpx01.ifca.es:7512/
myproxy://ii.biomed.kiev.ua:7512/
myproxy://kek2-px.cc.kek.jp:7512/
myproxy://lcg-px01.icepp.jp:7512/
myproxy://lcg2proxy.ific.uv.es:7512/
myproxy://lcgpx01.jinr.ru:7512/
myproxy://lcgrbp01.gridpp.rl.ac.uk:7512/
<...>

If your VO does not have a dedicated MyPRoxy server, then the catch-all MyProxy server of EGI can be used. This catch-all service is available from CESNET at myproxy.egi.eu.

Additional steps for finding the GridFTP endpoints for DPM and StoRM (Glue 1.3):

Extra step 1. Example script to obtain GridFTP endpoint information with Glue 1.3. (Command line tools from emi-ui-3.0.0-1.el6.x86_64):

#!/bin/bash

bdii="lcg-bdii.cern.ch";

lcg-infosites --is $bdii --vo $1 se |awk '{print $4}'|sort|uniq > $1

for host in `cat $1`; do

ServiceEndpoint=`lcg-info --list-service --bdii ldap://lcg-bdii.cern.ch:2170 --vo $1 --query "ServiceType=SRM" --attrs "ServiceEndpoint"|grep $host|cut -d" " -f3|head -1;`
   
VOInfoPath=`lcg-info --list-se --bdii ldap://$bdii:2170 --vo $1 --query "SE=$host" --attrs "VOInfoPath"|grep VOInfoPath|awk '{print $3}'`;

gsiftp=`ldapsearch -LLL -x -H ldap://$bdii:2170 -x -b o=grid "(&(GlueSEAccessProtocolType=gsiftp)(GlueChunkKey=GlueSEUniqueID=$host))" GlueSEAccessProtocolEndpoint | grep ^Glue |awk '{print $2}'| tail -1`;

details=`ldapsearch -LLL -h lcg-bdii.cern.ch -p 2170 -x -b o=grid "(&(objectClass=GlueSE)(GlueSEUniqueID=$host))" GlueSEImplementationName GlueSEImplementationVersion|grep ^Glue`;

echo -e "Host: $host\nSRMServiceEndpoint: $ServiceEndpoint\nVOInfoPath: $VOInfoPath\nGlueSEAccessProtocolEndpoint: $gsiftp\n$details\n";

done;


-bash-3.2$ ./go.sh biomed

Host: dcache-se-desy.desy.de
SRMServiceEndpoint: httpg://dcache-se-desy.desy.de:8443/srm/managerv2
VOInfoPath: /pnfs/desy.de/biomed
GlueSEAccessProtocolEndpoint: gsiftp://dcache-door-desy09.desy.de:2811
GlueSEImplementationVersion: 1.9.12-12 (ns=Chimera)
GlueSEImplementationName: dCache

Host: dc2-grid-64.brunel.ac.uk
SRMServiceEndpoint: httpg://dc2-grid-64.brunel.ac.uk:8446/srm/managerv2
VOInfoPath: /dpm/brunel.ac.uk/home/biomed
GlueSEAccessProtocolEndpoint: 
GlueSEImplementationVersion: 1.8.6
GlueSEImplementationName: DPM

Host: grid2.fe.infn.it
SRMServiceEndpoint: httpg://grid2.fe.infn.it:8444/srm/managerv2
VOInfoPath: /biomed
GlueSEAccessProtocolEndpoint: 
GlueSEImplementationVersion: 1.10.0
GlueSEImplementationName: StoRM

<...>

Extra step 2. Testing GridFTP endpoint for DPM or StoRM:


-bash-3.2$ touch test
-bash-3.2$ lcg-cp file://$PWD/test srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
-bash-3.2$ lcg-gt srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test gsiftp
gsiftp://dc2-grid-pool-a4-02.brunel.ac.uk/dc2-grid-pool-a4-02.brunel.ac.uk:/data2/dpmfs/biomed/2013-05-23/test.29428758.0
d4f44efc-b0c2-4ab2-ba0d-1c80f758e25b
-bash-3.2$ lcg-del -l srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
 

Testing GridFTP endpoint:


-bash-3.2$ uberftp dc2-grid-pool-a4-02.brunel.ac.uk "ls /dpm/brunel.ac.uk/home/biomed"


5. Register the endpoints in Globus Online:

a) Go to http://www.globusonline.eu/signup, create an account with your VO name e.g. biomed. (Note that usernames can include only letters and numbers and "_".)
(Using the VO name as an account name will ensure that VO members can easily find the endpoints that are available for them.)

b) Upload your ssh public key via globusonline.eu -> "manage identities"

c) Now you can manage your account using globusonline.eu client


-bash-3.2$ ssh biomed@cli.globusonline.eu "help"

d) Register the endpoints in the Globus Online service.

Endpoints can be registered through the command line interface or the graphical portal interface of Globus Online. The command line interface allows the association of a default directory with the endpoint and this simplifies the use of the endpoint by VO members. The below example therefore shows the command line tool for endpoint registration. The MyProxy server becomes the default MyProxy, and can be replaced with other MyProxy by VO members during the activation of the endpoint.


-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-add BRUNEL_UK -p gsiftp://dc2-grid-64.brunel.ac.uk:2811"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --public BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --myproxy-server=px.grid.sara.nl BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --default-directory=/dpm/brunel.ac.uk/home/biomed/ BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-list -v BRUNEL_UK"
Name                    : biomed#BRUNEL_UK
Host(s)                 : gsiftp://dc2-grid-64.brunel.ac.uk:2811
Subject(s)              : 
Target Endpoint         : n/a
Default Directory       : /dpm/brunel.ac.uk/home/biomed/
Force Encrypted Transfer: No
Disable Verify          : No
MyProxy Server          : px.grid.sara.nl
MyProxy DN              : n/a
MyProxy OAuth Server    : n/a
Credential Status       : EXPIRED
Credential Expires      : 
Credential Subject      : 


VO Members

1. Generate credentials.

Generate a VOMS proxy that identifies you at the EGI storage sites you want to use ás endpoints for the file transfers. The proxy can be generated for example on a User Interface machine provided for you by your VO or by your institute.


-bash-3.2$ voms-proxy-init --voms biomed

Enter GRID pass phrase for this identity:
Contacting cclcgvomsli01.in2p3.fr:15000 [/O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr] "biomed"...
Remote VOMS server contacted succesfully.

Created proxy in /tmp/x509up_u507.

Your proxy is valid until Fri May 24 04:12:03 CEST 2013

Upload VOMS proxy to a MyProxy server. If your VO does not have any MyProxy server, then you can use the EGI catch-all MyProxy server. Further information about the EGI catch-all MyProxy server.


-bash-3.2$ myproxy-init -s px.grid.sara.nl
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Enter GRID pass phrase for this identity:
Creating proxy ........................................................................ Done
Proxy Verify OK
Your proxy is valid until: Thu May 30 16:15:47 2013
Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
Verifying - Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
A proxy valid for 168 hours (7.0 days) for user karolis now exists on px.grid.sara.nl.

ATTENTION: The password entered for MyProxy will be used to authenticate the user via GlobusOnline.eu and the user name is the one which is chosen by the MyProxy server for the user, in this case - "karolis".


2. Perform file transfers

a) Go to http://www.globusonline.eu/signup and create an account. (The account can be later associated with your EGI Single Sign-On account (EGI SSO). Attention: login with the EGI Single Sign On account is possible only if you have a valid VOMS proxy in the EGI catch-all MyProxy server.)

b) Go to "Start Transfer"

c) Find the transfer endpoints that are available for your VO by searching for your VO name in the endpoint field e.g. biomed

Gobiomed.png


e) Click "Authenticate"

What you can do with the endpoints?

  • Transfer files from your laptop using GlobusConnect client to a endpoint using Globus Online.
  • Transfer files from an endpoint to your laptop using GlobusConnect client and Globus Online.
  • Transfer files from an endpoint to another endpoint using Globus Online.

Please consult with the Globus Online documentations that are available at http://www.globusonline.eu.

Additional materials

The tests that have been carried out while creating this Cookbook are available at GO_testing