HOWTO12 Globus Online cookbook for EGI VOs

About

This page provides guidance for Virtual Organisations (VO) of the European Grid Infrastructure (EGI) on how to use the Globus Online service that is available at http://www.globusonline.eu. The services provides robust and easy to use file transfer capabilities for EGI users. Globus Online manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status. The Cookbook demonstrates how SRM storages of EGI could be used as endpoints for file transfers in Globus Online.

The Cookbook consists of two parts:

• The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as endpoints that VO members can find and use for file tranfers. Although these steps could be performed by any member of a VO, we assume that for most VOs the VO Manager is the most suitable person to complete the steps, because the VO Managers have sufficient knowledge on storage sites of the VO and about using the BDII information system to obtain detailed information about these storages.

• The second part provides step-by-step instructions for VO members on how to use storage endpoints in Globus Online. These steps can be carried out by any VO member.

The "biomed" VO is used as an example in the Cookbook to demonstrate Globus Online usage.

Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.

VO Managers

1. Get all storage managers (Glue 2.0):

-bash-3.2$ ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=glue '(&(objectclass=GLUE2StorageService))' |perl -p00e 's/\r?\n //g'|grep ^dn:|cut -d" " -f2 > GLUE2StorageServiceDN

2. Filter out only storage managers for your VO (Glue 2.0):

-bash-3.2$ for i in `cat GLUE2StorageServiceDN`;do ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" "(&(objectClass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:biomed))" GLUE2PolicyRule| grep -q GLUE2PolicyRule && echo $i;done > GLUE2AccessPolicyDN

3. Get all GridFTP endpoints for your VO (Glue 2.0):

-bash-3.2$ for i in `cat GLUE2AccessPolicyDN`; do ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" "(&(objectClass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp))" GLUE2EndpointURL | grep ^GLUE |tail -n 1;done
GLUE2EndpointURL: gsiftp://gaeds023.ciemat.es:2811
GLUE2EndpointURL: gsiftp://se001.ipp.acad.bg:2811
GLUE2EndpointURL: gsiftp://juliet.zih.tu-dresden.de:2811
GLUE2EndpointURL: gsiftp://sedsk58.grid.hep.ph.ic.ac.uk:2811

ATTENTION: you might not get all the GridFTP endpoints, because at the moment only dCache is publishing them to information system. Currently there several discussions ongoing about what storage managers must publish to information system and make easier way of querying the information for DPM and StoRM. REF: https://ggus.eu/tech/ticket_show.php?ticket=94264.

Additional steps for finding the GridFTP endpoints for DPM and StoRM:

Extra step 1. Example script to get the needed info:

#!/bin/bash

bdii="lcg-bdii.cern.ch";

lcg-infosites --is $bdii --vo biomed se |awk '{print $4}'|sort|uniq> biomed_srm_hosts

for host in `cat biomed_srm_hosts`; do

ServiceEndpoint=`lcg-info --list-service --bdii ldap://lcg-bdii.cern.ch:2170 --vo biomed --query "ServiceType=SRM" --attrs "ServiceEndpoint"|grep $host|cut -d" " -f3|head -1;`
   
VOInfoPath=`lcg-info --list-se --bdii ldap://$bdii:2170 --vo biomed --query "SE=$host" --attrs "VOInfoPath"|grep VOInfoPath|awk '{print $3}'`;

gsiftp=`ldapsearch -LLL -x -H ldap://$bdii:2170 -x -b o=grid "(&(GlueSEAccessProtocolType=gsiftp)(GlueChunkKey=GlueSEUniqueID=$host))" GlueSEAccessProtocolEndpoint | grep ^Glue |awk '{print $2}'| tail -1`;

details=`ldapsearch -LLL -h lcg-bdii.cern.ch -p 2170 -x -b o=grid "(&(objectClass=GlueSE)(GlueSEUniqueID=$host))" GlueSEImplementationName GlueSEImplementationVersion|grep ^Glue`;

echo -e "Host: $host\nSRMServiceEndpoint: $ServiceEndpoint\nVOInfoPath: $VOInfoPath\nGlueSEAccessProtocolEndpoint: $gsiftp\n$details\n";

done;

-bash-3.2$ ./go.sh

Host: dcache-se-desy.desy.de
SRMServiceEndpoint: httpg://dcache-se-desy.desy.de:8443/srm/managerv2
VOInfoPath: /pnfs/desy.de/biomed
GlueSEAccessProtocolEndpoint: gsiftp://dcache-door-desy09.desy.de:2811
GlueSEImplementationVersion: 1.9.12-12 (ns=Chimera)
GlueSEImplementationName: dCache

Host: dc2-grid-64.brunel.ac.uk
SRMServiceEndpoint: httpg://dc2-grid-64.brunel.ac.uk:8446/srm/managerv2
VOInfoPath: /dpm/brunel.ac.uk/home/biomed
GlueSEAccessProtocolEndpoint: 
GlueSEImplementationVersion: 1.8.6
GlueSEImplementationName: DPM

Host: grid2.fe.infn.it
SRMServiceEndpoint: httpg://grid2.fe.infn.it:8444/srm/managerv2
VOInfoPath: /biomed
GlueSEAccessProtocolEndpoint: 
GlueSEImplementationVersion: 1.10.0
GlueSEImplementationName: StoRM

<...>

Extra step 2. Finding GridFTP endpoint for DPM or StoRM:

-bash-3.2$ touch test
-bash-3.2$ lcg-cp file://$PWD/test srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
-bash-3.2$ lcg-gt srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test gsiftp
gsiftp://dc2-grid-pool-a4-02.brunel.ac.uk/dc2-grid-pool-a4-02.brunel.ac.uk:/data2/dpmfs/biomed/2013-05-23/test.29428758.0
d4f44efc-b0c2-4ab2-ba0d-1c80f758e25b
-bash-3.2$ lcg-del -l srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
 

Making sure that GridFTP endpoint is accessible:

-bash-3.2$ uberftp dc2-grid-pool-a4-02.brunel.ac.uk "ls /dpm/brunel.ac.uk/home/biomed"

4. Go to http://www.globusonline.eu/

a) Create an account with your VO name e.g. biomed. (Note that usernames can include only letters and numbers.)

b) Upload your ssh public key via globusonline.eu -> "manage identities"

c) Now you can manage your account using globusonline.eu client

-bash-3.2$ ssh ekarolis@cli.globusonline.eu "help"

d) Lets create the endpoint with the all needed information we have

-bash-3.2$ ssh ekarolis@cli.globusonline.eu "endpoint-add BIOMED_BRUNEL_UK -p gsiftp://dc2-grid-64.brunel.ac.uk:2811"
-bash-3.2$ ssh ekarolis@cli.globusonline.eu "endpoint-modify --public BIOMED_BRUNEL_UK"
-bash-3.2$ ssh ekarolis@cli.globusonline.eu "endpoint-modify --myproxy-server=px.grid.sara.nl BIOMED_BRUNEL_UK"
-bash-3.2$ ssh ekarolis@cli.globusonline.eu "endpoint-modify --default-directory=/dpm/brunel.ac.uk/home/biomed/ BIOMED_BRUNEL_UK"
-bash-3.2$ ssh ekarolis@cli.globusonline.eu "endpoint-list -v BIOMED_BRUNEL_UK"
Name                    : ekarolis#BIOMED_BRUNEL_UK
Host(s)                 : gsiftp://dc2-grid-64.brunel.ac.uk:2811
Subject(s)              : 
Target Endpoint         : n/a
Default Directory       : /dpm/brunel.ac.uk/home/biomed/
Force Encrypted Transfer: No
Disable Verify          : No
MyProxy Server          : px.grid.sara.nl
MyProxy DN              : n/a
MyProxy OAuth Server    : n/a
Credential Status       : EXPIRED
Credential Expires      : 
Credential Subject      : 

VO Members

Commands are performed from the command line of a gLite User Interface (UI). The "biomed" VO is used as an example.

1. Generating credentials.

Generate VOMS proxy:

-bash-3.2$ voms-proxy-init --voms biomed

Enter GRID pass phrase:

Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis

Creating temporary proxy .............................................................................................................. Done

Contacting  voms.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr] "biomed" Done

Creating proxy ............................................................ Done

Your proxy is valid until Fri Nov  2 02:16:10 2012

Upload VOMS proxy to MyProxy.

-bash-3.2$ myproxy-init --voms biomed

Enter GRID pass phrase:

Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis

Creating temporary proxy ............................. Done

Contacting  voms.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr] "biomed" Done

Creating proxy ........................................... Done

Your proxy is valid until Thu Nov  8 14:17:47 2012

verify OK

Enter MyProxy pass phrase:

Verifying - Enter MyProxy pass phrase:

A proxy valid for 168 hours (7.0 days) for user karolise now exists on px.grid.sara.nl.

2. Go to to http://www.globusonline.eu/

a) Create account for yourself

b) Go to "Manage Endpoints" and click "View All".

c) Search for your VO name e.g. biomed

You could see something like:

biomed#SARA     activate

d) Click on "activate"

Fill in the details for your credentials to retrieve.

MyProxy Server: < myproxy server you have credentials at > (e.g. px.grid.sara.nl)

Username: < your user on a myproxy machine > (e.g. karolise)

Passphrase: < you chosen password > (e.g. You used to Enter MyProxy pass phrase in step 1.)

e) Click "Authenticate"

f) Go to "Start Transfer"

Choose your endpoint which is: biomed#SARA

Look for your VO path directory in the endpoint description: /pnfs/grid.sara.nl/data/biomed/

That is it. Now using Globus Online you can access Storage Element in EGI which is used also by SRM and FTS.

What you can do ?

• Transfer files from your laptop using GlobusConnect client to SRM storage endpoint in EGI using GO.
  

• Transfer files from SRM endpoint in EGI using GO to your laptop using GlobusConnect client.
  

• Transfer files from one SRM endpoint in EGI to another SRM endpoint in EGI or to any other Infrastructure using GO and receive nice notifications by email
  

Additional materials

Tests have been carried out while creating this Cookbook: GO_testing-->

<a _fcknotitle="true" href="Category:Community_Engagement">Community_Engagement</a>