HOWTO12 Globus Online cookbook for EGI VOs

About

This page provides guidance for Virtual Organisations (VO) of the European Grid Infrastructure (EGI) on how to use the Globus Online service that is available at http://www.globusonline.eu/. The services makes robust and easy to use file transfer capabilities available for EGI users. Globus Online manages file transfer for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status.

The Cookbook consists of two parts:

• The first part provides step-by-step instructions for VO Managers on how to register storage services of their VO in Globus Online in such a way, that these services appear as endpoints for other VO members. These steps need to be completed first. Although these steps can be performed by any member of the VO, we assume that for most of the VOs the VO Managers are the most suitable persons for this task, because they have enough knowledge to obtain information about the storage sites of the VO from the BDII information system.
• The second part provides step-by-step instructions for VO members on how to use storage endpoints in Globus Online for file transfer. These steps can be carried out by any user of the VO.

The "lsgrid" EGI VO is used as an example in the Cookbook to demonstrate the steps. Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: [ucst@egi.eu.

VO Managers

All commands are performed from the command line of a gLite User Interface (UI). The "lsgrid" VO is used as an example.

1. Find storage elements available for your VO:

(Our tests were successfull with DMP and dCache storages, however ther type of SRM storages may also work. This depends on the configurations of the these storages)

-bash-3.2$ lcg-infosites --is bdii.grid.sara.nl --vo lsgrid se 
 Avail Space(kB)  Used Space(kB)  Type 	SE
------------------------------------------
< output truncated >
       717032688       382478939  SRM  	se.grid.rug.nl
     17542103037       460842863  SRM  	se.lsg.hubrecht.eu
     16560797796      1442148104  SRM  	se.lsg.psy.vu.nl
     58680692895             n.a  SRM  	srm.grid.sara.nl
     14994733204         5266796  SRM  	srm.grid.sara.nl
             n.a             n.a  SRM  	srm.grid.sara.nl

2. Find the SRM endpoint for chosen storage element:

-bash-3.2$ lcg-info --list-service --bdii ldap://bdii.grid.sara.nl:2170 --vo lsgrid --query 'ServiceType=SRM' --attrs 'ServiceEndpoint'| grep srm.grid.sara.nl
- Service: httpg://srm.grid.sara.nl:8443/srm/managerv2
  - ServiceEndpoint     httpg://srm.grid.sara.nl:8443/srm/managerv2

3. Find your VO home directory path for the SRM endpoint:

(This is the path of the directory where VO users can write files.)

lcg-info --list-se --bdii ldap://bdii.grid.sara.nl:2170 --vo lsgrid --query 'SE=srm.grid.sara.nl' --attrs 'VOInfoPath'
- SE: srm.grid.sara.nl
  - VOInfoPath          /pnfs/grid.sara.nl/data/lsgrid

4. Find the GridFTP endpoint that belongs to this SRM endpoint with attempting copying a file:

-bash-3.2$ touch test
-bash-3.2$ srmcp -2 file:///test srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/lsgrid/test -debug
Storage Resource Manager (SRM) Client version 2.1.5-16
< output truncated >
copying CopyJob, source = file:///test destination = gsiftp://bee51.grid.sara.nl:2811/pnfs/grid.sara.nl/data/lsgrid/test
execution of CopyJob, source = file:///test destination = gsiftp://bee51.grid.sara.nl:2811/pnfs/grid.sara.nl/data/lsgrid/test completed
< output truncated >

From this line "copying CopyJob, source = file:///test destination = gsiftp://bee51.grid.sara.nl:2811/pnfs/grid.sara.nl/data/lsgrid/test" in the log we see that there is another host (bee51.grid.sara.nl) used by SRM to transfer files via GridFTP. This host (bee51.grid.sara.nl) and must be used at GlobusOnline.eu.

5. Go to http://www.globusonline.eu/

a) Create an account with your VO name e.g. lsgrid. (Note that usernames can include only letters and numbers. The name of your VO may need to be altered!)
b) Go to "Manage Endpoints" and click "Add Endpoint".
c) Filling the endpoint details:

Name lsgrid# < enter name of the Storage >  (e.g. SARA)
Short Description: < enter description > (add VO default path for this storage, e.g. /pnfs/grid.sara.nl/data/lsgrid/)
Server Address gsiftp:// < storage hostname with GridFTP access > (e.g. bee51.grid.sara.nl)
Visible To: Anyone (public)
       
Set the Identity Provider to be used with this endpoint. 
Type: MyProxy
MyProxy Hostname: < myproxy server for your VO > (e.g. px.grid.sara.nl)

VO Members

Commands are performed from the command line of a gLite User Interface (UI). The "lsgrid" VO is used as an example.

1. Generating credentials.

Generate VOMS proxy:

-bash-3.2$ voms-proxy-init --voms lsgrid
Enter GRID pass phrase:
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Creating temporary proxy .............................................................................................................. Done
Contacting  voms.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr] "lsgrid" Done
Creating proxy ............................................................ Done
Your proxy is valid until Fri Nov  2 02:16:10 2012

Upload VOMS proxy to MyProxy.

-bash-3.2$ myproxy-init --voms lsgrid
Enter GRID pass phrase:
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Creating temporary proxy ............................. Done
Contacting  voms.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr] "lsgrid" Done
Creating proxy ........................................... Done
Your proxy is valid until Thu Nov  8 14:17:47 2012
verify OK
Enter MyProxy pass phrase:
Verifying - Enter MyProxy pass phrase:
A proxy valid for 168 hours (7.0 days) for user karolise now exists on px.grid.sara.nl.

2. Go to GlobusOnline.eu.

a) Create account for yourself
b) Go to "Manage Endpoints" and click "View All".
c) Search for your VO name e.g. lsgrid

You could see something like:

lsgrid#SARA     activate

d) Click on "activate"

Fill in the details for your credentials to retrieve.

MyProxy Server: < myproxy server you have credentials at > (e.g. px.grid.sara.nl)
Username: < your user on a myproxy machine > (e.g. karolise) 
Passphrase: < you chosen password > (e.g. You used to Enter MyProxy pass phrase in step 1.)

e) Click "Authenticate"

f) Go to "Start Transfer"

Choose your endpoint which is: lsgrid#SARA
Look for your VO path directory in the endpoint description: /pnfs/grid.sara.nl/data/lsgrid/

That is it. Now using Globus Online you can access Storage Element in EGI which is used also by SRM and FTS.

What you can do ?

• Transfer files from your laptop using GlobusConnect client to SRM storage endpoint in EGI using GO.
  
• Transfer files from SRM endpoint in EGI using GO to your laptop using GlobusConnect client.
  
• Transfer files from one SRM endpoint in EGI to another SRM endpoint in EGI or to any other Infrastructure using GO and receive nice notifications by email
  

Additional materials

Tests have been carried out while creating this Cookbook: GO_testing