|
|
| Line 1: |
Line 1: |
| {| align="right"
| | The Globus Online service can be used with GridFTP servers of EGI that are registered in the GOCDB system. Using Globus Online with SRM services may cause damage to these storages and therefore is not recommended. |
| | __TOC__
| | If your EGI Virtual Organisation would like to use the Globus Online service, then the VO should request from its resource providers the setup of GridFTP servers that are registered in the EGI GOCDB. |
| |}
| | EGI.eu is in discussion with the DMP community about defining a safe configuration for DPM SRM servers that would allow the mixed use of DPM storages with SRM and Globus Online. |
| | | For further information please email the EGI.eu User Community Support Team: ucst@egi.eu |
| == About ==
| |
| | |
| This page provides guidance for Virtual Organisations (VO) of the European Grid Infrastructure (EGI) on how to use the Globus Online service that is available at http://www.globusonline.eu. The services provides robust and easy to use file transfer capabilities for EGI users. Globus Online manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status. The Cookbook demonstrates how SRM storages of EGI could be used as endpoints for file transfers in Globus Online.
| |
| | |
| The Cookbook consists of two parts:
| |
| * The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as endpoints that VO members can find and use for file tranfers. Although these steps could be performed by any member of a VO, we assume that for most VOs the VO Manager is the most suitable person to complete the steps, because the VO Managers have sufficient knowledge on storage sites of the VO and about using the BDII information system to obtain detailed information about these storages.
| |
| * The second part provides step-by-step instructions for VO members on how to use storage endpoints in Globus Online. These steps can be carried out by any VO member.
| |
| | |
| The "lsgrid" VO is used as an example in the Cookbook to demonstrate Globus Online usage.
| |
| | |
| '''Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.'''
| |
| | |
| == VO Managers ==
| |
| | |
| All commands are performed from the command line of a gLite User Interface (UI). The "lsgrid" VO is used as an example.
| |
| | |
| 1. Find storage elements available for your VO:
| |
| | |
| <pre>
| |
| -bash-3.2$ lcg-infosites --is bdii.grid.sara.nl --vo lsgrid se
| |
| Avail Space(kB) Used Space(kB) Type SE
| |
| ------------------------------------------
| |
| < output truncated >
| |
| 717032688 382478939 SRM se.grid.rug.nl
| |
| 17542103037 460842863 SRM se.lsg.hubrecht.eu
| |
| 16560797796 1442148104 SRM se.lsg.psy.vu.nl
| |
| 58680692895 n.a SRM srm.grid.sara.nl
| |
| 14994733204 5266796 SRM srm.grid.sara.nl
| |
| n.a n.a SRM srm.grid.sara.nl
| |
| </pre>
| |
| | |
| [[GO_testing|Our tests with Globus Online]] were successful with DMP and dCache type of SRM storages, however the same process may work with some Storm storages (Storm is the third SRM storage type in use in EGI). With this query you can find out the type of the SRM storage that support your VO (GlueSEUniqueID=<SE_hostname>):
| |
| | |
| <pre>
| |
| -bash-3.2$ ldapsearch -LLL -h bdii.grid.sara.nl -p 2170 -x -b mds-vo-name=local,o=grid '(&(objectClass=GlueSE)(GlueSEUniqueID=se.grid.rug.nl))' GlueSEImplementationName
| |
| GlueSEImplementationName: dCache
| |
| | |
| -bash-3.2$ ldapsearch -LLL -h bdii.grid.sara.nl -p 2170 -x -b mds-vo-name=local,o=grid '(&(objectClass=GlueSE)(GlueSEUniqueID=se.lsg.hubrecht.eu))' GlueSEImplementationName
| |
| GlueSEImplementationName: DPM
| |
| </pre>
| |
| | |
| 2. Find the SRM endpoint for chosen storage element:
| |
| | |
| <pre>
| |
| -bash-3.2$ lcg-info --list-service --bdii ldap://bdii.grid.sara.nl:2170 --vo lsgrid --query 'ServiceType=SRM' --attrs 'ServiceEndpoint'| grep srm.grid.sara.nl
| |
| - Service: httpg://srm.grid.sara.nl:8443/srm/managerv2
| |
| - ServiceEndpoint httpg://srm.grid.sara.nl:8443/srm/managerv2
| |
| </pre>
| |
| | |
| 3. Find your VO home directory path for the SRM endpoint:
| |
| | |
| (This is the path of the directory where VO users can write files.)
| |
| | |
| <pre>
| |
| lcg-info --list-se --bdii ldap://bdii.grid.sara.nl:2170 --vo lsgrid --query 'SE=srm.grid.sara.nl' --attrs 'VOInfoPath'
| |
| - SE: srm.grid.sara.nl
| |
| - VOInfoPath /pnfs/grid.sara.nl/data/lsgrid
| |
| </pre>
| |
| | |
| 4. Find the GridFTP endpoint that belongs to this SRM endpoint with attempting copying a file:
| |
| | |
| <pre>
| |
| -bash-3.2$ touch test
| |
| -bash-3.2$ srmcp -2 file:///test srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/lsgrid/test -debug
| |
| Storage Resource Manager (SRM) Client version 2.1.5-16
| |
| < output truncated >
| |
| copying CopyJob, source = file:///test destination = gsiftp://bee51.grid.sara.nl:2811/pnfs/grid.sara.nl/data/lsgrid/test
| |
| execution of CopyJob, source = file:///test destination = gsiftp://bee51.grid.sara.nl:2811/pnfs/grid.sara.nl/data/lsgrid/test completed
| |
| < output truncated >
| |
| </pre>
| |
| | |
| From this line "copying CopyJob, source = file:///test destination = gsiftp://bee51.grid.sara.nl:2811/pnfs/grid.sara.nl/data/lsgrid/test" in the log we see that there is another host (bee51.grid.sara.nl) used by SRM to transfer files via GridFTP. This host (bee51.grid.sara.nl) and must be used at GlobusOnline.eu.
| |
| | |
| 5. Go to http://www.globusonline.eu/
| |
| | |
| a) Create an account with your VO name e.g. lsgrid. (Note that usernames can include only letters and numbers. The name of your VO may need to be altered!)<br>
| |
| b) Go to "Manage Endpoints" and click "Add Endpoint".<br>
| |
| c) Filling the endpoint details:<br>
| |
| | |
| <pre>
| |
| Name lsgrid# < enter name of the Storage > (e.g. SARA)
| |
| Short Description: < enter description > (add VO default path for this storage, e.g. /pnfs/grid.sara.nl/data/lsgrid/)
| |
| Server Address gsiftp:// < storage hostname with GridFTP access > (e.g. bee51.grid.sara.nl)
| |
| Visible To: Anyone (public)
| |
|
| |
| Set the Identity Provider to be used with this endpoint.
| |
| Type: MyProxy
| |
| MyProxy Hostname: < myproxy server for your VO > (e.g. px.grid.sara.nl)
| |
| </pre>
| |
| | |
| == VO Members ==
| |
| | |
| Commands are performed from the command line of a gLite User Interface (UI). The "lsgrid" VO is used as an example.
| |
| | |
| 1. Generating credentials.
| |
| | |
| Generate VOMS proxy:
| |
| <pre>
| |
| -bash-3.2$ voms-proxy-init --voms lsgrid
| |
| Enter GRID pass phrase:
| |
| Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
| |
| Creating temporary proxy .............................................................................................................. Done
| |
| Contacting voms.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr] "lsgrid" Done
| |
| Creating proxy ............................................................ Done
| |
| Your proxy is valid until Fri Nov 2 02:16:10 2012
| |
| </pre>
| |
| | |
| Upload VOMS proxy to MyProxy.
| |
| <pre>
| |
| -bash-3.2$ myproxy-init --voms lsgrid
| |
| Enter GRID pass phrase:
| |
| Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
| |
| Creating temporary proxy ............................. Done
| |
| Contacting voms.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr] "lsgrid" Done
| |
| Creating proxy ........................................... Done
| |
| Your proxy is valid until Thu Nov 8 14:17:47 2012
| |
| verify OK
| |
| Enter MyProxy pass phrase:
| |
| Verifying - Enter MyProxy pass phrase:
| |
| A proxy valid for 168 hours (7.0 days) for user karolise now exists on px.grid.sara.nl.
| |
| </pre>
| |
| | |
| 2. Go to to http://www.globusonline.eu/
| |
| | |
| a) Create account for yourself<br>
| |
| b) Go to "Manage Endpoints" and click "View All".<br>
| |
| c) Search for your VO name e.g. lsgrid<br>
| |
| | |
| You could see something like:
| |
| <pre>
| |
| lsgrid#SARA activate
| |
| </pre>
| |
| | |
| | |
| d) Click on "activate"<br>
| |
| | |
| Fill in the details for your credentials to retrieve.
| |
| <pre>
| |
| MyProxy Server: < myproxy server you have credentials at > (e.g. px.grid.sara.nl)
| |
| Username: < your user on a myproxy machine > (e.g. karolise)
| |
| Passphrase: < you chosen password > (e.g. You used to Enter MyProxy pass phrase in step 1.)
| |
| </pre>
| |
| | |
| e) Click "Authenticate"<br>
| |
| | |
| f) Go to "Start Transfer"
| |
| | |
| Choose your endpoint which is: lsgrid#SARA<br>
| |
| Look for your VO path directory in the endpoint description: /pnfs/grid.sara.nl/data/lsgrid/<br>
| |
| | |
| That is it. Now using Globus Online you can access Storage Element in EGI which is used also by SRM and FTS.<br>
| |
| | |
| What you can do ? <br>
| |
| | |
| *Transfer files from your laptop using GlobusConnect client to SRM storage endpoint in EGI using GO.<br>
| |
| *Transfer files from SRM endpoint in EGI using GO to your laptop using GlobusConnect client.<br>
| |
| *Transfer files from one SRM endpoint in EGI to another SRM endpoint in EGI or to any other Infrastructure using GO and receive nice notifications by email<br>
| |
| | |
| == Additional materials ==
| |
| | |
| Tests have been carried out while creating this Cookbook: [[GO_testing]]
| |
The wiki is deprecated and due to be decommissioned by the end of September 2022.