Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "HOWTO12 Globus Online cookbook for EGI VOs"

From EGIWiki
Jump to navigation Jump to search
(47 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<p><br />
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}
</p>
[[Category:Operations Manuals]]
<table align="right">


<tr>
The '''Globus Online cookbook''' describes how the 'SRM type' storage services that are federated into EGI can be used as endpoints of file transfers managed by Globus Online. The "biomed" VO is used as an example in the Cookbook to demonstrate EGI storage usage, but the steps are the same for other EGI VOs as well.  
<td> __TOC__
</td></tr></table>
<h2> About </h2>
<p>This page provides guidance for Virtual Organisations (VO) of the European Grid Infrastructure (EGI) on how to use the Globus Online service that is available at http://www.globusonline.eu. GlobusOnline.eu provides robust and easy to use file transfer capabilities for EGI users. The service manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status. This Cookbook describes how the 'SRM type' storage services that are federated into EGI can be used as endpoints of file transfers managed by Globus Online. The "biomed" VO is used as an example in the Cookbook to demonstrate EGI storage usage, but the steps are the same for other EGI VOs as well.
<br><br>
The Cookbook consists of two parts:
# VO Managers: The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as transfer endpoints for VO members. This registration could be performed by any member of a VO, however for most VOs the VO Manager is the most suitable person to complete this step because the VO Manager has sufficient knowledge on storage sites that support the VO and about the BDII information system where detailed information about the storages is recorded.
# VO Members: The second part provides step-by-step instructions for VO members on how to use VO storage endpoints in Globus Online. This part is relevant for any meber of any EGI VO. The list of EGI VOs and information on joining these VOs is available at http://operations-portal.egi.eu/vo.  


The cookbook was prepared as a guide for EGI Virtual Organisations (VOs) on how to use the Globus Online service that is available at http://www.globusonline.eu. GlobusOnline.eu provides robust and easy to use file transfer capabilities for EGI users. The service manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status.


</p><p><b>This cookbook has been prepared by the EGI.eu User Community Support Team in consultation with representatives of EGI Operations, storage technology and information system developer groups. Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.</b></p>
The Cookbook consists of two parts:  


<h2>VO Managers </h2>
#[[Globus Online cookbook for EGI VOs#VO_Managers|For VO Managers ]]: The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as transfer endpoints for VO members. This registration could be performed by any member of a VO, however for most VOs the VO Manager is the most suitable person to complete this step because the VO Manager has sufficient knowledge on storage sites that support the VO and about the BDII information system where detailed information about the storages is recorded.
1. Get all storage managers (Glue 2.0):
#[[Globus Online cookbook for EGI VOs#VO_Members|For VO Members (researchers) ]]: The second part provides step-by-step instructions for VO members on how to use VO storage endpoints in Globus Online. This part is relevant for any meber of any EGI VO. The list of EGI VOs and information on joining these VOs is available at http://operations-portal.egi.eu/vo.


<pre>
''Important note: the http://www.globusonline.eu server is hosted in the US, but the files that the service moves between EGI sites do not leave Europe. The service orchestrates file copies with the GridFTP third party transfer, so files are copied directly between the EGI endpoints. ''


-bash-3.2$ ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=glue '(&amp;(objectclass=GLUE2StorageService))' \
This cookbook has been prepared by the EGI.eu User Community Support Team in consultation with representatives of EGI Operations, storage technology and information system developer groups. Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.  
-bash-3.2$ |perl -p00e 's/\r?\n //g'|grep ^dn:|cut -d&quot; &quot; -f2 &gt; GLUE2StorageServiceDN


</pre>
== VO Managers  ==
<p>2. Filter out only storage managers for biomed VO (Glue 2.0):</p>
<pre>


-bash-3.2$ for i in `cat GLUE2StorageServiceDN`;do \
1. Get all endpoints suporting gsiftp for biomed&nbsp;VO (Glue 2.0):
-bash-3.2$ ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b &quot;$i&quot; &quot;(&amp;(objectClass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:biomed))&quot; GLUE2PolicyRule \
<pre>-bash-3.2$ ldapsearch -LLL -x -h lcg-bdii.cern.ch -p 2170 -b o=glue '(&amp;(objectclass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:biomed))' \
-bash-3.2$ | grep -q GLUE2PolicyRule &amp;&amp; echo $i;done &gt; GLUE2AccessPolicyDN
-bash-3.2$ GLUE2AccessPolicyEndpointForeignKey | perl -p00e 's/\r?\n //g' | grep GLUE2AccessPolicyEndpointForeignKey | sort | uniq | awk '{ print $2 }' |grep -i gsiftp &gt; APEndpoints


</pre>
</pre>  
<p><br />
2. Get all GridFTP&nbsp;(gsiftp) endpoints for biomed VO (Glue 2.0):  
3. Get all GridFTP endpoints for biomed VO (Glue 2.0):
<pre>-bash-3.2$ for i in `cat APEndpoints`;do ldapsearch -LLL -x -h lcg-bdii.cern.ch -p 2170 -b o=glue \  
</p>
-bash-3.2$ '(&amp;(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID='$i'))' \
<pre>
-bash-3.2$ GLUE2EndpointURL GLUE2EndpointImplementationName GLUE2ENdpointImplementationVersion GLUE2EndpointInterfaceName | grep ^GLUE&nbsp;;echo "-------";done
 
</pre>  
-bash-3.2$ for i in `cat GLUE2AccessPolicyDN`; do ldapsearch -LLL -x -H ldap://lcg-bdii.cern.ch:2170 -x -b "$i" \
3. Find myproxy server for biomed VO:  
-bash-3.2$ "(&(objectClass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp))" GLUE2EndpointURL \
<pre>-bash-3.2$ lcg-infosites --is lcg-bdii.cern.ch --vo biomed myproxy
-bash-3.2$ | grep ^GLUE |tail -n 1; echo StorageServiceID: $i|cut -d"," -f1|sed 's/GLUE2ServiceID=//g';echo "-------";done
 
GLUE2EndpointURL: gsiftp://gaeds020.ciemat.es:2811
StorageServiceID: glue:srm.ciemat.es/data
-------
GLUE2EndpointURL: gsiftp://se001.ipp.acad.bg:2811
StorageServiceID: glue:se001.ipp.acad.bg/data
-------
GLUE2EndpointURL: gsiftp://juliet.zih.tu-dresden.de:2811
StorageServiceID: glue:ophelia.zih.tu-dresden.de/data
-------
GLUE2EndpointURL: gsiftp://sedsk58.grid.hep.ph.ic.ac.uk:2811
StorageServiceID: glue:gfe02.grid.hep.ph.ic.ac.uk/data
-------
</pre>
 
<p><b>ATTENTION</b>: you may not get all the GridFTP endpoints of your VO with the above query because at the moment (June 2013) only dCache type storage elements publish information in the EGI BDII with Glue 2. If your VO includes DPM and/or StoRM type storage elements too, then please complete the '''Additional steps for finding the GridFTP endpoints for DPM and StoRM (Glue 1.3):'''. (The DPM community is working on a new feature to publish Gridftp endpoints with Glue 2 (REF: https://its.cern.ch/jira/browse/LCGDM-1083).)
</p>
 
4. Find myproxy server for biomed VO:
 
<pre>
 
-bash-3.2$ lcg-infosites --is lcg-bdii.cern.ch --vo biomed myproxy
myproxy://px.grid.sara.nl:7512/
myproxy://px.grid.sara.nl:7512/
myproxy.cern.ch:7512
myproxy.cern.ch:7512
Line 78: Line 43:
myproxy://lcgpx01.jinr.ru:7512/
myproxy://lcgpx01.jinr.ru:7512/
myproxy://lcgrbp01.gridpp.rl.ac.uk:7512/
myproxy://lcgrbp01.gridpp.rl.ac.uk:7512/
<...>
&lt;...&gt;


</pre>
</pre>
If your VO does not have a dedicated MyPRoxy server, then the catch-all MyProxy server of EGI can be used. This catch-all service is available from CESNET at myproxy.egi.eu.<br>  


If your VO does not have a dedicated MyPRoxy server, then the catch-all MyProxy server of EGI can be used. This catch-all service is available from CESNET at myproxy.egi.eu.
<br>


'''Additional steps for finding the GridFTP endpoints for DPM and StoRM (Glue 1.3):'''
Extras: example script to obtain GridFTP endpoint information with Glue 2.0. (Command line tools from emi-ui-3.0.0-1.el6.x86_64):  
<pre>#!/bin/bash


Extra step 1. Example script to obtain GridFTP endpoint information with Glue 1.3. (Command line tools from emi-ui-3.0.0-1.el6.x86_64):
bdii="top-bdii.cern.ch";


<pre>
#!/bin/bash


bdii="lcg-bdii.cern.ch";
ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&amp;(objectclass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:$1))" GLUE2AccessPolicyEndpointForeignKey | perl -p00e 's/\r?\n //g' | grep GLUE2AccessPolicyEndpointForeignKey |sort | uniq | awk '{ print $2 }' |grep -i gsiftp &gt; APEndpoints


lcg-infosites --is $bdii --vo $1 se |awk '{print $4}'|sort|uniq > $1


for host in `cat $1`; do
for i in `cat APEndpoints`;do  


ServiceEndpoint=`lcg-info --list-service --bdii ldap://lcg-bdii.cern.ch:2170 --vo $1 --query "ServiceType=SRM" --attrs "ServiceEndpoint"|grep $host|cut -d" " -f3|head -1;`
 
VOInfoPath=`lcg-info --list-se --bdii ldap://$bdii:2170 --vo $1 --query "SE=$host" --attrs "VOInfoPath"|grep VOInfoPath|awk '{print $3}'`;


gsiftp=`ldapsearch -LLL -x -H ldap://$bdii:2170 -x -b o=grid "(&(GlueSEAccessProtocolType=gsiftp)(GlueChunkKey=GlueSEUniqueID=$host))" GlueSEAccessProtocolEndpoint | grep ^Glue |awk '{print $2}'| tail -1`;
SEtype=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&amp;(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointImplementationName |grep ^GLUE2| cut -d" " -f2`;


details=`ldapsearch -LLL -h lcg-bdii.cern.ch -p 2170 -x -b o=grid "(&(objectClass=GlueSE)(GlueSEUniqueID=$host))" GlueSEImplementationName GlueSEImplementationVersion|grep ^Glue`;


echo -e "Host: $host\nSRMServiceEndpoint: $ServiceEndpoint\nVOInfoPath: $VOInfoPath\nGlueSEAccessProtocolEndpoint: $gsiftp\n$details\n";
if [ "$SEtype" = "DPM" ]; then


done;
SE=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&amp;(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointServiceForeignKey |grep ^GLUE2| cut -d" " -f2`;
 
fi


</pre>


<pre>
if [ "$SEtype" = "dCache" ]; then


-bash-3.2$ ./go.sh biomed
SE=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&amp;(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointServiceForeignKey |grep ^GLUE2| cut -d" " -f2 | cut -d":" -f2|sed 's/\/data//g'`;


Host: dcache-se-desy.desy.de
fi
SRMServiceEndpoint: httpg://dcache-se-desy.desy.de:8443/srm/managerv2
VOInfoPath: /pnfs/desy.de/biomed
GlueSEAccessProtocolEndpoint: gsiftp://dcache-door-desy09.desy.de:2811
GlueSEImplementationVersion: 1.9.12-12 (ns=Chimera)
GlueSEImplementationName: dCache


Host: dc2-grid-64.brunel.ac.uk
SRMServiceEndpoint: httpg://dc2-grid-64.brunel.ac.uk:8446/srm/managerv2
VOInfoPath: /dpm/brunel.ac.uk/home/biomed
GlueSEAccessProtocolEndpoint:
GlueSEImplementationVersion: 1.8.6
GlueSEImplementationName: DPM


Host: grid2.fe.infn.it
SRM=`lcg-info --list-service --bdii ldap://$bdii:2170 --vo $1 --query "ServiceType=SRM" --attrs "ServiceEndpoint"|grep $SE | cut -d" " -f3|head -1`;
SRMServiceEndpoint: httpg://grid2.fe.infn.it:8444/srm/managerv2
VOInfoPath: /biomed
GlueSEAccessProtocolEndpoint:
GlueSEImplementationVersion: 1.10.0
GlueSEImplementationName: StoRM


<...>
details=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&amp;(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointURL GLUE2EndpointImplementationName GLUE2ENdpointImplementationVersion | grep ^GLUE`;


</pre>
VOInfoPath=`lcg-info --list-se --bdii ldap://$bdii:2170 --vo biomed --query "SE=$SE" --attrs "VOInfoPath"|grep VOInfoPath|awk '{print $3}'`;


Extra step 2. Testing GridFTP endpoint for DPM or StoRM:
echo -e "$details\nVOInfoPath: $VOInfoPath\nHost: $SE\nSRM: $SRM\n";


<pre>
done;


-bash-3.2$ touch test
</pre> <pre>
-bash-3.2$ lcg-cp file://$PWD/test srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
-bash-3.2$ ./go.sh biomed
-bash-3.2$ lcg-gt srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test gsiftp
gsiftp://dc2-grid-pool-a4-02.brunel.ac.uk/dc2-grid-pool-a4-02.brunel.ac.uk:/data2/dpmfs/biomed/2013-05-23/test.29428758.0
d4f44efc-b0c2-4ab2-ba0d-1c80f758e25b
-bash-3.2$ lcg-del -l srm://dc2-grid-64.brunel.ac.uk:8446/dpm/brunel.ac.uk/home/biomed/test
</pre>


Testing GridFTP endpoint:


<pre>
GLUE2EndpointImplementationName: DPM
GLUE2EndpointURL: gsiftp://glite-se.scai.fraunhofer.de:2811
GLUE2EndpointImplementationVersion: 1.8.8
VOInfoPath: /dpm/scai.fraunhofer.de/home/biomed
Host: glite-se.scai.fraunhofer.de
SRM: httpg://glite-se.scai.fraunhofer.de:8446/srm/managerv2


-bash-3.2$ uberftp dc2-grid-pool-a4-02.brunel.ac.uk "ls /dpm/brunel.ac.uk/home/biomed"


</pre>


GLUE2EndpointImplementationVersion: 2.6.19
GLUE2EndpointURL: gsiftp://dcache-door-desy09.desy.de:2811
GLUE2EndpointImplementationName: dCache
VOInfoPath: /pnfs/desy.de/biomed
Host: dcache-se-desy.desy.de
SRM: httpg://dcache-se-desy.desy.de:8443/srm/managerv2


&lt;...&gt;


5. Register the endpoints in Globus Online:


a) Go to http://www.globusonline.eu/signup, create an account with your VO name e.g. biomed. (Note that usernames can include only letters and numbers and "_".)<br> (Using the VO name as an account name will ensure that VO members can easily find the endpoints that are available for them.)
</pre>
Testing access to GridFTP endpoint:  
<pre>-bash-3.2$ uberftp glite-se.scai.fraunhofer.de "ls /dpm/scai.fraunhofer.de/home/biomed"


b) Upload your ssh public key via globusonline.eu -> "manage identities"
</pre>
<br>  


c) Now you can manage your account using globusonline.eu client
4. Register the endpoints in Globus Online:


<pre>
a) Go to http://www.globusonline.eu/signup, create an account with your VO name e.g. biomed. (Note that usernames can include only letters and numbers and "_".)<br> (Using the VO name as an account name will ensure that VO members can easily find the endpoints that are available for them.)


-bash-3.2$ ssh biomed@cli.globusonline.eu "help"
b) Upload your ssh public key via globusonline.eu -&gt; "manage identities"  


</pre>
c) Now you can manage your account using globusonline.eu client
<pre>-bash-3.2$ ssh biomed@cli.globusonline.eu "help"


</pre>
d) Register the endpoints in the Globus Online service.  
d) Register the endpoints in the Globus Online service.  


Endpoints can be registered through the command line interface or the graphical portal interface of Globus Online. The command line interface allows the association of a default directory with the endpoint and this simplifies the use of the endpoint by VO members. The below example therefore shows the command line tool for endpoint registration. The MyProxy server becomes the default MyProxy, and can be replaced with other MyProxy by VO members during the activation of the endpoint.  
Endpoints can be registered through the command line interface or the graphical portal interface of Globus Online. The command line interface allows the association of a default directory with the endpoint and this simplifies the use of the endpoint by VO members. The below example therefore shows the command line tool for endpoint registration. The MyProxy server becomes the default MyProxy, and can be replaced with other MyProxy by VO members during the activation of the endpoint.  
<pre>-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-add fraunhofer_DE -p gsiftp://glite-se.scai.fraunhofer.de:2811"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --public fraunhofer_DE"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --myproxy-server=px.grid.sara.nl fraunhofer_DE"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --default-directory=/dpm/scai.fraunhofer.de/home/biomed fraunhofer_DE"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-list -v fraunhofer_DE"
Name                  &nbsp;: biomed#fraunhofer_DE
Host(s)                &nbsp;: gsiftp://glite-se.scai.fraunhofer.de:2811
Subject(s)            &nbsp;:
Target Endpoint        &nbsp;: n/a
Default Directory      &nbsp;: /dpm/scai.fraunhofer.de/home/biomed
Force Encrypted Transfer: No
Disable Verify        &nbsp;: No
MyProxy Server        &nbsp;: px.grid.sara.nl
MyProxy DN            &nbsp;: n/a
MyProxy OAuth Server  &nbsp;: n/a
Credential Status      &nbsp;: EXPIRED
Credential Expires    &nbsp;:
Credential Subject    &nbsp;:


<pre>
</pre>
<br>


-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-add BRUNEL_UK -p gsiftp://dc2-grid-64.brunel.ac.uk:2811"
== VO Members  ==
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --public BRUNEL_UK"
 
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --myproxy-server=px.grid.sara.nl BRUNEL_UK"
'''1. Generate a VOMS proxy and upload it into a MyProxy server.''' '
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --default-directory=/dpm/brunel.ac.uk/home/biomed/ BRUNEL_UK"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-list -v BRUNEL_UK"
Name                    : biomed#BRUNEL_UK
Host(s)                : gsiftp://dc2-grid-64.brunel.ac.uk:2811
Subject(s)              :
Target Endpoint        : n/a
Default Directory      : /dpm/brunel.ac.uk/home/biomed/
Force Encrypted Transfer: No
Disable Verify          : No
MyProxy Server          : px.grid.sara.nl
MyProxy DN              : n/a
MyProxy OAuth Server    : n/a
Credential Status      : EXPIRED
Credential Expires      :
Credential Subject      :


</pre>
There are two ways to do this:


a). With a graphical tool, such as GSISSH-Term.


== VO Members ==
OR


1. Generate credentials.
b). With the command line tools of the User Interface machine of your VO.  


Generate a VOMS proxy that identifies you at those EGI storage sites that you want to use for the file transfers. The proxy can be generated for example on a User Interface machine provided for you by your VO or institute.
The usage of GSISSH-Term for proxy management is explained on a [[MyProxy tool GUI|dedicated page]]. A usage of the command line tools is detailed below.  


<pre>
<br>  


-bash-3.2$ voms-proxy-init --voms biomed
Generating a VOMS proxy with the command line tools (You should have these installed on the User Interface machine of your VO):
<pre>-bash-3.2$ voms-proxy-init --voms biomed


Enter GRID pass phrase for this identity:
Enter GRID pass phrase for this identity:
Line 223: Line 181:
Your proxy is valid until Fri May 24 04:12:03 CEST 2013
Your proxy is valid until Fri May 24 04:12:03 CEST 2013


</pre>
</pre>  
 
'''VOMS proxy lifetime: by default voms proxy extension is generated for 12hours, some voms servers within EGI allow to have lifetime for 1 week (168hours), some allow up to 24hours, please consult your VO manager.'''
Upload VOMS proxy to a MyProxy server. If your VO does not have any MyProxy server, then you can use the EGI catch-all MyProxy server. (XXX Further information about the EGI MyProxy server...)
 
<pre>


-bash-3.2$ myproxy-init -s px.grid.sara.nl
<br> Upload VOMS proxy to a MyProxy server with the command line tool. Note: If your VO does not have any MyProxy server, then you can use the EGI catch-all MyProxy server. [[EGI Myproxy with OAuth|Further information about the EGI catch-all MyProxy server]].
<pre>-bash-3.2$ myproxy-init -l &lt;CHOOSE ANY USERNAME&gt; -s px.grid.sara.nl
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Enter GRID pass phrase for this identity:
Enter GRID pass phrase for this identity:
Line 235: Line 191:
Proxy Verify OK
Proxy Verify OK
Your proxy is valid until: Thu May 30 16:15:47 2013
Your proxy is valid until: Thu May 30 16:15:47 2013
Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
Enter MyProxy pass phrase: &lt;YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE&gt;
Verifying - Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
Verifying - Enter MyProxy pass phrase: &lt;YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE&gt;
A proxy valid for 168 hours (7.0 days) for user karolis now exists on px.grid.sara.nl.
A proxy valid for 168 hours (7.0 days) for user &lt;YOUR USERNAME SPECIFIED WITH -l WITHIN THE COMMAND&gt; now exists on px.grid.sara.nl.


</pre>
</pre>
'''ATTENTION:''' The password entered for MyProxy will be used to authenticate the user via GlobusOnline.eu and the user name is the one which is chosen by you and provided with "-l" argument.<br>  


'''ATTENTION:''' The password entered for MyProxy will be used to authenticate the user via GlobusOnline.eu and the user name is the one which is chosen by the MyProxy server for the user, in this case - "karolis".
<br>


'''2. Perform file transfers'''


2. Perform file transfers
a) Go to http://www.globusonline.eu/signup and create an account. (The account can be later associated with your EGI Single Sign-On account ([http://egi.eu/sso EGI SSO]). '''Attention:''' login with the EGI Single Sign On account is possible only if you have a valid VOMS proxy in the EGI catch-all MyProxy server.)<br>


a) Go to http://www.globusonline.eu/signup and create an account, later you can associate it and login with your EGI Single Sign-On account ([http://egi.eu/sso EGI SSO]).<br>
b) Go to "Start Transfer"<br>  


b) Go to "Start Transfer"<br>
c) Find the transfer endpoints that are available for your VO by searching for your VO name in the endpoint field e.g. biomed<br>  


c) Find the transfer endpoints that are available for your VO by searching for your VO name in the endpoint field e.g. biomed<br>
[[Image:Gobiomed.png|thumb|none|800px]]<br>  


[[File:gobiomed.png|none|thumb|800px]]<br>
'''IMPORTANT TO KNOW:''' you may use field "Credential Lifetime (hours)" and enter 168hours - 1 week of proxy lifetime. (Default is 12hours). What is important to understand is that GlobusOnline.eu will retrieve the instance of your proxy from MyProxy server and will activate the endpoint for 168hours - within the GlobusOnline.eu interface you will see that endpoint is activated for 168hours, but this might not be true because your voms proxy lifetime depends on your VOMS server, where you should consult your VO Manager. Some EGI voms servers allow up to 24hours only of the voms proxy lifetime to be.  


e) Click "Authenticate"<br>
e) Click "Authenticate"<br>  


What you can do with the endpoints? <br>
What you can do with the endpoints? <br>  


*Transfer files from your laptop using GlobusConnect client to a endpoint using Globus Online.<br>
*Transfer files from your laptop using GlobusConnect client to a endpoint using Globus Online.<br>
Line 264: Line 222:
*Transfer files from an endpoint to another endpoint using Globus Online.<br>
*Transfer files from an endpoint to another endpoint using Globus Online.<br>


Please consult with the Globus Online documentations that are available at http://www.globusonline.eu.
Please consult with the Globus Online documentations that are available at http://www.globusonline.eu.  


== Additional materials ==
== Additional materials ==


The tests that have been carried out while creating this Cookbook are available at [[GO_testing]]
The tests that have been carried out while creating this Cookbook are available at [[GO testing]]

Revision as of 10:32, 28 April 2015

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


The Globus Online cookbook describes how the 'SRM type' storage services that are federated into EGI can be used as endpoints of file transfers managed by Globus Online. The "biomed" VO is used as an example in the Cookbook to demonstrate EGI storage usage, but the steps are the same for other EGI VOs as well.

The cookbook was prepared as a guide for EGI Virtual Organisations (VOs) on how to use the Globus Online service that is available at http://www.globusonline.eu. GlobusOnline.eu provides robust and easy to use file transfer capabilities for EGI users. The service manages file transfers for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status.

The Cookbook consists of two parts:

  1. For VO Managers : The first part provides step-by-step instructions for VO Managers on how to register SRM storage services in Globus Online in such a way, that these appear as transfer endpoints for VO members. This registration could be performed by any member of a VO, however for most VOs the VO Manager is the most suitable person to complete this step because the VO Manager has sufficient knowledge on storage sites that support the VO and about the BDII information system where detailed information about the storages is recorded.
  2. For VO Members (researchers) : The second part provides step-by-step instructions for VO members on how to use VO storage endpoints in Globus Online. This part is relevant for any meber of any EGI VO. The list of EGI VOs and information on joining these VOs is available at http://operations-portal.egi.eu/vo.

Important note: the http://www.globusonline.eu server is hosted in the US, but the files that the service moves between EGI sites do not leave Europe. The service orchestrates file copies with the GridFTP third party transfer, so files are copied directly between the EGI endpoints.

This cookbook has been prepared by the EGI.eu User Community Support Team in consultation with representatives of EGI Operations, storage technology and information system developer groups. Please email any feedback about this Cookbook to the EGI.eu User Community Support Team: ucst@egi.eu.

VO Managers

1. Get all endpoints suporting gsiftp for biomed VO (Glue 2.0):

-bash-3.2$ ldapsearch -LLL -x -h lcg-bdii.cern.ch -p 2170 -b o=glue '(&(objectclass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:biomed))' \
-bash-3.2$ GLUE2AccessPolicyEndpointForeignKey | perl -p00e 's/\r?\n //g' | grep GLUE2AccessPolicyEndpointForeignKey | sort | uniq | awk '{ print $2 }' |grep -i gsiftp > APEndpoints

2. Get all GridFTP (gsiftp) endpoints for biomed VO (Glue 2.0):

-bash-3.2$ for i in `cat APEndpoints`;do ldapsearch -LLL -x -h lcg-bdii.cern.ch -p 2170 -b o=glue \ 
-bash-3.2$ '(&(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID='$i'))' \
-bash-3.2$ GLUE2EndpointURL GLUE2EndpointImplementationName GLUE2ENdpointImplementationVersion GLUE2EndpointInterfaceName | grep ^GLUE ;echo "-------";done

3. Find myproxy server for biomed VO:

-bash-3.2$ lcg-infosites --is lcg-bdii.cern.ch --vo biomed myproxy
myproxy://px.grid.sara.nl:7512/
myproxy.cern.ch:7512
myproxy.usatlas.bnl.gov:7512
myproxy://cluster6.knu.ac.kr:7512/
myproxy://grid-mypx.feit.ukim.edu.mk:7512/
myproxy://grid-px0.desy.de:7512/
myproxy://grid153.kfki.hu:7512/
myproxy://gridpx01.ifca.es:7512/
myproxy://ii.biomed.kiev.ua:7512/
myproxy://kek2-px.cc.kek.jp:7512/
myproxy://lcg-px01.icepp.jp:7512/
myproxy://lcg2proxy.ific.uv.es:7512/
myproxy://lcgpx01.jinr.ru:7512/
myproxy://lcgrbp01.gridpp.rl.ac.uk:7512/
<...>

If your VO does not have a dedicated MyPRoxy server, then the catch-all MyProxy server of EGI can be used. This catch-all service is available from CESNET at myproxy.egi.eu.


Extras: example script to obtain GridFTP endpoint information with Glue 2.0. (Command line tools from emi-ui-3.0.0-1.el6.x86_64):

#!/bin/bash

bdii="top-bdii.cern.ch";


ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&(objectclass=GLUE2AccessPolicy)(GLUE2PolicyRule=*:$1))" GLUE2AccessPolicyEndpointForeignKey | perl -p00e 's/\r?\n //g' | grep GLUE2AccessPolicyEndpointForeignKey |sort | uniq | awk '{ print $2 }' |grep -i gsiftp > APEndpoints


for i in `cat APEndpoints`;do 


SEtype=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointImplementationName |grep ^GLUE2| cut -d" " -f2`;


if [ "$SEtype" = "DPM" ]; then

SE=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointServiceForeignKey |grep ^GLUE2| cut -d" " -f2`;

fi


if [ "$SEtype" = "dCache" ]; then

SE=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointServiceForeignKey |grep ^GLUE2| cut -d" " -f2 | cut -d":" -f2|sed 's/\/data//g'`;

fi


SRM=`lcg-info --list-service --bdii ldap://$bdii:2170 --vo $1 --query "ServiceType=SRM" --attrs "ServiceEndpoint"|grep $SE | cut -d" " -f3|head -1`;

details=`ldapsearch -LLL -x -H ldap://$bdii:2170 -b o=glue "(&(objectclass=GLUE2Endpoint)(GLUE2EndpointInterfaceName=gsiftp)(GLUE2EndpointID=$i))" GLUE2EndpointURL GLUE2EndpointImplementationName GLUE2ENdpointImplementationVersion | grep ^GLUE`;

VOInfoPath=`lcg-info --list-se --bdii ldap://$bdii:2170 --vo biomed --query "SE=$SE" --attrs "VOInfoPath"|grep VOInfoPath|awk '{print $3}'`;

echo -e "$details\nVOInfoPath: $VOInfoPath\nHost: $SE\nSRM: $SRM\n";

done;

-bash-3.2$ ./go.sh biomed


GLUE2EndpointImplementationName: DPM GLUE2EndpointURL: gsiftp://glite-se.scai.fraunhofer.de:2811 GLUE2EndpointImplementationVersion: 1.8.8 VOInfoPath: /dpm/scai.fraunhofer.de/home/biomed Host: glite-se.scai.fraunhofer.de SRM: httpg://glite-se.scai.fraunhofer.de:8446/srm/managerv2


GLUE2EndpointImplementationVersion: 2.6.19 GLUE2EndpointURL: gsiftp://dcache-door-desy09.desy.de:2811 GLUE2EndpointImplementationName: dCache VOInfoPath: /pnfs/desy.de/biomed Host: dcache-se-desy.desy.de SRM: httpg://dcache-se-desy.desy.de:8443/srm/managerv2

<...>


Testing access to GridFTP endpoint:

-bash-3.2$ uberftp glite-se.scai.fraunhofer.de "ls /dpm/scai.fraunhofer.de/home/biomed"


4. Register the endpoints in Globus Online:

a) Go to http://www.globusonline.eu/signup, create an account with your VO name e.g. biomed. (Note that usernames can include only letters and numbers and "_".)
(Using the VO name as an account name will ensure that VO members can easily find the endpoints that are available for them.)

b) Upload your ssh public key via globusonline.eu -> "manage identities"

c) Now you can manage your account using globusonline.eu client

-bash-3.2$ ssh biomed@cli.globusonline.eu "help"

d) Register the endpoints in the Globus Online service.

Endpoints can be registered through the command line interface or the graphical portal interface of Globus Online. The command line interface allows the association of a default directory with the endpoint and this simplifies the use of the endpoint by VO members. The below example therefore shows the command line tool for endpoint registration. The MyProxy server becomes the default MyProxy, and can be replaced with other MyProxy by VO members during the activation of the endpoint.

-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-add fraunhofer_DE -p gsiftp://glite-se.scai.fraunhofer.de:2811"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --public fraunhofer_DE"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --myproxy-server=px.grid.sara.nl fraunhofer_DE"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-modify --default-directory=/dpm/scai.fraunhofer.de/home/biomed fraunhofer_DE"
-bash-3.2$ ssh biomed@cli.globusonline.eu "endpoint-list -v fraunhofer_DE"
Name                    : biomed#fraunhofer_DE
Host(s)                 : gsiftp://glite-se.scai.fraunhofer.de:2811
Subject(s)              : 
Target Endpoint         : n/a
Default Directory       : /dpm/scai.fraunhofer.de/home/biomed
Force Encrypted Transfer: No
Disable Verify          : No
MyProxy Server          : px.grid.sara.nl
MyProxy DN              : n/a
MyProxy OAuth Server    : n/a
Credential Status       : EXPIRED
Credential Expires      : 
Credential Subject      : 


VO Members

1. Generate a VOMS proxy and upload it into a MyProxy server. '

There are two ways to do this:

a). With a graphical tool, such as GSISSH-Term.

OR

b). With the command line tools of the User Interface machine of your VO.

The usage of GSISSH-Term for proxy management is explained on a dedicated page. A usage of the command line tools is detailed below.


Generating a VOMS proxy with the command line tools (You should have these installed on the User Interface machine of your VO):

-bash-3.2$ voms-proxy-init --voms biomed

Enter GRID pass phrase for this identity:
Contacting cclcgvomsli01.in2p3.fr:15000 [/O=GRID-FR/C=FR/O=CNRS/OU=CC-IN2P3/CN=cclcgvomsli01.in2p3.fr] "biomed"...
Remote VOMS server contacted succesfully.

Created proxy in /tmp/x509up_u507.

Your proxy is valid until Fri May 24 04:12:03 CEST 2013

VOMS proxy lifetime: by default voms proxy extension is generated for 12hours, some voms servers within EGI allow to have lifetime for 1 week (168hours), some allow up to 24hours, please consult your VO manager.


Upload VOMS proxy to a MyProxy server with the command line tool. Note: If your VO does not have any MyProxy server, then you can use the EGI catch-all MyProxy server. Further information about the EGI catch-all MyProxy server.

-bash-3.2$ myproxy-init -l <CHOOSE ANY USERNAME> -s px.grid.sara.nl
Your identity: /O=dutchgrid/O=users/O=egi/CN=Karolis Eigelis
Enter GRID pass phrase for this identity:
Creating proxy ........................................................................ Done
Proxy Verify OK
Your proxy is valid until: Thu May 30 16:15:47 2013
Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
Verifying - Enter MyProxy pass phrase: <YOU NEW PASSWORD TO BE USED LATER AT GLOBUSONLINE>
A proxy valid for 168 hours (7.0 days) for user <YOUR USERNAME SPECIFIED WITH -l WITHIN THE COMMAND> now exists on px.grid.sara.nl.

ATTENTION: The password entered for MyProxy will be used to authenticate the user via GlobusOnline.eu and the user name is the one which is chosen by you and provided with "-l" argument.


2. Perform file transfers

a) Go to http://www.globusonline.eu/signup and create an account. (The account can be later associated with your EGI Single Sign-On account (EGI SSO). Attention: login with the EGI Single Sign On account is possible only if you have a valid VOMS proxy in the EGI catch-all MyProxy server.)

b) Go to "Start Transfer"

c) Find the transfer endpoints that are available for your VO by searching for your VO name in the endpoint field e.g. biomed

Gobiomed.png


IMPORTANT TO KNOW: you may use field "Credential Lifetime (hours)" and enter 168hours - 1 week of proxy lifetime. (Default is 12hours). What is important to understand is that GlobusOnline.eu will retrieve the instance of your proxy from MyProxy server and will activate the endpoint for 168hours - within the GlobusOnline.eu interface you will see that endpoint is activated for 168hours, but this might not be true because your voms proxy lifetime depends on your VOMS server, where you should consult your VO Manager. Some EGI voms servers allow up to 24hours only of the voms proxy lifetime to be.

e) Click "Authenticate"

What you can do with the endpoints?

  • Transfer files from your laptop using GlobusConnect client to a endpoint using Globus Online.
  • Transfer files from an endpoint to your laptop using GlobusConnect client and Globus Online.
  • Transfer files from an endpoint to another endpoint using Globus Online.

Please consult with the Globus Online documentations that are available at http://www.globusonline.eu.

Additional materials

The tests that have been carried out while creating this Cookbook are available at GO testing