Jointly exploit EGI and EUDAT services

Overview

For users

For resource providers

Infrastructure status

Site-specific configuration

Architecture

EGI and EUDAT interoperability

Interconnect EGI Federated Cloud VMs with EUDAT B2STAGE and B2SAFE services

Prerequisites

• You need to have granted access to the EGI Federated Cloud resources. See how to get access.
• You need to have granted access to a B2STAGE and B2SAFE instances with the same credential used to access the EGI Federated Cloud. Directly contact the EUDAT contact person at the computing centre offering B2STAGE/B2SAFE services. They will need the Distinguished Name (DN) of your personal certificate, e.g.

  /DC=org/DC=terena/DC=tcs/C=SE/O=<your organization>/CN=<your name>

• Setup the OCCI client as described in HOWTO11 or request access to AppDB VMOps Dashboard (currently in Alpha)

Create a VM in the EGI Federated Cloud

This section describes how to create a VM in the EGI Federated Cloud with pre-configured on board all the software needed to interact with B2STAGE and B2SAFE services.

• Select a VM image in the AppDB or upload a new one.
• Download the contextualisation script from the EGI-EUDAT integration pilot software appliance. The script was tested on an Ubuntu 14 machine, small changes could be needed if applied to other distributions.
  • Click on the Software Appliance Version tab and download the EGI-EUDAT-Integration.yml script available under the Contextualisation Script section.
• Open the EGI-EUDAT-Integration.yml contextualization script and copy your SSH public key where indicated.
• Create a VM from the chosen image applying the downloaded contextualisation script. You can follow the instructions available here.
• The GridFTP client globus-url-copy (needed to access EUDAT B2SAFE services) is installed in the just started VM and username/password are set to egieudat/egieudat (do not forget to change the password!).
• Get the IP of the machine using the OCCI describe command and access it through SSH. More information on managing VMs is found at Managing_VMs.

Add block storage to the VM

• Create a block storage, at the same resource provider that is hosting the VM and attach it to the VM following this instructions.
• Block Storage will appear as block devices into your VM. Usually these devices are empty upon creation. The device will be initially empty, so you will need to format it and mount it. This will remove all data stored on the device, then execute the following steps only the first time the device is used.
  • Log in in the machine through SSH
  • Find the disk via the command fdisk. This returns a list of devices, select the one which has the corresponding size:

    fdisk -l

  • Follow this instructions to format and mount the device.

Access to B2STAGE/B2SAFE from your EGI Federated Cloud VM

• Create a proxy and copy it to the VM (e.g. using scp). This is required for access to B2STAGE/B2SAFE.
  • This step is a bit more complicated when using the egifedcloud/fedcloud-userinterface docker image. The proxy exists only in the egifedcloud/fedcloud-userinterface docker container, because occi is only installed there. But scp is not available in the docker container. You can use the following command to install openssh-client in egifedcloud/fedcloud-userinterface and to copy proxy certificate to VM:

docker run --rm --volumes-from occi-voms-proxy -it \ 
-v $PWD:/data:rw -v ~/.ssh_egi:/ssh:ro \
egifedcloud/fedcloud-userinterface \
/bin/bash -c "(apt-get install -y openssh-client && scp -i /ssh/fedcloud ${X509_USER_PROXY} egieudat@< IP-address >:)"

Upload and download files to/from B2STAGE instance with globus-url-copy

• Upload a file to B2STAGE:

globus-url-copy -vb -cred <X509_USER_PROXY on VM> file:///<local filename> gsiftp://<B2STAGEinstance>/<path>/<filename>

• Download a file from B2STAGE:

globus-url-copy -vb -cred <X509_USER_PROXY on VM> gsiftp://<B2STAGEinstance>/<path>/<filename> file:///<local filename>

• List remote files on B2STAGE instance:

globus-url-copy -vb -cred <X509_USER_PROXY on VM> -list gsiftp://<B2STAGEinstance>/<path>/

• If you do not know the path on the B2STAGE instance use

  globus-url-copy -vb -cred <X509_USER_PROXY on VM> -list gsiftp://<B2STAGEinstance>/~/

  where ~ is the home directory

Install a GridFTP client in the EGI Federated Cloud VM

A GridFTP client, e.g. UberFTP, allows also other file manipulations like deleting files on B2STAGE/B2SAFE.

• Instruction how to install UberFTP:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:maarten-kooyman-6/ppa
sudo apt-get update
sudo apt-get install uberftp

• To use uberftp the certificate needs to be copied to /tmp/x509up_u1000 (this path can change according to the id of your user in the machine):

  cp <X509_USER_PROXY on VM> /tmp/x509up_u1000

• For file transfer or manipulations:

  uberftp <B2STAGEinstance>

• See B2SAFE-B2STAGE-Training for more information.

EGI DataHub - Stage-In/Stage-Out data to B2STAGE/B2SAFE

Intro

• Setup a OneProvider
  • Use the context script
  • Use datahub.egi.eu...
• Move data...