Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Jointly exploit EGI and EUDAT services

From EGIWiki
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Overview For users For resource providers Infrastructure status Site-specific configuration Architecture




EGI and EUDAT interoperability

This page describes how to jointly exploit EGI and EUDAT services.

Interconnect EGI Federated Cloud VMs with EUDAT B2STAGE and B2SAFE services

Prerequisites

  • You need to have granted access to the EGI Federated Cloud resources. See how to get access.
  • You need to have granted access to a B2STAGE and B2SAFE instances with the same credential used to access the EGI Federated Cloud. Directly contact the EUDAT contact person at the computing centre offering B2STAGE/B2SAFE services. They will need the Distinguished Name (DN) of your personal certificate, e.g.
    /DC=org/DC=terena/DC=tcs/C=SE/O=<your organization>/CN=<your name>
    You can also contact the B2STAGE or B2SAFE support teams.
  • Setup your client to manage VMs in the EGI Federated Cloud. Please, refer to this documentation page. You can also use the AppDB VMOps Dashboard (currently in Alpha) but in such case you need to manually install the gridftp software to interact with B2STAGE and B2SAFE.

Create a VM in the EGI Federated Cloud

This section describes how to create a VM in the EGI Federated Cloud with pre-configured on board all the software needed to interact with B2STAGE and B2SAFE services.

  • Select a VM image in the AppDB or upload a new one.
  • Download the contextualisation script from the EGI-EUDAT integration pilot software appliance. The script was tested on an Ubuntu 14 machine, small changes could be needed if applied to other distributions.
    • Click on the Software Appliance Version tab and download the EGI-EUDAT-Integration.yml script available under the Contextualisation Script section.
  • Open the EGI-EUDAT-Integration.yml contextualization script and copy your SSH public key where indicated.
  • Create a VM from the chosen image applying the downloaded contextualisation script. You can follow the instructions available here.
  • The GridFTP client globus-url-copy (needed to access EUDAT B2SAFE services) is installed in the just started VM and username/password are set to egieudat/egieudat (do not forget to change the password!).
  • Get the IP of the machine using the OCCI describe command or the OpenStack CLI and access it through SSH. More information on managing VMs with OCCI is found at Managing VMs.

Add block storage to the VM

  • Create a block storage at the same resource provider that is hosting the VM and attach it to the VM. Instructions for OCCI are available here.
  • Block Storage will appear as a block device into your VM. Usually these devices are empty upon creation and you need to format it and mount it. This will remove all data stored on the device, then execute the following steps only the first time the device is used.
    • Log in in the machine through SSH
    • Find the disk via the command fdisk. This returns a list of devices, select the one which has the corresponding size:
      fdisk -l
    • Follow this instructions to format and mount the device.

Access to B2STAGE/B2SAFE from your EGI Federated Cloud VM

  • Create a proxy and copy it to the VM (e.g. using scp). This is required for access to B2STAGE/B2SAFE.
    • This step is a bit more complicated when using the egifedcloud/fedcloud-userinterface docker image. The proxy exists only in the egifedcloud/fedcloud-userinterface docker container, because occi is only installed there. But scp is not available in the docker container. You can use the following command to install openssh-client in egifedcloud/fedcloud-userinterface and to copy proxy certificate to VM:
docker run --rm --volumes-from occi-voms-proxy -it \ 
-v $PWD:/data:rw -v ~/.ssh_egi:/ssh:ro \
egifedcloud/fedcloud-userinterface \
/bin/bash -c "(apt-get install -y openssh-client && scp -i /ssh/fedcloud ${X509_USER_PROXY} egieudat@< IP-address >:)"

Upload and download files to/from B2STAGE instance with globus-url-copy

  • Upload a file to B2STAGE:
globus-url-copy -vb -cred <X509_USER_PROXY on VM> file:///<local filename> gsiftp://<B2STAGEinstance>/<path>/<filename>
  • Download a file from B2STAGE:
globus-url-copy -vb -cred <X509_USER_PROXY on VM> gsiftp://<B2STAGEinstance>/<path>/<filename> file:///<local filename>
  • List remote files on B2STAGE instance:
globus-url-copy -vb -cred <X509_USER_PROXY on VM> -list gsiftp://<B2STAGEinstance>/<path>/
  • If you do not know the path on the B2STAGE instance use
globus-url-copy -vb -cred <X509_USER_PROXY on VM> -list gsiftp://<B2STAGEinstance>/~/

where ~ is the home directory

Install a GridFTP client in the EGI Federated Cloud VM

Other GridFTP clients, e.g. UberFTP, allow also other file manipulations like deleting files on B2STAGE/B2SAFE. In the following, you can find the instruction to install and use UberFTP:

  • Install the software packages:
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:maarten-kooyman-6/ppa
sudo apt-get update
sudo apt-get install uberftp
  • To use uberftp the certificate needs to be copied to /tmp/x509up_u1000 (this path can change according to the id of your user in the machine):
cp <X509_USER_PROXY on VM> /tmp/x509up_u1000
  • For file transfer or manipulations:
uberftp <B2STAGEinstance>

EGI DataHub - Stage-In/Stage-Out data to B2STAGE/B2SAFE

In this section is explained how to connect storage managed via the EGI Data Hub with the EUDAT B2STAGE/B2SAFE services. This setup allows to have a distributed file-system shared by many VMs, that could be located also in different resource providers, connected to the EUDAT preservation services.

  • As first step, you need to setup a OneProvider instance in the EGI Federated Cloud following this instructions.
    • Use the EGI-EUDAT integration pilot software appliance contextualisation script to create the VM as described here. In such way, the globus-url-copy gridftp client will be installed in the machine.
    • Connect your OneProvider to the datahub.egi.eu. To do it, use datahub.egi.eu in place of [OneZone-IP] in the command to setup the OneProvider instance:
      sudo ./run_onedata.sh --provider --provider-fqdn [OneData-IP] --zone-fqdn [OneZone-IP] &
  • Now you can stage-in or stage-out data from/to the EGI DataHub to/from B2STAGE/B2SAFE using gridftp commands with files stored in the storage published by the OneProvider instance (the ${ONEPROVIDER_DATA_DIR}). Files copied in OneData from B2STAGE/B2SAFE will be automatically visible to all the EGI Federated Cloud VMs that mounted the OneProvider shared storage. See Access your data from an instance running into the EGI FedCloud for more details.