Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "EGI Data Transfer"

From EGIWiki
Jump to navigation Jump to search
Line 32: Line 32:
* User who apply to use the service should have already have a valid Certificate. Otherwise, [[Federated_Cloud_user_support#Getting a user certificate|Getting a user certificate]]
* User who apply to use the service should have already have a valid Certificate. Otherwise, [[Federated_Cloud_user_support#Getting a user certificate|Getting a user certificate]]


* Support for new VOs must be explicitly requested as they need to be configured in WebFTS and on the FTS cluster nodes (vomses files)
* Support for new VOs must be explicitly requested as they need to be configured in WebFTS and on the FTS cluster nodes (voms files)


''For users''
''For users''

Revision as of 10:05, 16 April 2019

Overview

The EGI Data Transfer service allows you to move any type of data files asynchronously from one place to another within a specified VO. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.

Data Transfer is ideal to move large amounts of files or very large files within a VO. The Data Transfer service has mechanisms to ensure automatic retry in case of failure. Users won't anymore have to manage transfers manually, they will use a service dedicated to this task that will be able to monitor, manage and optimise the submitted transfers.

The service could be enabled for a specific VO, for LToS and for Research Infrastructures' users allowing them to transfer data between two endpoints they have access to.

It could also be integrated by some higher level service such as Rucio, DIRAC and Science Gateways.

A WebFTS frontend is provided. The web interface will allow to do this graphically without having to use a CLI.

Technology

The following services are made available to users for managing their data transfers:

FTS3

FTS3 is the service responsible for globally distributing the majority of the Large Hadron Collider (LHC) data across the Worldwide LHC Computing Grid (WLCG) infrastructure. It is a low level, multi-protocol data movement service, responsible for reliable bulk transfer of files from one site to another while allowing participating sites to control the network resource usage. A full description is available in the paper FTS3: New Data Movement Service For WLCG - IOPscience and in the documentation.

In addition to effectuating data transfer jobs, the service can also be used to monitor and log transfer jobs. Users can interact with FTS3 using command line tools, RESTful API calls or a web gui.

WebFTS

WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses a FTS3 endpoint as transfer engine.

More information about WebFTS is at http://fts3-docs.web.cern.ch/fts3-docs/docs/webfts.html

Get Starts

A user can order the Data Transfer service via EGI Marketplace, and from the eInfraCentro Platform

The service order will be handled by the support team.

Pre-requests

  • Support for new VOs must be explicitly requested as they need to be configured in WebFTS and on the FTS cluster nodes (voms files)

For users

Service Provision and Support

The service is provided by two providers: CERN and STFC/UKRI, and EGI has signed OLAs with them.

  • CERN provides the following access instances:
    • an FTS3 instance at https://fts3-public.cern.ch:8446/. fts3-public.cern.ch is deployed as a load-balanced alias across a number of machines (5 as of 2018/09/19). FTS3 service offers multiple interfaces: Command Line Interface (CLI), API and web interface using WebFTS.
    • a WebFTS instance linked to their FTS3 instance: https://webfts.cern.ch/. It is a single instance. WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses an FTS3 endpoint as transfer engine.

FTS3 can be used on its own. WebFTS is interacting with a specific FTS3 instance.

Information about technology supports and the help desk can be found at https://wiki.egi.eu/wiki/GGUS:EGI_Data_Transfer_Service_FAQ

Use Cases

The ELIXIR experience

A webFTS instance is customised for ELIXIR is available at https://fts3.du2.cesnet.cz/index.php It is integrated with ELIXIR AAI and supports OIDC. It also has interfaces with RCauth Online CA. VO attributes are integrated.

The use case is described in Enable Federated Authentication with FTS

Experience from ELIXIR shows:

  • The FTS3 service seems to fulfil most of the requirements that we have for an ELIXIR data transfer service: multi-protocol, monitoring and logging, checksums, API, web gui, …
  • The software is free software (Apache License, Version 2.0) and seems straight-forward to deploy. All components are available in the RHEL/CENTOS EPEL repository. Installation and configuration guide.
  • The fts3 development team is very approachable.
  • Some development is needed if we want to make the webfts interface connect to the ELIXIR credential translation service for proxy certificates.
  • It seems like logs of transfers jobs are per default public. This might be a problem in some cases (e.g. pharma companies that want to hide what they are working on).

For more information please refer to ELIXIR Test of FTS3 for transfers

Reference