EGI Data Transfer

From EGIWiki
Jump to: navigation, search

Contents

Overview

The EGI Data Transfer service allows you to move any type of data files asynchronously from one place to another within a specified VO. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.

Data Transfer is ideal to move large amounts of files or very large files within a VO. The Data Transfer service has mechanisms to ensure automatic retry in case of failure. Users won't anymore have to manage transfers manually, they will use a service dedicated to this task that will be able to monitor, manage and optimise the submitted transfers.

The service could be enabled for a specific VO, for LToS and for Research Infrastructures' users allowing them to transfer data between two endpoints they have access to.

It could also be integrated by some higher level service such as Rucio, DIRAC and Science Gateways.

A WebFTS frontend is provided. The web interface will allow to do this graphically without having to use a CLI.

Technology

The following services are made available to users for managing their data transfers:

FTS3

FTS3 is the service responsible for globally distributing the majority of the Large Hadron Collider (LHC) data across the Worldwide LHC Computing Grid (WLCG) infrastructure. It is a low level, multi-protocol data movement service, responsible for reliable bulk transfer of files from one site to another while allowing participating sites to control the network resource usage. A full description is available in the paper FTS3: New Data Movement Service For WLCG - IOPscience and in the documentation.

In addition to effectuating data transfer jobs, the service can also be used to monitor and log transfer jobs. Users can interact with FTS3 using command line tools, RESTful API calls or a web gui.

WebFTS

WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses a FTS3 endpoint as transfer engine.

More information about WebFTS is at http://fts3-docs.web.cern.ch/fts3-docs/docs/webfts.html

Get Started

A user can order the Data Transfer service via EGI Marketplace, and from the eInfraCentral Platform

The service order will be handled by the support team.

Pre-requests

For users

Service Provision and Support

The service is provided by two providers: CERN and STFC/UKRI, and EGI has signed OLAs with them.

FTS3 can be used on its own. WebFTS is interacting with a specific FTS3 instance.

Information about technology supports and the help desk can be found at https://wiki.egi.eu/wiki/GGUS:EGI_Data_Transfer_Service_FAQ

Use Cases

The ELIXIR experience

A webFTS instance is customised for ELIXIR is available at https://fts3.du2.cesnet.cz/index.php It is integrated with ELIXIR AAI and supports OIDC. It also has interfaces with RCauth Online CA. VO attributes are integrated.

The use case is described in Enable Federated Authentication with FTS

Experience from ELIXIR shows:

For more information please refer to ELIXIR Test of FTS3 for transfers

FAQ

What can FTS talk to?

It mainly talks to XrootD, HTTPS, GridFTP and possibly some S3

How a VO gets these endpoints ready to accept FTS transfers?

This needs to ask the FTS admin to enable the VO on their site. There is no need to register any endpoints and the authentication is done using X509 (so sites needs to use a supported protocol with X509 for authN/authZ)

What's the roadmap for Check-in integration?

We don’t have yet a date, for the CLI I don’t really except we have something anytime soon (We are not funding the FTS services), but for WebFTS RAL/STFC may try to do it but no concrete plan

Reference

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Print/export