EGI Data Transfer

Overview

The EGI Data Transfer service allows you to move any type of data files asynchronously from one place to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.

Data Transfer is ideal to move large amounts of files or very large files. The Data Transfer service has mechanisms to ensure automatic retry in case of failure. Users will be able to easily and confidently submit cross-site transfer jobs. Using the WebFTS frontend, it's now possible to do this in a graphical way using a web interface

This service maybe used by different kind of users, individual researchers and large research groups could use it to transfer their private or public data and potentially site admins may use it to transfer data between endpoints. It could also be integrated by some higher level service such as Rucio, DIRAC and Science Gateways.

Users won't anymore have to manage transfers manually, they will use a service dedicated to this task that will be able to monitor, manage and optimise the submitted transfers. The web interface will allow to do this graphically without having to use a CLI.

The service could be enabled for a specific VO, for LToS and for Research Infrastructures' users allowing them to transfer data between two endpoints they have access to.

Technology

The service is based on FTS3 and Globus Online.

FTS3

FTS3 is a low level data movement service, responsible for reliable bulk transfer of files from one site to another while allowing participating sites to control the network resource usage. The software is open source to transfer data reliably, in a robust manner and at large scale between storage systems using protocols such as HTTPS, WebDav, GridFTP and SRM.

FTS3 provides a sophisticated and easy to use web interface for transfer management and monitoring, and a data management software, to delete files efficiently, optimising resource usage by enabling adaptive concurrency, reuse of connections and a smart retry mechanism.

Globus Online

Globus Online is a service for moving, sharing, and archiving large volumes of data among distributed sites.

With Globus Online, you hand-off data movement tasks to a hosted service that manages the entire operation, monitoring performance and errors, retrying failed transfers, correcting problems automatically whenever possible, and reporting status to keep you informed while you focus on your research. Command line and web-based interfaces are available.

The command line interface, which requires only ssh to be installed on the client, is the method of choice for grid-based workflows.

Get Starts

The service can be ordered via EGI Marketplace

Service Provision and Support

The service is provided by two providers: CERn and STFC/UKRI, and EGI has signed OLAs with them.

• CERN provides the following access instances:
  • an FTS3 instance at https://fts3-public.cern.ch:8446/. fts3-public.cern.ch is deployed as a load-balanced alias across a number of machines (5 as of 2018/09/19). FTS3 service offers multiple interfaces: Command Line Interface (CLI), API and web interface using WebFTS.
  • a WebFTS instance linked to their FTS3 instance: https://webfts.cern.ch/. It is a single instance. WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses an FTS3 endpoint as transfer engine.

FTS3 can be used on its own. WebFTS is interacting with a specific FTS3 instance.

• STFC/UKRI provides the following access instance:
  • FTS3: https://lcgfts3.gridpp.rl.ac.uk:8446/

Information about technology supports and the help desk can be found at https://wiki.egi.eu/wiki/GGUS:EGI_Data_Transfer_Service_FAQ

Reference

• FTS documentation: http://fts3-docs.web.cern.ch/fts3-docs/
• FTS source code: http://gitlab.cern.ch/fts/
• FTS Main site: http://fts.web.cern.ch/
• https://svnweb.cern.ch/trac/glitefts/wiki/FTSUserGuide
• Tutorial: https://webfts.cern.ch/