Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

EGI Data Transfer

From EGIWiki
Jump to navigation Jump to search

Overview

The EGI Data Transfer service allows you to move any type of data files asynchronously from one place to another within a specified VO. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.

Data Transfer is ideal to move large amounts of files or very large files within a VO. The Data Transfer service has mechanisms to ensure automatic retry in case of failure. Users won't anymore have to manage transfers manually, they will use a service dedicated to this task that will be able to monitor, manage and optimise the submitted transfers.

The service could be enabled for a specific VO, for LToS and for Research Infrastructures' users allowing them to transfer data between two endpoints they have access to.

It could also be integrated by some higher level service such as Rucio, DIRAC and Science Gateways.

A WebFTS frontend is provided. The web interface will allow to do this graphically without having to use a CLI.

Technology

The following services are made available to users for managing their data transfers:

FTS3

FTS3 is a low-level data movement service, responsible for reliable bulk transfer of files from one site to another while allowing participating sites to control the network resource usage. The software is open source to transfer data reliably, in a robust manner and at large scale between storage systems using protocols such as HTTPS, WebDav, GridFTP and SRM.

FTS3 provides a sophisticated and easy to use web interface for transfer management and monitoring, and a data management software, to delete files efficiently, optimising resource usage by enabling adaptive concurrency, reuse of connections and a smart retry mechanism.

For more information about FTS3: http://fts3-docs.web.cern.ch/fts3-docs/

WebFTS

WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses a FTS3 endpoint as transfer engine.

More information about WebFTS is at http://fts3-docs.web.cern.ch/fts3-docs/docs/webfts.html

Get Starts

The service can be ordered via EGI Marketplace, and from the eInfraCentro Platform

User who apply to use the service should have already have a valid Certificate. Otherwise, Getting a user certificate

Support for new VOs must be explicitly requested as they need to be configured in WebFTS and on the FTS cluster nodes (vomses files)

Service Provision and Support

The service is provided by two providers: CERN and STFC/UKRI, and EGI has signed OLAs with them.

  • CERN provides the following access instances:
    • an FTS3 instance at https://fts3-public.cern.ch:8446/. fts3-public.cern.ch is deployed as a load-balanced alias across a number of machines (5 as of 2018/09/19). FTS3 service offers multiple interfaces: Command Line Interface (CLI), API and web interface using WebFTS.
    • a WebFTS instance linked to their FTS3 instance: https://webfts.cern.ch/. It is a single instance. WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses an FTS3 endpoint as transfer engine.

FTS3 can be used on its own. WebFTS is interacting with a specific FTS3 instance.

Information about technology supports and the help desk can be found at https://wiki.egi.eu/wiki/GGUS:EGI_Data_Transfer_Service_FAQ

Use Cases

Reference