EGI Data Transfer
The EGI Data Transfer service allows you to move any type of data files asynchronously from one place to another within a specified VO. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.
Data Transfer is ideal to move large amounts of files or very large files within a VO. The Data Transfer service has mechanisms to ensure automatic retry in case of failure. Users won't anymore have to manage transfers manually, they will use a service dedicated to this task that will be able to monitor, manage and optimise the submitted transfers.
The service could be enabled for a specific VO, for LToS and for Research Infrastructures' users allowing them to transfer data between two endpoints they have access to.
It could also be integrated by some higher level service such as Rucio, DIRAC and Science Gateways.
A WebFTS frontend is provided. The web interface will allow to do this graphically without having to use a CLI.
The following services are made available to users for managing their data transfers:
FTS3 is the service responsible for globally distributing the majority of the Large Hadron Collider (LHC) data across the Worldwide LHC Computing Grid (WLCG) infrastructure. It is a low level, multi-protocol data movement service, responsible for reliable bulk transfer of files from one site to another while allowing participating sites to control the network resource usage. A full description is available in the paper FTS3: New Data Movement Service For WLCG - IOPscience and in the documentation.
In addition to effectuating data transfer jobs, the service can also be used to monitor and log transfer jobs. Users can interact with FTS3 using command line tools, RESTful API calls or a web gui.
WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses a FTS3 endpoint as transfer engine.
More information about WebFTS is at http://fts3-docs.web.cern.ch/fts3-docs/docs/webfts.html
The service order will be handled by the support team.
- User who apply to use the service should have already have a valid Certificate. Otherwise, Getting a user certificate
- Support for new VOs must be explicitly requested as they need to be configured in WebFTS and on the FTS cluster nodes (voms files)
- User guide for command Line Tools: http://fts3-docs.web.cern.ch/fts3-docs/docs/cli.html
- User guide for REST API: http://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/index.html
- User guide for WebFTS: http://fts3-docs.web.cern.ch/fts3-docs/docs/webfts.html
Service Provision and Support
The service is provided by two providers: CERN and STFC/UKRI, and EGI has signed OLAs with them.
- CERN provides the following access instances:
- an FTS3 instance at https://fts3-public.cern.ch:8446/. fts3-public.cern.ch is deployed as a load-balanced alias across a number of machines (5 as of 2018/09/19). FTS3 service offers multiple interfaces: Command Line Interface (CLI), API and web interface using WebFTS.
- a WebFTS instance linked to their FTS3 instance: https://webfts.cern.ch/. It is a single instance. WebFTS is a web-based file transfer and management solution which allows users to invoke reliable, managed data transfers on distributed infrastructures. WebFTS uses an FTS3 endpoint as transfer engine.
FTS3 can be used on its own. WebFTS is interacting with a specific FTS3 instance.
- STFC/UKRI provides the following access instance:
- FTS3: https://lcgfts3.gridpp.rl.ac.uk:8446/. This is provided as a HA Proxy load-balanced alias across a pool of servers
Information about technology supports and the help desk can be found at https://wiki.egi.eu/wiki/GGUS:EGI_Data_Transfer_Service_FAQ
The ELIXIR experience
A webFTS instance is customised for ELIXIR is available at https://fts3.du2.cesnet.cz/index.php It is integrated with ELIXIR AAI and supports OIDC. It also has interfaces with RCauth Online CA. VO attributes are integrated.
The use case is described in Enable Federated Authentication with FTS
Experience from ELIXIR shows:
- The FTS3 service seems to fulfil most of the requirements that we have for an ELIXIR data transfer service: multi-protocol, monitoring and logging, checksums, API, web gui, …
- The software is free software (Apache License, Version 2.0) and seems straight-forward to deploy. All components are available in the RHEL/CENTOS EPEL repository. Installation and configuration guide.
- The fts3 development team is very approachable.
- Some development is needed if we want to make the webfts interface connect to the ELIXIR credential translation service for proxy certificates.
- It seems like logs of transfers jobs are per default public. This might be a problem in some cases (e.g. pharma companies that want to hide what they are working on).
For more information please refer to ELIXIR Test of FTS3 for transfers
What can FTS talk to?
It mainly talks to XrootD, HTTPS, GridFTP and possibly some S3
How a VO gets these endpoints ready to accept FTS transfers?
This needs to ask the FTS admin to enable the VO on their site. There is no need to register any endpoints and the authentication is done using X509 (so sites needs to use a supported protocol with X509 for authN/authZ)
What's the roadmap for Check-in integration?
We don’t have yet a date, for the CLI I don’t really except we have something anytime soon (We are not funding the FTS services), but for WebFTS RAL/STFC may try to do it but no concrete plan
- FTS documentation: http://fts3-docs.web.cern.ch/fts3-docs/
- FTS source code: http://gitlab.cern.ch/fts/
- FTS Main site: http://fts.web.cern.ch/
- Tutorial: https://webfts.cern.ch/