Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "CVMFS Task Force"

From EGIWiki
Jump to navigation Jump to search
Line 176: Line 176:
|-
|-
| OSG  
| OSG  
|  
| xxx
| replicates the non-LHC repositories from RAL-LCG2  
| replicates the non-LHC repositories from RAL-LCG2  
|  
|  

Revision as of 15:52, 3 October 2013

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security



Coordinator: Catalin Condurache/NGI_UK

Meetings page: Agendas

Mailing list: cvmfs-tf (at) mailman.egi.eu

Members:

  • NGIs:
    • NGI FI - Ulf Tigerstedt
    • NGI HR - Luko Gjenero, Emir Imamagic
    • NGI IBERGRID - Goncalo Borges
    • OSG - Scott Teige
    • NGI ZA - Bruce Becker
    • NGI IT - Paolo Veronesi, Alessandro Constantini
    • NGI GRNET - Dimitris Dellis, Kostas Koumantaros
  • MPI VT - John Walsh
  • VOs:
    • VLEMED VO - Silvia Olabarriaga, Hurng-Chun Lee
    • BIOMED Johan Montagnat
    • WeNMR Alexandre Bonvin
    • AUGER VO - Jiri Chudoba
    • glast.org - Michael Kuss, Francesco Longo

Introduction

CVMFS (CERN Virtual Machine File System) has already proved to be a good solution for the distribution of application software across the resource centres, and several NGIs are offering CVMFS to national VOs.

During the April OMB in Manchester the idea of offering CVMFS as a service for all EGI VOs was discussed. It is about to have a CVMFS infrastructure that not only can be used within EGI, but also in collaboration with OSG for VOs that access resources hosted by both infrastructures).


NGI Status

NGI Status
NGI Ibergrid IBERGRID is already using CVMFS stratum 0 deployed at RAL as a service
NGI_ZA We were planning to deploy a stratum 0 in South Africa for VO sagrid applications, as well as those in use in other sub-Saharan countries.
NGI HR We are running CVMFS for our regional VO.
NGI FI We are one of the non-WLCG organizations running cvmfs.
OSG We recently implemented a CVMFS based service for the OSG. Our intended use case is for our virtual organizations and software group to distribute their content by this mechanism. Our design is quite simple, we allow a very few people write access and everyone read access. We are actively adding resources with access enabled, wide adoption being the goal.
NGI IT We have a very basic deployment for our catch-all VO, but it hasn't been very used until now.
NGI UK

We run stratum-0 repos (v2.0.15) for 5 VOs (mice, na62, hone, enmr.eu, phys.ibergrid.vo.eu). Our design allows installation jobs (run by VO_SGM) to write in an NFS area and then rsync to the CVMFS stratum-0 areas. For VOs not supported by GridPP UK (in terms of CPU allocations), we manually upload and maintain their CVMFS repos, but we are currently working to a web interface that will allow their own software management.

Also we run a CVMFS stratum-1 service (part of the LHC CVMFS infrastructure) which also includes replicas for the above mentioned VOs.

As plans, we look to migrate the stratum-0 nodes to v2.1.X on SL6. Also we work with CERN to replicate these 5 replicas on their stratum-1 service (3 out of 5 so far replicated). And looking to establish a network of stratum-1 servers to consolidate the non-LHC (or small) VOs cvmfs service infrastructure (tests to start soon with NIKHEF).

VO status

  • (1) is your community already familiar with CVMFS and is CVMFS already used in some form to support production activities?
  • (2) are you interested in trying CVMFS?
VO Status
VLEMED VO (1) used in some form to support production activities?

we were not until this email, but we asked around. The Dutch NGI is taking action already for non HEP users. we look forward to their findings.

(2) yes.

Johan Montagnat (1) No, we were not aware of this tool before.

(2) There is a clear need for easily deploying software packages grid-wise in the LS community. The existing solution of populating VO software space per site is often considered tedious, and leading to incoherent state when software cannot be installed on some sites due to installation job issues. So an alternative and global scale solution would be welcome. We would need to know a bit more on CVMFS though. From the slides shown, it is not clear how the software registered on the central CVMFS root directory is accessible from the sites / worker nodes in particular. (Is the CVMFS root mounted from all worker nodes? What is the performance impact of accessing to this remote software?).

Alexandre Bonvin 1. we are making use of it already. Very nice and simple. Only a few sites have that in place at this time
Jiri Chudoba - auger 1. The auger VO plans to use CVMFS. We have not yet built our own system of servers and we would like to

try "central" service.

Michael Kuss - glast.org 2. Participated in the webinar and would like to give CVMFS a try.

Objectives

Milestones

Actions

Initial working plan

It has been proposed following CVMFS-TF kick-off meeting (15 August 2013):

- Collect the expression of interest from VOs.

- Understand sites availability to install clients.

- Understand NGI availability to install regional squid.

- Mirroring between sites hosting stratus0 repositories. Creating a network of stratum1 servers.

Webinar - 5 September 2013

Aimed to provide needed information to NGIs/sites and user communities about the technical details and possible architecture.

The complete abstract for the presentation and registration details are available on INDICO at: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1809

QA webinar chat window: https://wiki.egi.eu/wiki/File:CVMFS_webinar_QA_chat_window.doc

Recording - https://documents.egi.eu/public/ShowDocument?docid=1932

EGI Technical Forum - Madrid, 16 - 20 September 2013

A presentation on "CVMFS for EGI VOs" has been given during the User Community Board session (https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1851#20130917) followed by Q&A and other discussions about creating the EGI CVMFS infrastructure.

Express of interest has been gathered from VOs representatives (biomed, auger, vlemed, glast.org) during the Technical Forum and steps of actions have been agreed.

Results

Site Stratum-0 Stratum-1 Squid Clients
EGI CVMFS Deployment Status
RAL-LCG2

mice, na62, hone, phys.vo.ibergrid.eu, enmr.eu - yes

hyperk.org, biomed, vlemed, auger, glast.org - to be done

replicated by the LHC Stratum-1; separate Stratum-1 planned to be deployed yes all VOs
Tier-2s UK as requested (mice, na62)
ZA_UJ enmr.eu
OSG xxx replicates the non-LHC repositories from RAL-LCG2
CERN replicates mice, na62, hone repos from RAL-LCG2

Useful Links

CVMFS home page http://cernvm.cern.ch/portal/filesystem

CVMFS - Beyond LHC Computing https://indico.egi.eu/indico/getFile.py/access?contribId=7&resId=0&materialId=slides&confId=1235

RAL Tier1 CVMFS https://www.gridpp.ac.uk/wiki/RAL_Tier1_CVMFS

CVMFS for non LHC VOs https://www.gridpp.ac.uk/wiki/RALnonLHCCVMFS