Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Preview 2.27.0"

From EGIWiki
Jump to navigation Jump to search
(Created page with "Category:Middleware Back to [https://wiki.egi.eu/wiki/Preview_Repository#Repository_configuration repository configuration] TO UPDATE = ARC 6.4.1 = The [http://www.nordu...")
 
 
(15 intermediate revisions by the same user not shown)
Line 2: Line 2:
Back to [https://wiki.egi.eu/wiki/Preview_Repository#Repository_configuration repository configuration]
Back to [https://wiki.egi.eu/wiki/Preview_Repository#Repository_configuration repository configuration]


TO UPDATE
= ARC 6.5.0 and 6.6.0 =
= ARC 6.4.1 =


The [http://www.nordugrid.org/arc/ Advanced Resource Connector (ARC) middleware] is an Open Source software solution to enable distributed computing infrastructures with the emphasis on processing large volumes of data. ARC provides an abstraction layer over computational resources, complete with input and output data movement functionalities. The security model of ARC is identical to that of Grid solutions, relying on delegation of user credentials and the concept of Virtual Organisations. ARC also provides client tools, as well as API in C++, Python and Java.
The [http://www.nordugrid.org/arc/ Advanced Resource Connector (ARC) middleware] is an Open Source software solution to enable distributed computing infrastructures with the emphasis on processing large volumes of data. ARC provides an abstraction layer over computational resources, complete with input and output data movement functionalities. The security model of ARC is identical to that of Grid solutions, relying on delegation of user credentials and the concept of Virtual Organisations. ARC also provides client tools, as well as API in C++, Python and Java.


'''Release notes'''
'''[http://www.nordugrid.org/arc/releases/6.5/ Release notes: 6.5.0]'''


This is an express release to repair two blocker bugs discovered in ARC.
This release comes with quite a large set of enhancements and improvements.


An improvement to the handling of xrootd in ARC 6.3.0 (Bugzilla ticket 3870) turned out to break xrootd transfers (see Bugzilla ticket 3890). As the reimplementation will take too long to properly test, we instead roll back to using the old xrootd implementation.
Bug [http://bugzilla.nordugrid.org/show_bug.cgi?id=3841 3841] (jobs piling up in FINISHING state) which one known site suffers with, is proving resilient. We believe that a busy file system hosting the job's session directory had strong negative impact on the data staging. This needs more investigation and future development to optimize the load on session directories.
Accounting: Since ARC 6.4.0 the accounting could not handle cases when a job was submitted by a proxy without a VO name. Protection against zero values are now in place.
In addition we have sneaked in a small fix related to PBS, allowing queue names to include a dot.
For information regarding all the changes in the regular 6.4.0 release, including the new accounting subsystem, please see the [http://www.nordugrid.org/arc/releases/6.4.1/release_notes_6.4.1.html release notes].


'''A new accounting system implementation'''.
However bug [http://bugzilla.nordugrid.org/show_bug.cgi?id=3890 3890] related to xrootd transfers, which had to be reverted in [http://www.nordugrid.org/arc/releases/6.4.1/release_notes_6.4.1.html release 6.4.1], is now solved.


Since ARC 6.4.0, the ARC-CE accounting subsystem has been reimplemented. The central point of the next generation Accounting Subsystem is a local SQLite accounting database that stores all the A-Rex Accounting Record (AAR) information. AAR defines all accounting information stored about a single ARC CE job.
Of enhancements and new features we would like to highlight:


The new system improves scalability, eliminates bottlenecks of the legacy architecture and provides much more information about the ARC CE jobs on site. The publishing and republishing of the records also have been improved, in particular APEL has received support for summary and sync messages.
* Arcctl:
**arcctl can now be installed standalone without A-REX. It comes with the test CA and 3rd party deploy (CA certs, VOMS) on board.
**As before you get the full arcctl if you install A-REX
*Client area enhancements:
**installation automated with standalone arcctl
**new consistent and streamlined submission endpoint selection options
**consistent target-area oriented globus plugins split
**bash completion for client tools
**documentation for clients!
*ARCHERY management:
**new JSON configuration for flexible services topology definition
*Technology preview:
**This release also includes the technology preview of a new functionality: The "Community-defined RTEs" ecosystem that enables automated software environment provisioning on ARC CEs by community software managers. The ecosystem consists of two layers: An ARCHERY-based software (RTE) registry and a new set of ARC control tool modules. More details can be found in [http://www.nordugrid.org/arc/arc6/admins/details/rtes_community.html documentation].


The switch to the new accounting is fully transparent except for one change in the benchmark values in the ARC configuration, see below. The documentation for the new accounting can be found [http://www.nordugrid.org/documents/arc6/admins/details/accounting-ng.html here].
'''[http://www.nordugrid.org/arc/releases/6.6/release_notes_6.6.html Release notes: 6.6.0]'''


Some important accounting details:
In this release we offer the very first technology preview of the handling of OIDC tokens in ARC. With it comes a new configuration option enabling the feature. Please see the [http://www.nordugrid.org/documents/arc6/misc/oidc_tokens.html documentation] for how to use tokens with ARC.
 
*The benchmark values should now be configured per-queue ''[queue:name]'' instead of the ''[arex/jura/apel]'' block. The configuration validator will prevent A-REX to start if the values are specified in the old way (pre 6.4.0).
*The new system automatically provides archiving of all accounting records in the database, therefore the old way of record archiving via the JURA ''[arex/jura/archiving]'' is DEPRECATED.
*APEL publishing now sends summary records instead of individual records by default.
*The ''arcctl accounting'' provides a new set of commands for flexible analysis of local accounting data. Old archived records can also be analyzed or republished using ''arcctl accounting legacy'' commands.
See [http://www.nordugrid.org/arc/releases/6.4/release_notes_6.4.html here] for more details.


'''Future Support of ARC 5-series'''
'''Future Support of ARC 5-series'''
Line 43: Line 45:
*ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.
*ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.


= CVMFS 2.7.0 =
= CVMFS 2.7.2 =


The [http://cernvm.cern.ch/portal/filesystem CernVM File System] provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a [https://github.com/libfuse/libfuse FUSE] module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs.
The [http://cernvm.cern.ch/portal/filesystem CernVM File System] provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a [https://github.com/libfuse/libfuse FUSE] module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs.
Line 49: Line 51:
'''Release notes'''
'''Release notes'''


CernVM-FS 2.4 is a feature release that comes with performance improvements, new functionality, and bugfixes. Please find detailed release notes in the [https://cvmfs.readthedocs.io/en/2.7/cpt-releasenotes.html technical documentation].
CernVM-FS 2.7.2 is a bugfix release. Please find detailed release notes in the [https://cvmfs.readthedocs.io/en/2.7/cpt-releasenotes.html technical documentation].


 
= dCache 5.2.20=
= dCache 5.2.13 =


[https://www.dcache.org/index.shtml dCache] provides a system for storing and retrieving huge amounts of data, distributed among a large number of heterogeneous server nodes, under a single virtual filesystem tree with a variety of standard access methods.
[https://www.dcache.org/index.shtml dCache] provides a system for storing and retrieving huge amounts of data, distributed among a large number of heterogeneous server nodes, under a single virtual filesystem tree with a variety of standard access methods.
Line 58: Line 59:
'''Detailed release notes''' on the official product site: https://www.dcache.org/downloads/1.9/release-notes-5.2.shtml
'''Detailed release notes''' on the official product site: https://www.dcache.org/downloads/1.9/release-notes-5.2.shtml


= gfal2 2.17.1 =
= frontier-squid 4.11.2 =
 
The frontier-squid software package is a patched version of the standard [http://squid-cache.org/ squid] http proxy cache software, pre-configured for use by the [https://twiki.cern.ch/twiki/bin/view/Frontier/WebHome Frontier distributed database caching system]. This installation is recommended for use by Frontier in the LHC CMS & ATLAS projects, and also works well with the [http://cernvm.cern.ch/portal/filesystem CernVM FileSystem]. Many people also use it for other applications as well
 
'''Release notes''': http://frontier.cern.ch/dist/rpms/frontier-squidRELEASE_NOTES
 
= gfal2 2.17.2 =


[http://dmc.web.cern.ch/projects/gfal-2/home GFAL] (Grid File Access Library ) is a C library providing an abstraction layer of the grid storage system complexity.
[http://dmc.web.cern.ch/projects/gfal-2/home GFAL] (Grid File Access Library ) is a C library providing an abstraction layer of the grid storage system complexity.
Line 64: Line 71:
The complexity of the grid is hidden from the client side behind a simple common POSIX API.
The complexity of the grid is hidden from the client side behind a simple common POSIX API.


Detailed release notes at http://dmc.web.cern.ch/release/gfal2-2.17.1
Detailed release notes at http://dmc.web.cern.ch/release/gfal2-2172
 
= STORM 1.11.17 =
The StoRM Product Team is pleased to announce the release of [http://italiangrid.github.io/storm/release-notes/StoRM-v1.11.17.html StoRM 1.11.17] that includes the following updated components:
*StoRM Backend [http://italiangrid.github.io/storm/release-notes/storm-backend-server/1.11.17/ v. 1.11.17]
*StoRM WebDAV [http://italiangrid.github.io/storm/release-notes/storm-webdav/1.2.1/ v. 1.2.1]
*StoRM GridFTP [http://italiangrid.github.io/storm/release-notes/storm-gridftp-dsi/1.2.3/ v. 1.2.3]
 
This release introduces the '''support on CentOS 7 for StoRM WebDAV and StoRM GridFTP'''. Please, follow the [http://italiangrid.github.io/storm/documentation/sysadmin-guide/1.11.17#upgrading upgrade instructions].
 
'''Read the [http://italiangrid.github.io/storm/release-notes/StoRM-v1.11.17.html release notes] for more details'''.


= xrootd 4.11.1 =
= xrootd 4.11.3 =


[http://xrootd.org/ XRootD software framework] is a fully generic suite for fast, low latency and scalable data access, which can serve natively any kind of data, organized as a hierarchical filesystem-like namespace, based on the concept of directory. As a general rule, particular emphasis has been put in the quality of the core software parts.
[http://xrootd.org/ XRootD software framework] is a fully generic suite for fast, low latency and scalable data access, which can serve natively any kind of data, organized as a hierarchical filesystem-like namespace, based on the concept of directory. As a general rule, particular emphasis has been put in the quality of the core software parts.


'''Detailled release notes''' at https://github.com/xrootd/xrootd/blob/v4.11.1/docs/ReleaseNotes.txt
'''Detailled release notes''' at https://github.com/xrootd/xrootd/blob/v4.11.3/docs/ReleaseNotes.txt

Latest revision as of 09:28, 8 May 2020

Back to repository configuration

ARC 6.5.0 and 6.6.0

The Advanced Resource Connector (ARC) middleware is an Open Source software solution to enable distributed computing infrastructures with the emphasis on processing large volumes of data. ARC provides an abstraction layer over computational resources, complete with input and output data movement functionalities. The security model of ARC is identical to that of Grid solutions, relying on delegation of user credentials and the concept of Virtual Organisations. ARC also provides client tools, as well as API in C++, Python and Java.

Release notes: 6.5.0

This release comes with quite a large set of enhancements and improvements.

Bug 3841 (jobs piling up in FINISHING state) which one known site suffers with, is proving resilient. We believe that a busy file system hosting the job's session directory had strong negative impact on the data staging. This needs more investigation and future development to optimize the load on session directories.

However bug 3890 related to xrootd transfers, which had to be reverted in release 6.4.1, is now solved.

Of enhancements and new features we would like to highlight:

  • Arcctl:
    • arcctl can now be installed standalone without A-REX. It comes with the test CA and 3rd party deploy (CA certs, VOMS) on board.
    • As before you get the full arcctl if you install A-REX
  • Client area enhancements:
    • installation automated with standalone arcctl
    • new consistent and streamlined submission endpoint selection options
    • consistent target-area oriented globus plugins split
    • bash completion for client tools
    • documentation for clients!
  • ARCHERY management:
    • new JSON configuration for flexible services topology definition
  • Technology preview:
    • This release also includes the technology preview of a new functionality: The "Community-defined RTEs" ecosystem that enables automated software environment provisioning on ARC CEs by community software managers. The ecosystem consists of two layers: An ARCHERY-based software (RTE) registry and a new set of ARC control tool modules. More details can be found in documentation.

Release notes: 6.6.0

In this release we offer the very first technology preview of the handling of OIDC tokens in ARC. With it comes a new configuration option enabling the feature. Please see the documentation for how to use tokens with ARC.

Future Support of ARC 5-series

Now that ARC 6 is released, we will only provide security updates of ARC 5.

In particular:

  • No new feature development is planned or going on for ARC5 and no bug-fixing development will happen on ARC5 code base in the future except for security issues.
  • Security fixes for ARC5 will be provided till end of June 2020.
  • Production Sites already running ARC 5 will be able to get deployment and configuration troubleshooting help via GGUS till end June 2021. This we call "operational site support".
  • ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.

CVMFS 2.7.2

The CernVM File System provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs.

Release notes

CernVM-FS 2.7.2 is a bugfix release. Please find detailed release notes in the technical documentation.

dCache 5.2.20

dCache provides a system for storing and retrieving huge amounts of data, distributed among a large number of heterogeneous server nodes, under a single virtual filesystem tree with a variety of standard access methods.

Detailed release notes on the official product site: https://www.dcache.org/downloads/1.9/release-notes-5.2.shtml

frontier-squid 4.11.2

The frontier-squid software package is a patched version of the standard squid http proxy cache software, pre-configured for use by the Frontier distributed database caching system. This installation is recommended for use by Frontier in the LHC CMS & ATLAS projects, and also works well with the CernVM FileSystem. Many people also use it for other applications as well

Release notes: http://frontier.cern.ch/dist/rpms/frontier-squidRELEASE_NOTES

gfal2 2.17.2

GFAL (Grid File Access Library ) is a C library providing an abstraction layer of the grid storage system complexity. The version 2 of GFAL tries to simplify at the maximum the file operations in a distributed environment. The complexity of the grid is hidden from the client side behind a simple common POSIX API.

Detailed release notes at http://dmc.web.cern.ch/release/gfal2-2172

xrootd 4.11.3

XRootD software framework is a fully generic suite for fast, low latency and scalable data access, which can serve natively any kind of data, organized as a hierarchical filesystem-like namespace, based on the concept of directory. As a general rule, particular emphasis has been put in the quality of the core software parts.

Detailled release notes at https://github.com/xrootd/xrootd/blob/v4.11.3/docs/ReleaseNotes.txt