Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Preview 2.29.0"

From EGIWiki
Jump to navigation Jump to search
Line 2: Line 2:
Back to [https://wiki.egi.eu/wiki/Preview_Repository#Repository_configuration repository configuration]
Back to [https://wiki.egi.eu/wiki/Preview_Repository#Repository_configuration repository configuration]


= ARC 6.5.0 and 6.6.0 =
= ARC 6.8.0 =


The [http://www.nordugrid.org/arc/ Advanced Resource Connector (ARC) middleware] is an Open Source software solution to enable distributed computing infrastructures with the emphasis on processing large volumes of data. ARC provides an abstraction layer over computational resources, complete with input and output data movement functionalities. The security model of ARC is identical to that of Grid solutions, relying on delegation of user credentials and the concept of Virtual Organisations. ARC also provides client tools, as well as API in C++, Python and Java.
The [http://www.nordugrid.org/arc/ Advanced Resource Connector (ARC) middleware] is an Open Source software solution to enable distributed computing infrastructures with the emphasis on processing large volumes of data. ARC provides an abstraction layer over computational resources, complete with input and output data movement functionalities. The security model of ARC is identical to that of Grid solutions, relying on delegation of user credentials and the concept of Virtual Organisations. ARC also provides client tools, as well as API in C++, Python and Java.


'''[http://www.nordugrid.org/arc/releases/6.5/ Release notes: 6.5.0]'''
We are pleased to announce the release of ARC 6.8.0. This release comes with two important changes that affect the update, namely changes in the accounting system, and a dependency on xrootd5. '''Both will need manual steps when updating to ARC 6.8.0''', the latter only if your CE has the nordugrid-plugins-xrootd package installed.


This release comes with quite a large set of enhancements and improvements.
'''Accounting changes'''


Bug [http://bugzilla.nordugrid.org/show_bug.cgi?id=3841 3841] (jobs piling up in FINISHING state) which one known site suffers with, is proving resilient. We believe that a busy file system hosting the job's session directory had strong negative impact on the data staging. This needs more investigation and future development to optimize the load on session directories.
The largest change in this release is the removal of the pre 6.4.0 accounting system. Since the accounting subsystem changes in 6.4.0 where we went from files to database, the old system has lived alongside the new. From 6.8.0 the legacy code is all removed. For more details of the consequences, and '''especially changes in arc.conf''',
http://www.nordugrid.org/arc/arc6/admins/reference.html#configuration-options-removed-in-arc-6-8-0


However bug [http://bugzilla.nordugrid.org/show_bug.cgi?id=3890 3890] related to xrootd transfers, which had to be reverted in [http://www.nordugrid.org/arc/releases/6.4.1/release_notes_6.4.1.html release 6.4.1], is now solved.
If you want to read more about the accounting changes, please consult this file: http://www.nordugrid.org/documents/jura-ng.html
And finally http://www.nordugrid.org/documents/arc6/admins/details/accounting-ng.html for the description of the new system.  


Of enhancements and new features we would like to highlight:
Please be aware that you will need to '''make changes in your arc.conf for ARC to start with 6.8.0'''.


* Arcctl:
Options and blocks that are no longer supported in arc.conf:
**arcctl can now be installed standalone without A-REX. It comes with the test CA and 3rd party deploy (CA certs, VOMS) on board.
**As before you get the full arcctl if you install A-REX
*Client area enhancements:
**installation automated with standalone arcctl
**new consistent and streamlined submission endpoint selection options
**consistent target-area oriented globus plugins split
**bash completion for client tools
**documentation for clients!
*ARCHERY management:
**new JSON configuration for flexible services topology definition
*Technology preview:
**This release also includes the technology preview of a new functionality: The "Community-defined RTEs" ecosystem that enables automated software environment provisioning on ARC CEs by community software managers. The ecosystem consists of two layers: An ARCHERY-based software (RTE) registry and a new set of ARC control tool modules. More details can be found in [http://www.nordugrid.org/arc/arc6/admins/details/rtes_community.html documentation].


'''[http://www.nordugrid.org/arc/releases/6.6/release_notes_6.6.html Release notes: 6.6.0]'''
* In the [arex/jura] block:  
    * urdelivery_keepfailed (in [arex/jura] block)
* In the [arex/jura/sgas:targetname] block:
    * legacy_fallback
* In the [arex/jura/apel:targetname] block:
      * use_ssl
      * benchmark_type
      * benchmark_value
      * benchmark_description
      * legacy_fallback
* [arex/jura/archiving] block  (the whole block is deprecated)
      * archivedir
        * logfile
        * archivettl


In this release we offer the very first technology preview of the handling of OIDC tokens in ARC. With it comes a new configuration option enabling the feature. Please see the [http://www.nordugrid.org/documents/arc6/misc/oidc_tokens.html documentation] for how to use tokens with ARC.
 
'''Important note on xrootd version for sites using the nordugrid-arc-plugins-xrootd package'''
 
For sites who install the nordugrid-arc-plugins-xrootd package note that ARC 6.8.0 is built with '''xrootd 5'''.
On the date of the release xrootd 5 has not yet reached EPEL-stable which means that to install(update) ARC 6.8.0 on RHEL-systems, you must enable the epel-testing repo:
yum install(update) "nordugrid-arc*" --enablerepo=epel-testing
or alternatively install xrootd 5 before updating ARC.  
 
For more details on bugs and highlights, please consult the release notes: http://www.nordugrid.org/arc/releases/6.8/release_notes_6.8.html


'''Future Support of ARC 5-series'''
'''Future Support of ARC 5-series'''

Revision as of 15:46, 8 October 2020

Back to repository configuration

ARC 6.8.0

The Advanced Resource Connector (ARC) middleware is an Open Source software solution to enable distributed computing infrastructures with the emphasis on processing large volumes of data. ARC provides an abstraction layer over computational resources, complete with input and output data movement functionalities. The security model of ARC is identical to that of Grid solutions, relying on delegation of user credentials and the concept of Virtual Organisations. ARC also provides client tools, as well as API in C++, Python and Java.

We are pleased to announce the release of ARC 6.8.0. This release comes with two important changes that affect the update, namely changes in the accounting system, and a dependency on xrootd5. Both will need manual steps when updating to ARC 6.8.0, the latter only if your CE has the nordugrid-plugins-xrootd package installed.

Accounting changes

The largest change in this release is the removal of the pre 6.4.0 accounting system. Since the accounting subsystem changes in 6.4.0 where we went from files to database, the old system has lived alongside the new. From 6.8.0 the legacy code is all removed. For more details of the consequences, and especially changes in arc.conf, http://www.nordugrid.org/arc/arc6/admins/reference.html#configuration-options-removed-in-arc-6-8-0

If you want to read more about the accounting changes, please consult this file: http://www.nordugrid.org/documents/jura-ng.html And finally http://www.nordugrid.org/documents/arc6/admins/details/accounting-ng.html for the description of the new system.

Please be aware that you will need to make changes in your arc.conf for ARC to start with 6.8.0.

Options and blocks that are no longer supported in arc.conf:

* In the [arex/jura] block: 
    * urdelivery_keepfailed (in [arex/jura] block)
* In the [arex/jura/sgas:targetname] block: 
    * legacy_fallback 
* In the [arex/jura/apel:targetname] block: 
      * use_ssl
      * benchmark_type
      * benchmark_value
      * benchmark_description
      * legacy_fallback
* [arex/jura/archiving] block  (the whole block is deprecated)
      * archivedir
       * logfile
       * archivettl


Important note on xrootd version for sites using the nordugrid-arc-plugins-xrootd package

For sites who install the nordugrid-arc-plugins-xrootd package note that ARC 6.8.0 is built with xrootd 5. On the date of the release xrootd 5 has not yet reached EPEL-stable which means that to install(update) ARC 6.8.0 on RHEL-systems, you must enable the epel-testing repo:

yum install(update) "nordugrid-arc*" --enablerepo=epel-testing

or alternatively install xrootd 5 before updating ARC.

For more details on bugs and highlights, please consult the release notes: http://www.nordugrid.org/arc/releases/6.8/release_notes_6.8.html

Future Support of ARC 5-series

Now that ARC 6 is released, we will only provide security updates of ARC 5.

In particular:

  • No new feature development is planned or going on for ARC5 and no bug-fixing development will happen on ARC5 code base in the future except for security issues.
  • Security fixes for ARC5 will be provided till end of June 2020.
  • Production Sites already running ARC 5 will be able to get deployment and configuration troubleshooting help via GGUS till end June 2021. This we call "operational site support".
  • ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.

BDII 5.5.26

The Berkeley Database Information Index (BDII) consists of two or more standard LDAP databases that are populated by an update process. Port forwarding is used to enable one or more databases to serve data while one database is being refreshed. The databases are refreshed cyclically. Any incoming connection is forwarded to the most recently updated database, while old connections are allowed to linger until it is the turn of their database to be refreshed and restarted. The update process obtains LDIF from either doing an ldapsearch on LDAP URLs or by running a local script (given by a URL with "file" protocol) that generates LDIF. The LDIF is then inserted into the LDAP database. Options exist to update the list of LDAP URLs from a web page and to use an LDIF file from a web page to modify the data before it is inserted into the database.

See more information here

Release notes: https://github.com/EGI-Foundation/bdii/releases/tag/v5.2.26

glite-info-update-endpoints 3.0.2

This component is used with Top BDII and is intented to update LDAP endpoits for EGI. BDII documentation is available here: https://gridinfo-documentation.readthedocs.io/

glite-info-update-endpoints is a cron job that runs every hour to download the list of site BDII URLs that are going to be used by the top level BDII to publish their resources.

The script uses the /etc/glite/glite-info-update-endpoints.conf file which by default is configured to use EGI's list of site BDIIs. The list of site BDIIs is taken from the EGI GOCDBs.

See more information here.

Release notes: https://github.com/EGI-Foundation/glite-info-update-endpoints/releases/tag/v3.0.2

lcg-info 1.12.5

This command line tool allows the user to query the WLCG/EGI LDAP-based information system. More information here.

Release notes:

  • Bug fix for GGUS 140478: Try all host addresses after a configurable timeout

STORM 1.11.18

The StoRM Product Team is pleased to announce the release of StoRM 1.11.18 that includes the following updated components:

This release introduces the support for CentOS 7 for all StoRM components.

It also provides fixes to some outstanding bugs, in particular:

  • fixes errors on published storage space occupancy in case multiple storage area shares the same VO;
  • fixes not published WebDAV endpoints when latest logic is used;
  • fixes not dropped Authorization header in WebDAV TPC redirects;
  • fixes leaked file descriptors when Conscrypt is enabled on StoRM WebDAV;
  • sets correctly HTTP content-length for large files;
  • fixes errors on transferred files through GridFTP that leave empty files with an adler32 checksum for a non-empty file;
  • fixes KillMode on GridFTP systemd unit.

It also provides several improvements, in particular:

  • fixes wrong ERROR log messages when file does not exist on srmRm requests;
  • changes the way info provider checks if Backend is running;
  • introduces a Background DU Service (disabled by default) that periodically updates the storage space info for non-GPFS storage areas (read more info here);
  • adds Date and thread pools metrics in the metrics logged info;
  • updates spring boot to 2.2.6 release for StoRM WebDAV;
  • adds SystemD support for StoRM Backend and StoRM Frontend.

Please, follow the upgrade instructions.

Starting from StoRM v1.11.18 StoRM production repository has been migrated. Read how to install/upgrade StoRM repositories into the Downloads section.

Read the release notes for more details.