Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "APEL/Storage"

From EGIWiki
Jump to navigation Jump to search
 
(21 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Storage accounting in APEL is based on the StAR (Storage Accounting Record) developed by EMI in conjuction with the OGF Usage Record Work Group (UR-WG). The format is documented [http://cds.cern.ch/record/1452920/files/GFD.201.pdf here].  
Storage space usage accounting in APEL is based on the StAR (Storage Accounting Record) developed by EMI in conjunction with the OGF Usage Record Work Group (UR-WG). The format is [http://cds.cern.ch/record/1452920/files/GFD.201.pdf documented here].  


EMI delivered StAR solutions for dcache and DPM in EMI-3. In both cases the storage service queries its database at a site and extracts data to populate StAR usage records. The site then uses SSM as a transport method to send the StAR records across the EGI Messaging Service to APEL's central accounting repository. 
EMI delivered StAR solutions for dCache and DPM in EMI-3. In both cases the storage service queries its database at a site and extracts data to populate StAR usage records. The site then uses SSM as a transport method to send the StAR records via the EGI Messaging Service to APEL's central Accounting Repository.  
 
<br>


== Deployment Instructions  ==
== Deployment Instructions  ==
Line 9: Line 7:
=== Get the accounting script  ===
=== Get the accounting script  ===


Follow the link for your product.
Follow the link for your product:


[[APEL/Dcache|dcache]]  
[[APEL/Dcache|dCache]]  


[https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/Dev/Recipes/EGI_APEL_Accounting DPM]&nbsp;- you need to run 1.8.7 or higher. For earlier releases it can be found here http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py  
[https://twiki.cern.ch/twiki/bin/view/DPM/DpmSetupPuppetInstallation#DPM_Accounting DPM] - you need to run 1.8.7 or higher. Please be sure to have installed the star-accounting.py script v1.0.4 at least ([https://gitlab.cern.ch/lcgdm/dmlite/-/blob/master/scripts/StAR-accounting/star-accounting.py here]).


On the command line which runs the script, add --site="your-GOCDB-sitename"  
On the command line which runs the script, add <code>--site="your-GOCDB-sitename"</code> (replacing the quoted section).


=== Get the SSM software  ===
=== Get the SSM software  ===


The APEL SSM software can be installed from the yum repo or by downloading an rpm or tarball. See here for more information: [[APEL/SSM|SSM]]  
The APEL SSM software can be installed from the yum repo or by downloading an RPM or tarball. [[APEL/SSM|See here]] for more information.


=== Get authorised  ===
=== Get authorised  ===


While storage accounting publishing is still in test mode please send the host certificate DN to [mailto:apel-admins@stfc.ac.uk apel-admins@stfc.ac.uk] and ask them to authorise you for storage publishing.
To authorise your host to publish storage accounting see the section on configuring GOCDB.
 
=== Configure GOCDB  ===
 
You need to add a new service for that host to GOCDB with the service type "eu.egi.storage.accounting" and the correct host certificate DN. The Accounting Repository takes up to an hour to update its ACL from GOCDB and the Message Brokers take up to 4 hours. If you get warnings in your SSM log about invalid user name or password you can just retry again after a period. If this persists for over 4 hours, then do open a GGUS ticket.


=== Configure SSM  ===
=== Configure SSM  ===


To configure apel-ssm for sending acounting records we have to edit the file /etc/apel/sender.cfg, and modify some parameters:  
'''2020-03 UPDATE''': '''Please follow the instructions https://github.com/apel/ssm/blob/dev/migrating_to_ams.md'''
 
<strike>To configure apel-ssm for sending accounting records edit the file /etc/apel/sender.cfg, and modify some parameters:
 
*In the '''<code>[broker]</code>''' section, you need:


* In the '''[broker]''' section, you need:
<code>network: PROD</code>
<code>network: PROD</code>
and '''not''':
<code>network: TEST-NWOB</code>
as for initial testing. Also set:


<code>use_ssl: false</code>
(Not this: <strike> <code>network: TEST-NWOB</code></strike>)
* In the '''[certificates]''' section, you need to set the host certificate to be used by the sender:  
 
Also set:
 
<code>use_ssl: true</code>  
 
*In the '''<code>[certificates]</code>''' section, you need to set the host certificate to be used by the sender:
 
<code>certificate: /path/to/your/host/certificate</code><br> <code>key: /path/to/your/key</code>
 
*In the '''<code>[messaging]</code>''' section, you need:


<code>certificate: /path/to/your/host/certificate</code><br>
For production publishing -> <code>destination: /queue/global.accounting.storage.central</code>
<code>key: /path/to/your/key</code>


* In the '''[messaging]''' section, you need:
For test publishing -> <code>destination: /queue/global.accounting.test.storage.central</code>
<code>destination: /queue/global.accounting.test.storage.central</code> -  as described here: ​https://wiki.egi.eu/wiki/APEL/SSM2Configuration.


dcache and dpm should send one StAR record per VO. Please set up a cron to run once per day.
As described here: ​https://wiki.egi.eu/wiki/APEL/SSM2Configuration.


=== Configure GOCDB ===
<code>path:</code> set to the directory to which the StAR records are written by your storage software.  
When storage accounting moves to the production message broker network then you will need to add a servicetype to GOCDB for the host which runs SSM. The servicetype has not yet been defined but it will require the host certificate DN.  


== Running the Accounting Software ==
dCache and DPM should send one StAR record per VO. Please set up a cron to run once per day.</strike>


Create a cron job to run your accounting script followed by the SSM sender. We recommend that you send storage accounting data once per day. There will be a delay of up to 24 hours before you see the data you have sent in the Accounting Portal.
== Running the Accounting Software  ==
 
Create a cron job to run your accounting script followed by the SSM sender. We recommend that you send storage accounting data once per day. There will be a delay of up to 24 hours before you see the data you have sent reflected in the Accounting Portal.
 
* Example
# cat /etc/cron.daily/dmlite-StAR-accounting
#!/bin/sh
/bin/mkdir -p /var/spool/apel/outgoing/`date +%Y%m%d` && /usr/share/dmlite/StAR-accounting/star-accounting.py --reportgroups --dbhost=[hostname] --dbuser=[username] --dbpwd=[password] --nsdbname=cns_db --dpmdbname=dpm_db --site=[site name] > /var/spool/apel/outgoing/`date +%Y%m%d`/`date +%Y%m%d%H%M%S` && ssmsend
 
You can check that your data is being received by the Accounting Repository here: [http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html Sites publishing storage accounting records] (updated daily)


== Storage Data at the EGI Accounting Portal ==
== Storage Data at the EGI Accounting Portal ==
[http://accounting-devel.egi.eu/storage.php? Accounting Portal Prototype view]&nbsp;
Note that the default view of 'Minimum Resource TeraBytes Consumed' will probably show zero when you start to publish. Chose Average or Maximum instead


[http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html A list of sites publishing storage accounting, with last publish dates.]&nbsp;
[http://accounting-devel.egi.eu/storage.php Accounting Portal Prototype view]

Latest revision as of 11:19, 17 March 2021

Storage space usage accounting in APEL is based on the StAR (Storage Accounting Record) developed by EMI in conjunction with the OGF Usage Record Work Group (UR-WG). The format is documented here.

EMI delivered StAR solutions for dCache and DPM in EMI-3. In both cases the storage service queries its database at a site and extracts data to populate StAR usage records. The site then uses SSM as a transport method to send the StAR records via the EGI Messaging Service to APEL's central Accounting Repository.

Deployment Instructions

Get the accounting script

Follow the link for your product:

dCache

DPM - you need to run 1.8.7 or higher. Please be sure to have installed the star-accounting.py script v1.0.4 at least (here).

On the command line which runs the script, add --site="your-GOCDB-sitename" (replacing the quoted section).

Get the SSM software

The APEL SSM software can be installed from the yum repo or by downloading an RPM or tarball. See here for more information.

Get authorised

To authorise your host to publish storage accounting see the section on configuring GOCDB.

Configure GOCDB

You need to add a new service for that host to GOCDB with the service type "eu.egi.storage.accounting" and the correct host certificate DN. The Accounting Repository takes up to an hour to update its ACL from GOCDB and the Message Brokers take up to 4 hours. If you get warnings in your SSM log about invalid user name or password you can just retry again after a period. If this persists for over 4 hours, then do open a GGUS ticket.

Configure SSM

2020-03 UPDATE: Please follow the instructions https://github.com/apel/ssm/blob/dev/migrating_to_ams.md

To configure apel-ssm for sending accounting records edit the file /etc/apel/sender.cfg, and modify some parameters:

  • In the [broker] section, you need:

network: PROD

(Not this: network: TEST-NWOB)

Also set:

use_ssl: true

  • In the [certificates] section, you need to set the host certificate to be used by the sender:

certificate: /path/to/your/host/certificate
key: /path/to/your/key

  • In the [messaging] section, you need:

For production publishing -> destination: /queue/global.accounting.storage.central

For test publishing -> destination: /queue/global.accounting.test.storage.central

As described here: ​https://wiki.egi.eu/wiki/APEL/SSM2Configuration.

path: set to the directory to which the StAR records are written by your storage software.

dCache and DPM should send one StAR record per VO. Please set up a cron to run once per day.

Running the Accounting Software

Create a cron job to run your accounting script followed by the SSM sender. We recommend that you send storage accounting data once per day. There will be a delay of up to 24 hours before you see the data you have sent reflected in the Accounting Portal.

  • Example
# cat /etc/cron.daily/dmlite-StAR-accounting
#!/bin/sh
/bin/mkdir -p /var/spool/apel/outgoing/`date +%Y%m%d` && /usr/share/dmlite/StAR-accounting/star-accounting.py --reportgroups --dbhost=[hostname] --dbuser=[username] --dbpwd=[password] --nsdbname=cns_db --dpmdbname=dpm_db --site=[site name] > /var/spool/apel/outgoing/`date +%Y%m%d`/`date +%Y%m%d%H%M%S` && ssmsend

You can check that your data is being received by the Accounting Repository here: Sites publishing storage accounting records (updated daily)

Storage Data at the EGI Accounting Portal

Accounting Portal Prototype view