Difference between revisions of "APEL/Storage"
Line 64: | Line 64: | ||
# cat /etc/cron.daily/dmlite-StAR-accounting | # cat /etc/cron.daily/dmlite-StAR-accounting | ||
#!/bin/sh | #!/bin/sh | ||
/bin/mkdir -p /var/spool/apel/outgoing/`date +%Y%m%d` && /usr/share/dmlite/StAR-accounting/star-accounting.py --reportgroups --dbhost=[hostname] --dbuser= | /bin/mkdir -p /var/spool/apel/outgoing/`date +%Y%m%d` && /usr/share/dmlite/StAR-accounting/star-accounting.py --reportgroups --dbhost=[hostname] --dbuser=[username] --dbpwd=[password] --nsdbname=cns_db --dpmdbname=dpm_db --site=[site name] > /var/spool/apel/outgoing/`date +%Y%m%d`/`date +%Y%m%d%H%M%S` && ssmsend | ||
You can check that your data is being received by the Accounting Repository here: [http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html Sites publishing storage accounting records] (updated daily) | You can check that your data is being received by the Accounting Repository here: [http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html Sites publishing storage accounting records] (updated daily) |
Revision as of 12:26, 6 July 2020
Storage space usage accounting in APEL is based on the StAR (Storage Accounting Record) developed by EMI in conjunction with the OGF Usage Record Work Group (UR-WG). The format is documented here.
EMI delivered StAR solutions for dCache and DPM in EMI-3. In both cases the storage service queries its database at a site and extracts data to populate StAR usage records. The site then uses SSM as a transport method to send the StAR records via the EGI Messaging Service to APEL's central Accounting Repository.
Deployment Instructions
Get the accounting script
Follow the link for your product:
DPM - you need to run 1.8.7 or higher. Please be sure to have installed the star-accounting.py script v1.0.4 at least (here).
On the command line which runs the script, add --site="your-GOCDB-sitename"
(replacing the quoted section).
Get the SSM software
The APEL SSM software can be installed from the yum repo or by downloading an RPM or tarball. See here for more information.
Get authorised
To authorise your host to publish storage accounting see the section on configuring GOCDB.
Configure GOCDB
You need to add a new service for that host to GOCDB with the service type "eu.egi.storage.accounting" and the correct host certificate DN. The Accounting Repository takes up to an hour to update its ACL from GOCDB and the Message Brokers take up to 4 hours. If you get warnings in your SSM log about invalid user name or password you can just retry again after a period. If this persists for over 4 hours, then do open a GGUS ticket.
Configure SSM
To configure apel-ssm for sending accounting records edit the file /etc/apel/sender.cfg, and modify some parameters:
- In the
[broker]
section, you need:
network: PROD
(Not this: )
network: TEST-NWOB
Also set:
use_ssl: true
- In the
[certificates]
section, you need to set the host certificate to be used by the sender:
certificate: /path/to/your/host/certificate
key: /path/to/your/key
- In the
[messaging]
section, you need:
For production publishing -> destination: /queue/global.accounting.storage.central
For test publishing -> destination: /queue/global.accounting.test.storage.central
As described here: https://wiki.egi.eu/wiki/APEL/SSM2Configuration.
path:
set to the directory to which the StAR records are written by your storage software.
dCache and DPM should send one StAR record per VO. Please set up a cron to run once per day.
Running the Accounting Software
Create a cron job to run your accounting script followed by the SSM sender. We recommend that you send storage accounting data once per day. There will be a delay of up to 24 hours before you see the data you have sent reflected in the Accounting Portal.
- Example
# cat /etc/cron.daily/dmlite-StAR-accounting #!/bin/sh /bin/mkdir -p /var/spool/apel/outgoing/`date +%Y%m%d` && /usr/share/dmlite/StAR-accounting/star-accounting.py --reportgroups --dbhost=[hostname] --dbuser=[username] --dbpwd=[password] --nsdbname=cns_db --dpmdbname=dpm_db --site=[site name] > /var/spool/apel/outgoing/`date +%Y%m%d`/`date +%Y%m%d%H%M%S` && ssmsend
You can check that your data is being received by the Accounting Repository here: Sites publishing storage accounting records (updated daily)