APEL/Server

From EGIWiki
Jump to: navigation, search

Current Status

The APEL accounting repository receives records from sites running the APEL client and also from sites running SSM, in this case the sites send either Job Records or Summary records to the new APEL server and these records are integrated with those in the existing APEL accounting repository from where they are retrieved by the EGI Accounting Portal.

New APEL Overview

  • Sites which do not run the APEL client will send summary records, via SSM, which will be inserted in the Summaries table in the central APEL database.
  • The APEL clients will send Job records, via SSM, which will be inserted in the JobRecords table in the central APEL database.
  • Summary records will be created from the data in the JobRecords table. These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table).
  • The records from external clients in the Summaries table will be inserted (replaced) into the SuperSummaries table.
  • The SpecInt data will also be extracted and stored in a SpecInt table.
  • The relevant Summary and SpecInt data will then be sent to the Accounting Portal using another instance of the SSM.

New APEL Server and Accounting Portal

APEL Server Design

The new APEL server is written in Python, the database is MySQL. It has the following components:

  • SSM
  • Record loader
  • Database
  • Summarizer
  • Record publisher
  • Authentication
  • Monitoring

Test server

Other notes

SSM

For all documentation on the SSM, see APEL/SSM

Here is the document about APEL and messaging: File:APEL-Messaging-v2.2.pdf

Here is the discussion about the APEL/MessagingProtocol.

Record loader

This has the task of taking the received messages and inserting the records in the database. At present there are three message types defined:

  • Job Record messages
  • Summary Record messages
  • Sync Record messages

The header in the message defines the type of records the message will contain and the version of the format. This will enable us to identify newer versions if they are defined.

This simple model allows the definition of alternative message types for different records which the record loader can then be configured to handle, for example, there will be a SpecIntHistory Record message type which is a requirement of the Accounting portal.

APEL Message Format

Database

Summarizer

This processes the Job Records and Summary Records to create the SuperSummaries (User and CPU summaries) and SpecIntHistory tables required by the portal.

Record publisher

This takes the information from a database and produces messages to be sent to the portal from the server using an instance of the SSM. (The Record publisher will also be used by the new version of the APEL client to create messages from the APEL client database to be sent to the APEL server.)

Authentication

This verifies that the senders of the messages via the SSM are known to APEL.

Monitoring

Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type. These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.