Difference between revisions of "APEL/Server"

From EGIWiki
Jump to: navigation, search
(APEL Message Format)
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''This page is under construction and is not yet complete.'''
+
== Current Status ==
 +
The APEL accounting repository receives records from sites running the APEL client and also from sites running [[APEL/SSM|SSM]], in this case the sites send either Job Records or Summary records to the new APEL server and these records are integrated with those in the existing APEL accounting repository from where they are retrieved by the EGI Accounting Portal.
 
== New APEL Overview ==
 
== New APEL Overview ==
* Sites which do not run the APEL client will send summary records, via SSM, which will be inserted in the Summaries table in the central APEL database.   
+
* Sites which do not run the APEL client will send summary records, via [[APEL/SSM|SSM]], which will be inserted in the Summaries table in the central APEL database.   
 
* The APEL clients will send Job records, via SSM, which will be inserted in the JobRecords table in the central APEL database.   
 
* The APEL clients will send Job records, via SSM, which will be inserted in the JobRecords table in the central APEL database.   
 
* Summary records will be created from the data in the JobRecords table.  These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table).   
 
* Summary records will be created from the data in the JobRecords table.  These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table).   
Line 13: Line 14:
 
== APEL Server Design ==
 
== APEL Server Design ==
 
The new APEL server is written in Python, the database is MySQL.  It has the following components:
 
The new APEL server is written in Python, the database is MySQL.  It has the following components:
* SSM
+
* [[APEL/SSM|SSM]]
 
* Record loader
 
* Record loader
 
* Database
 
* Database
Line 20: Line 21:
 
* Authentication
 
* Authentication
 
* Monitoring
 
* Monitoring
 +
 +
==== Test server ====
 +
* [[APEL/Test_APEL_Server]]
 +
 +
==== Other notes ====
 +
* [[APEL/TimesAndDates]]
  
 
== SSM ==
 
== SSM ==
The SSM is the messaging system used by APEL to transmit messages.  It is written in Python and uses the STOMP protocol.
 
  
* [[APEL/SSMOverview|SSM Overview]]
+
For all documentation on the SSM, see [[APEL/SSM]]
* [[APEL/SSMInstallation|SSM installation instructions]]
+
 
* [[APEL/SSMProblems|Common SSM problems]]
+
Here is the document about APEL and messaging: [[File:APEL-Messaging-v2.2.pdf]]
  
==== External SSM Testing ====
+
Here is the discussion about the [[APEL/MessagingProtocol]].
[[APEL/APELSSMExternalTesting|APEL SSM Testing]]: If you are interested in testing the new STOMP interface or want to develop your own client.
 
  
 
== Record loader ==
 
== Record loader ==
Line 45: Line 50:
  
 
== Database ==
 
== Database ==
* [[APEL Database Tables]]
+
* [[APEL/DatabaseTables]]
  
 
== Summarizer ==
 
== Summarizer ==
Line 52: Line 57:
 
== Record publisher ==
 
== Record publisher ==
 
This takes the information from a database and produces messages to be sent to the portal from the server using an instance of the SSM.  (The Record publisher will also be used by the new version of the APEL client to create messages from the APEL client database to be sent to the APEL server.)
 
This takes the information from a database and produces messages to be sent to the portal from the server using an instance of the SSM.  (The Record publisher will also be used by the new version of the APEL client to create messages from the APEL client database to be sent to the APEL server.)
 
==== APEL Message Format ====
 
* [[APEL/MessageFormat|APEL Message Format]]
 
  
 
== Authentication ==
 
== Authentication ==
Line 63: Line 65:
 
== Monitoring ==
 
== Monitoring ==
 
Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type.  These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.
 
Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type.  These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.
 +
 +
[[Category:Accounting]]

Latest revision as of 14:34, 24 January 2013

Current Status

The APEL accounting repository receives records from sites running the APEL client and also from sites running SSM, in this case the sites send either Job Records or Summary records to the new APEL server and these records are integrated with those in the existing APEL accounting repository from where they are retrieved by the EGI Accounting Portal.

New APEL Overview

  • Sites which do not run the APEL client will send summary records, via SSM, which will be inserted in the Summaries table in the central APEL database.
  • The APEL clients will send Job records, via SSM, which will be inserted in the JobRecords table in the central APEL database.
  • Summary records will be created from the data in the JobRecords table. These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table).
  • The records from external clients in the Summaries table will be inserted (replaced) into the SuperSummaries table.
  • The SpecInt data will also be extracted and stored in a SpecInt table.
  • The relevant Summary and SpecInt data will then be sent to the Accounting Portal using another instance of the SSM.

New APEL Server and Accounting Portal

APEL Server Design

The new APEL server is written in Python, the database is MySQL. It has the following components:

  • SSM
  • Record loader
  • Database
  • Summarizer
  • Record publisher
  • Authentication
  • Monitoring

Test server

Other notes

SSM

For all documentation on the SSM, see APEL/SSM

Here is the document about APEL and messaging: File:APEL-Messaging-v2.2.pdf

Here is the discussion about the APEL/MessagingProtocol.

Record loader

This has the task of taking the received messages and inserting the records in the database. At present there are three message types defined:

  • Job Record messages
  • Summary Record messages
  • Sync Record messages

The header in the message defines the type of records the message will contain and the version of the format. This will enable us to identify newer versions if they are defined.

This simple model allows the definition of alternative message types for different records which the record loader can then be configured to handle, for example, there will be a SpecIntHistory Record message type which is a requirement of the Accounting portal.

APEL Message Format

Database

Summarizer

This processes the Job Records and Summary Records to create the SuperSummaries (User and CPU summaries) and SpecIntHistory tables required by the portal.

Record publisher

This takes the information from a database and produces messages to be sent to the portal from the server using an instance of the SSM. (The Record publisher will also be used by the new version of the APEL client to create messages from the APEL client database to be sent to the APEL server.)

Authentication

This verifies that the senders of the messages via the SSM are known to APEL.

Monitoring

Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type. These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.