Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "APEL/Server"

From EGIWiki
Jump to navigation Jump to search
 
(36 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''This page is under construction and is not yet complete.'''
== Current Status ==
The APEL accounting repository receives records from sites running the APEL client and also from sites running [[APEL/SSM|SSM]], in this case the sites send either Job Records or Summary records to the new APEL server and these records are integrated with those in the existing APEL accounting repository from where they are retrieved by the EGI Accounting Portal.
== New APEL Overview ==
* Sites which do not run the APEL client will send summary records, via [[APEL/SSM|SSM]], which will be inserted in the Summaries table in the central APEL database. 
* The APEL clients will send Job records, via SSM, which will be inserted in the JobRecords table in the central APEL database. 
* Summary records will be created from the data in the JobRecords table.  These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table). 
* The records from external clients in the Summaries table will be inserted (replaced) into the SuperSummaries table. 
* The SpecInt data will also be extracted and stored in a SpecInt table.
* The relevant Summary and SpecInt data will then be sent to the Accounting Portal using another instance of the SSM.
 
== New APEL Server and Accounting Portal ==
* [[APEL Server and Accounting Portal]]


== APEL Server Design ==
== APEL Server Design ==
The new APEL server is written in Python.  It has the following components:
The new APEL server is written in Python, the database is MySQL.  It has the following components:
* SSM
* [[APEL/SSM|SSM]]
* Record loader
* Record loader
* Database
* Summarizer
* Summarizer
* Record publisher
* Record publisher
Line 10: Line 22:
* Monitoring
* Monitoring


== APEL Message Format ==
==== Test server ====
* [[APEL/MessageFormat|APEL Message Format]]
* [[APEL/Test_APEL_Server]]
 
==== Other notes ====
* [[APEL/TimesAndDates]]


== SSM ==
== SSM ==
The SSM is the messaging system used by APEL to transmit messages.  It is written in Python and uses the STOMP protocol.


* [[APEL/SSMOverview|SSM Overview]]
For all documentation on the SSM, see [[APEL/SSM]]
* [[APEL/SSMInstallation|SSM installation instructions]]
* [[APEL/SSMProblems|Common SSM problems]]


==== External SSM Testing ====
Here is the document about APEL and messaging: [[File:APEL-Messaging-v2.2.pdf]]
[[APEL/APELSSMExternalTesting|APEL SSM Testing]]: If you are interested in testing the new STOMP interface or want to develop your own client.
 
Here is the discussion about the [[APEL/MessagingProtocol]].


== Record loader ==
== Record loader ==
Line 31: Line 44:
The header in the message defines the type of records the message will contain and the version of the format.  This will enable us to identify newer versions if they are defined.
The header in the message defines the type of records the message will contain and the version of the format.  This will enable us to identify newer versions if they are defined.


This simple model allows the definition of alternative message types for different records which the record loader can then be configured to handle, for example, there will be a SpecIntHistory Record message type which is a requirement of the APEL portal.
This simple model allows the definition of alternative message types for different records which the record loader can then be configured to handle, for example, there will be a SpecIntHistory Record message type which is a requirement of the Accounting portal.
 
==== APEL Message Format ====
* [[APEL/MessageFormat|APEL Message Format]]
 
== Database ==
* [[APEL/DatabaseTables]]


== Summarizer ==
== Summarizer ==
This process the Job Records and Summary Records to create the CPU, User Summaries and SpecIntHistory tables required by the portal.  
This processes the Job Records and Summary Records to create the SuperSummaries (User and CPU summaries) and SpecIntHistory tables required by the portal.


== Record publisher ==
== Record publisher ==
Line 43: Line 62:


* [[APEL/UsingAuth]]
* [[APEL/UsingAuth]]
== Monitoring ==
Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type.  These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.
[[Category:Accounting]]

Latest revision as of 13:34, 24 January 2013

Current Status

The APEL accounting repository receives records from sites running the APEL client and also from sites running SSM, in this case the sites send either Job Records or Summary records to the new APEL server and these records are integrated with those in the existing APEL accounting repository from where they are retrieved by the EGI Accounting Portal.

New APEL Overview

  • Sites which do not run the APEL client will send summary records, via SSM, which will be inserted in the Summaries table in the central APEL database.
  • The APEL clients will send Job records, via SSM, which will be inserted in the JobRecords table in the central APEL database.
  • Summary records will be created from the data in the JobRecords table. These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table).
  • The records from external clients in the Summaries table will be inserted (replaced) into the SuperSummaries table.
  • The SpecInt data will also be extracted and stored in a SpecInt table.
  • The relevant Summary and SpecInt data will then be sent to the Accounting Portal using another instance of the SSM.

New APEL Server and Accounting Portal

APEL Server Design

The new APEL server is written in Python, the database is MySQL. It has the following components:

  • SSM
  • Record loader
  • Database
  • Summarizer
  • Record publisher
  • Authentication
  • Monitoring

Test server

Other notes

SSM

For all documentation on the SSM, see APEL/SSM

Here is the document about APEL and messaging: File:APEL-Messaging-v2.2.pdf

Here is the discussion about the APEL/MessagingProtocol.

Record loader

This has the task of taking the received messages and inserting the records in the database. At present there are three message types defined:

  • Job Record messages
  • Summary Record messages
  • Sync Record messages

The header in the message defines the type of records the message will contain and the version of the format. This will enable us to identify newer versions if they are defined.

This simple model allows the definition of alternative message types for different records which the record loader can then be configured to handle, for example, there will be a SpecIntHistory Record message type which is a requirement of the Accounting portal.

APEL Message Format

Database

Summarizer

This processes the Job Records and Summary Records to create the SuperSummaries (User and CPU summaries) and SpecIntHistory tables required by the portal.

Record publisher

This takes the information from a database and produces messages to be sent to the portal from the server using an instance of the SSM. (The Record publisher will also be used by the new version of the APEL client to create messages from the APEL client database to be sent to the APEL server.)

Authentication

This verifies that the senders of the messages via the SSM are known to APEL.

Monitoring

Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type. These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.