Difference between revisions of "APEL/Server"

From EGIWiki
Jump to: navigation, search
Line 42: Line 42:
== Database ==
== Database ==
* [[APEL Database Tables]]
{| cellspacing="1" cellpadding="1" border="1" width="800"
{| cellspacing="1" cellpadding="1" border="1" width="800"

Revision as of 13:40, 18 January 2012

This page is under construction and is not yet complete.

New APEL Overview

  • Sites which do not run the APEL client will send summary records via SSM which will be inserted in the Summaries table in the central APEL database.
  • The APEL clients will send Job records via SSM which will be inserted in the JobRecords table in the central APEL database.
  • Summary records will be created from the data in the JobRecords table. These JobRecords Summaries will be loaded into a SuperSummaries table (the SuperSummaries table will have the same schema as the Summaries table).
  • The records from external clients in the Summaries table will be inserted (replaced) into the SuperSummaries table.
  • The SpecInt data will also be extracted and stored in a SpecInt table.
  • The relevant Summary and SpecInt data will then be sent to the Accounting Portal using another instance of the SSM.

APEL Server Design

The new APEL server is written in Python, the database is MySQL. It has the following components:

  • SSM
  • Record loader
  • Database
  • Summarizer
  • Record publisher
  • Authentication
  • Monitoring

APEL Message Format


The SSM is the messaging system used by APEL to transmit messages. It is written in Python and uses the STOMP protocol.

External SSM Testing

APEL SSM Testing: If you are interested in testing the new STOMP interface or want to develop your own client.

Record loader

This has the task of taking the received messages and inserting the records in the database. At present there are three message types defined:

  • Job Record messages
  • Summary Record messages
  • Sync Record messages

The header in the message defines the type of records the message will contain and the version of the format. This will enable us to identify newer versions if they are defined.

This simple model allows the definition of alternative message types for different records which the record loader can then be configured to handle, for example, there will be a SpecIntHistory Record message type which is a requirement of the Accounting portal.


ColumnName Type Null Key Default Normalised Indexed
UpdateTime TimeStamp Auto
Site varchar(255) No Yes Yes
SubmitHost varchar(255) No MUL Yes
LocalJobID varchar(255) No MUL
LocalUserID varchar(255)
GlobalUserName varchar(255) No 'None' Yes Yes
Role varchar(255) No 'None' Yes Yes
GridGroup varchar(255) No 'None' Yes Yes
VO varchar(255) No 'None' Yes Yes
WallDuration int
CpuDuration int
Processors int
NodeCount int
StartTime datetime No
EndTime datetime MUL
MemoryReal int
MemoryVirtual int
ScalingFactorType varchar(50) No
ScalingFactor double No


This processes the Job Records and Summary Records to create the CPU, User SuperSummaries and SpecIntHistory tables required by the portal.

Record publisher

This takes the information from a database and produces messages to be sent to the portal from the server using an instance of the SSM. (The Record publisher will also be used by the new version of the APEL client to create messages from the APEL client database to be sent to the APEL server.)


This verifies that the senders of the messages via the SSM are known to APEL.


Two APEL accounting tests will continue to be provided, APEL-Pub and APEL-Sync. These tests are required to publish details of every production/certified site registered in GOCDB with at least one CE defined as an "APEL" service type. These results are sent as Nagios metrics via a network of ActiveMQ brokers to be consumed by the ROC (Regional Operations Centre) Nagios instances.