Difference between revisions of "APEL/MessageFormat"
Line 135: | Line 135: | ||
== Summary Job Records == | == Summary Job Records == | ||
A new version of the summary message format for use by external sites is currently in development. | |||
=== Description === | === Description === | ||
Line 140: | Line 142: | ||
* See [[APEL/MessageFormatV01]] for version 0.1 of the message formats. | * See [[APEL/MessageFormatV01]] for version 0.1 of the message formats. | ||
Revision as of 15:50, 5 June 2013
APEL Message Format
Please note: we have changed the formats of the job record message and the summary record message, to bring them in line with the EMI Compute Accounting Record. You can find a definition of the EMI CAR here: https://twiki.cern.ch/twiki/bin/view/EMI/ComputeAccountingRecord. The document includes the definition of the Aggregated Usage Record (AUR) which is equivalent to our summary records.
This describes a new message format for getting data between the APEL clients and the server.
It also describes the mapping between its fields and those of the EMI Compute Accounting Record (CAR), described here: https://twiki.cern.ch/twiki/bin/view/EMI/ComputeAccounting.
Terminology:
- A message is one file which is sent and received by the SSM. Usually a message will contain a number of records (eg 1000)
- A record corresponds to one row in the database. It contains a number of key-value pairs as specified by the tables below
- The header in each message tells the server which type of records are in that message. You need one header per message, so one header per file.
- Epoch time, also known as a Unix timestamp, is an integer number of seconds since 1st January 1970. For example, the epoch time now is 1311675474. The command
date +%s
will give you the current epoch time on linux.
Optional fields:
- If you do not have a value for a field in a record, use "null" or "none" (not case-sensitive) as the value for this attribute OR leave the field out of the record completely.
- This applies to all the optional fields in JobRecords and Summary records.
Job Records
A message is one file. It can contain multiple records. Different records must be separated by the end of record marker (%%).
Description
- See APEL/MessageFormatV01 for version 0.1 of the message formats.
Header APEL-individual-job-message: v0.3
The header only appears once at the top of each message (that is once at the top of each file). It defines the type of record and the schema version.
The table shows the equivalent field in the CAR, under the container element urf:UsageRecord
. If not specified, it refers to the text value of urf:Key
, where the element is a direct child of urf:UsageRecord
.
Key | Value | Description | Mandatory | CAR equivalent (if different) |
---|---|---|---|---|
Site | String | GOCDB sitename | Yes | urf:Site[@type="gocdb"] |
Infrastructure | String | grid OR local | ||
SubmitHostType | String | CE-ID OR LRMS | ||
SubmitHost | String | The CE-ID (see example) OR LRMS hostname | Yes | |
LocalJobId | String | Batch System Job ID | Yes | urf:JobIdentity/urf:LocalJobId |
LocalUserId | String | Local username | urf:UserIdentity/urf:LocalUserId | |
GlobalUserName | String | User's X509 DN | urf:UserIdentity/urf:GlobalUserName | |
FQAN | String | User's VOMS attributes | urf:UserIdentity/urf:GroupAttribute[@type="FQAN"] | |
WallDuration | int | Wallclock time for the job (seconds) | Yes | CAR has ISO 8601 time duration |
CpuDuration | int | CPU time for the job (seconds) | Yes | CAR has ISO 8601 time duration |
Processors | int | Number of processors | urf:Processors[@mteric="max"] | |
NodeCount | int | Number of nodes | ||
StartTime | int | Start time of the job (epoch time) | Yes | CAR has ISO 8601 datetime |
EndTime | int | Stop time of the job (epoch time) | Yes | CAR has ISO 8601 datetime |
MemoryReal | int | Memory consumed by job (kbytes) | urf:Memory[@metric="max" and @type="Physical" and @storageUnit="KB"] | |
MemoryVirtual | int | Virtual memory consumed by job (kbytes) | urf:Memory[@metric="max" and @type="Shared" and @storageUnit="KB"] | |
ServiceLevelType | String | Si2k OR HEPSPEC | Yes | urf:ServiceLevel[@type] |
ServiceLevel | double | Value of either HepSpec06 or SpecInt2000 | Yes | urf:ServiceLevel |
End of record: %%
Changes since version 0.2
- Infrastructure field (optional)
- SubmitHostType field (optional)
Changes from version 0.1 to version 0.2
- LocalJobID has changed to LocalJobId
- LocalUserID has changed to LocalUserId
- UserFQAN has changed to FQAN
- ScalingFactorUnit has changed to ServiceLevelType
- The possible values of ScalingFactorType have changed from ["SpecInt2000", "HepSpec06", "custom"] to ["Si2k"], ["HEPSPEC"]
- ScalingFactor has changed to ServiceLevel
Notes
If GlobalUserName or UserFQAN is not published, the value for these fields on the server will be set to 'None'.
Jobs are assumed to be grid jobs. To specify local jobs:
- Infrastructure: local
- SubmitHostType: LRMS
- SubmitHost: <LRMS-hostname>
The Group value specified for local jobs must be different to equivalent grid jobs, or you will not be able to differentiate them in the accounting portal. Suggestion:
Group: atlas
- grid jobGroup: local-atlas
- local job
This advice may change as we get more sites publishing local jobs.
Example Message
APEL-individual-job-message: v0.2 Site: RAL-LCG2 SubmitHost: ce01.ncg.ingrid.pt:2119/jobmanager-lcgsge-atlasgrid LocalJobId: 31564872 LocalUserId: atlasprd019 GlobalUserName: /C=whatever/D=someDN FQAN: /voname/Role=NULL/Capability=NULL WallDuration: 234256 CpuDuration: 2345 Processors: 2 NodeCount: 2 StartTime: 1234567890 EndTime: 1234567899 MemoryReal: 1000 MemoryVirtual: 2000 ServiceLevelType: Si2k ServiceLevel: 1000 %% ...another job record... %% ... %%
Summary Job Records
A new version of the summary message format for use by external sites is currently in development.
Description
- See APEL/MessageFormatV01 for version 0.1 of the message formats.
- See APEL/MessageFormatV02 for version 0.2 of the message formats.
Summary Sync Records
The summary Sync records are used for the creation of the apel-sync Nagios test. It is a mechanism for the central APEL server to know the number of records that each site is storing locally. It is in general only used by sites which publish via the standard APEL client.
Description
Header: APEL-sync-message: v0.1
The header only appears once at the top of each message. It defines the type of record and the schema version.
Key | Value | Description | Mandatory |
---|---|---|---|
Site | String | GOCDB sitename | Yes |
SubmitHost | String | CE ID | Yes |
NumberOfJobs | int | Total number of jobs for that month | Yes |
Month | int | Month | Yes |
Year | int | Year | Yes |
End of record: %%
Notes:
Each record indicates the number of jobs run on the site per month. This data is used to create the Nagios apel-sync test.
Example Message
APEL-sync-message: v0.1 Site: RAL-LCG2 SubmitHost: raltest.rl.ac.uk:8443/cream-pbs-demo NumberOfJobs: 3479 Month: 1 Year: 2010 %% ...another sync record... %% ... %%