Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "APEL/MessageFormat"

From EGIWiki
Jump to navigation Jump to search
Line 13: Line 13:
The header only appears once at the top of each message. It defines the type of record and the schema version.  
The header only appears once at the top of each message. It defines the type of record and the schema version.  


{| cellspacing="1" cellpadding="1" border="1" align="left" width="200"
{| cellspacing="1" cellpadding="1" border="1" align="left" width="600"
|-
|-
! scope="col" | Key
! scope="col" | Key  
! scope="col" | Value
! scope="col" | Value  
! scope="col" | Description
! scope="col" | Description  
! scope="col" | Mandatory
! scope="col" | Mandatory
|-
|-
| Site
| Site  
| String
| String  
| GOCDB sitename
| GOCDB sitename  
| Yes
| Yes
|-
|-
| SubmitHost
| SubmitHost  
| String
| String  
| Head node where the job was submitted
| Head node where the job was submitted  
| Yes
| Yes
|-
|-
| LocalJobID
| LocalJobID  
| String
| String  
| Batch System Job ID
| Batch System Job ID  
| Yes
| Yes
|-
|-
| LocalUserID
| LocalUserID  
| String
| String  
| Local username
| Local username  
|  
|  
|-
|-
| GlobalUserName
| GlobalUserName  
| String
| String  
| User's X509 DN
| User's X509 DN  
|  
|  
|-
|-
| UserFQAN
| UserFQAN  
| String
| String  
| User's VOMS attributes
| User's VOMS attributes  
|  
|  
|-
|-
| WallDuration
| WallDuration  
| int
| int  
| Wallclock time for the job (seconds)
| Wallclock time for the job (seconds)  
| Yes
| Yes
|-
|-
| CpuDuration
| CpuDuration  
| int
| int  
| CPU time for the job (seconds)
| CPU time for the job (seconds)  
| Yes
| Yes
|-
|-
| Processors
| Processors  
| int
| int  
| Number of processors
| Number of processors  
|  
|  
 
<br>


|-
|-
| NodeCount
| NodeCount  
| int
| int  
| Number of nodes
| Number of nodes  
|  
|  
|-
|-
| StartTime
| StartTime  
| int
| int  
| Start time of the job (epoch)
| Start time of the job (epoch)  
| Yes
| Yes
|-
|-
| EndTime
| EndTime  
| int
| int  
| Stop time of the job (epoch)
| Stop time of the job (epoch)  
|  
|  
|-
|-
| MemoryReal
| MemoryReal  
| int
| int  
| Memory consumed by job (kbytes)
| Memory consumed by job (kbytes)  
|  
|  
|-
|-
| MemoryVirtual
| MemoryVirtual  
| int
| int  
| Virtual memory consumed by job&nbsp;(kbytes)
| Virtual memory consumed by job&nbsp;(kbytes)  
|  
|  
|-
|-
| ScalingFactorUnit
| ScalingFactorUnit  
| String
| String  
| HepSpec &#124;&nbsp;SpecInt &#124; custom
| HepSpec &#124;&nbsp;SpecInt &#124; custom  
| Yes
| Yes
|-
|-
| ScalingFactor
| ScalingFactor  
| double
| double  
| Value of either HepSpec, SpecInt or custom
| Value of either HepSpec, SpecInt or custom  
| Yes
| Yes
|}
|}


'''End of record:''' &nbsp;%%  
'''End of record:''' &nbsp;%%  
Line 110: Line 113:
  ScalingFactor = 1
  ScalingFactor = 1


If&nbsp;!GlobalUserName or&nbsp;!UserFQAN is not published, the value for these fields on the server will be set to 'None'.  
If&nbsp;!GlobalUserName or&nbsp;!UserFQAN is not published, the value for these fields on the server will be set to 'None'.


=== Example Message  ===
=== Example Message  ===

Revision as of 13:05, 3 May 2011

APEL Message Format

This describes a new message format for getting data between the APEL clients and the server.

Job Records

A message can contain multiple records. Different records must be separated by the end of record marker (%%).

Description

Header APEL-individual-job-message: v0.1

The header only appears once at the top of each message. It defines the type of record and the schema version.

Key Value Description Mandatory
Site String GOCDB sitename Yes
SubmitHost String Head node where the job was submitted Yes
LocalJobID String Batch System Job ID Yes
LocalUserID String Local username
GlobalUserName String User's X509 DN
UserFQAN String User's VOMS attributes
WallDuration int Wallclock time for the job (seconds) Yes
CpuDuration int CPU time for the job (seconds) Yes
Processors int Number of processors


NodeCount int Number of nodes
StartTime int Start time of the job (epoch) Yes
EndTime int Stop time of the job (epoch)
MemoryReal int Memory consumed by job (kbytes)
MemoryVirtual int Virtual memory consumed by job (kbytes)
ScalingFactorUnit String HepSpec | SpecInt | custom Yes
ScalingFactor double Value of either HepSpec, SpecInt or custom Yes



End of record:  %%

Notes: If !ScalingFactorUnit/Value is not available it should be set to:

ScalingFactorUnit = 'custom'
ScalingFactor = 1

If !GlobalUserName or !UserFQAN is not published, the value for these fields on the server will be set to 'None'.

Example Message

{{{ APEL-individual-job-message: v0.1 Site: RAL-LCG2 SubmitHost: ce01.ncg.ingrid.pt:2119/jobmanager-lcgsge-atlasgrid LocalJobID: 31564872 LocalUserID: atlasprd019 GlobalUserName: /C=whatever/D=someDN UserFQAN: /voname/Role=NULL/Capability=NULL WallDuration: 234256 CpuDuration: 2345 Processors: 2 NodeCount: 2 StartTime: 1234567890 EndTime: 1234567899 MemoryReal: 1000 MemoryVirtual: 2000 ScalingFactorUnit: SpecInt2000 ScalingFactor: 1000 %% ...another job record... %% ... %% }}}

Summary Job Records

Description

Header: APEL-summary-job-message: v0.1

The header only appears once at the top of each message. It defines the type of record and the schema version. || Key || Value || Description || Mandatory || || Site || String || GOCDB/official name of the site where the job run || Yes || || Month || int || Month of summary || Yes || || Year || int || Year of summary || Yes || || !GlobalUserName || String || User's X.509 DN || || || VO || String || User's VO || || || Group || String || User's VOMS Group || || || Role || String ||User's VOMS Role || || || !WallDuration || int || clock time for the job || Yes || || !CpuDuration || int || CPU time for the job || || || !NormalisedCpuDuration || int || Normalised CPU time for the job || Yes || || !NormalisedWallDuration || int || Normalised Wall clock time || Yes || || !NumberOfJobs || int || Total number of jobs || Yes ||


End of record: %%

Notes:BR

If !GlobalUserName, VO, Role or Group are not published, the value for these fields on the server will be set to 'None'.BR

A single job record must only be included in one summary record to avoid duplication of data.BR

Example Message

{{{ APEL-summary-job-message: v0.1 Site: RAL-LCG2 Month: 3 Year: 2010 GlobalUserName: /C=whatever/D=someDN VO: atlas Group: /atlas Role: Role=production WallDuration: 234256 CpuDuration: 2345 NormalisedCpuDuration: 2500 NormalisedWallDuration: 244435 NumberOfJobs: 100 %% ...another summary job record... %% ... %% }}}

Summary Sync Records

The summary Sync records are used for the creation of the apel-sync Nagios test. It is a mechanism for the central APEL server to know the number of records that each site is storing locally.

Description

Header: APEL-sync-message: v0.1

The header only appears once at the top of each message. It defines the type of record and the schema version. || Key || Value || Description || Mandatory || || Site || String || GOCDB/official name of the site where the job run || Yes || || NJobs || int || Total number of jobs for that month || Yes || || NDays || int || Number of days between earliest and latest job in month || Yes || ||Month || int || Month || Yes || ||Year || int ||Year || Yes ||


End of record: %%

Notes:BR

Each record indicates the number of jobs run on the site per month. This data is used to create the Nagios apel-sync test.

Example Message

{{{ APEL-sync-message: v0.1 Site: RAL-LCG2 NJobs: 3479 NDays: 29 Month: 1 Year: 2010 %% ...another sync record... %% ... %% }}}