Difference between revisions of "Accounting Portal API View /egi/htc/operations centres/"
(29 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
This view offers an overview of the EGI HTC infrastructure viewed by Operations Centres (NGIs). | |||
= Metric = | = Metric = | ||
The Row and Column | The Row and Column variables define the dimensions in which the data are grouped, for each view usually the data are grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. | ||
These are the valid values for both of these: | These are the valid values for both of these: | ||
Metric | {|class="wikitable" | ||
Name | !Metric | ||
Description | !Name | ||
!Description | |||
Number of jobs | |- | ||
Number of discrete jobs launched, independent on completion status. | |'''number_of_jobs''' | ||
|Number of jobs | |||
Normalized Sum CPU | |Number of discrete jobs launched, independent on completion status. | ||
Hours of CPU time consumed in calculations by jobs, normalised using the HEPSPEC06 benchmark. | |- | ||
|'''normalized_sum_cpu''' | |||
Sum CPU | |Normalized Sum CPU | ||
Hours of not normalised CPU time consumed by jobs. | |Hours of CPU time consumed in calculations by jobs, normalised using the HEPSPEC06 benchmark. | ||
|- | |||
Normalized Sum Elapsed | |'''sum_cpu''' | ||
Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the HEPSPEC06 benchmark. | |Sum CPU | ||
|Hours of not normalised CPU time consumed by jobs. | |||
Normalized Sum Elapsed * Number of Processors | |- | ||
Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the multiplied by the number of logical cores used by each job. | |'''normalized_sum_elapsed''' | ||
|Normalized Sum Elapsed | |||
Sum Elapsed * Number of Processors | |Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the HEPSPEC06 benchmark. | ||
Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by the job. | |- | ||
|'''normalized_sum_elapsed_times_number_of_processors''' | |||
Unnormalized Wall CPU Hours | |Normalized Sum Elapsed * Number of Processors | ||
Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks. | |Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the multiplied by the number of logical cores used by each job. | ||
|- | |||
CPU Efficiency | |'''sum_elapsed_times_number_of_processors''' | ||
Normalised Sum CPU / (Normalised Sum Elapsed * Number of Processors). | |Sum Elapsed * Number of Processors | ||
|Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by the job. | |||
|- | |||
|'''sum_elapsed''' | |||
|Unnormalized Wall CPU Hours | |||
|Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks. | |||
|- | |||
|'''cpu_efficiency''' | |||
|CPU Efficiency | |||
|Normalised Sum CPU / (Normalised Sum Elapsed * Number of Processors). Measures the percentage of running time of the process used to do actual computations. | |||
|} | |||
<!-- | |||
Metric values that are only useable in cloud views: | Metric values that are only useable in cloud views: | ||
Metric | {|class="wikitable" | ||
Name | !Metric | ||
Description | !Name | ||
!Description | |||
Number of Virtual Machines | |- | ||
Number of VM instances running in the selected time period. | |'''number_of_virtual_machines''' | ||
|Number of Virtual Machines | |||
Sum Wall hours (Elapsed time) | |Number of VM instances running in the selected time period. | ||
Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by each job.. | |- | ||
|'''sum_elapsed''' | |||
Sum Elapsed * Number of Processors | |Sum Wall hours (Elapsed time) | ||
Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by each job.. | |Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by each job.. | ||
|- | |||
Inbound Network Traffic | |'''sum_elapsed_times_number_of_processors''' | ||
Inbound network traffic in bytes towards running VM instances. | |Sum Elapsed * Number of Processors | ||
|Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by each job.. | |||
Outbound Network Traffic | |- | ||
Outcoming network traffic in bytes from running VM instances. | |'''inbound_network_traffic''' | ||
|Inbound Network Traffic | |||
Memory Used | |Inbound network traffic in bytes towards running VM instances. | ||
Amount of memory in bytes allocated by running VM instances. | |- | ||
|'''outbound_network_traffic''' | |||
Disk Used | |Outbound Network Traffic | ||
Amount of storage in bytes allocated by running VMs instances. | |Outcoming network traffic in bytes from running VM instances. | ||
|- | |||
|'''memory_used''' | |||
|Memory Used | |||
|Amount of memory in bytes allocated by running VM instances. | |||
|- | |||
|'''disk_used''' | |||
|Disk Used | |||
|Amount of storage in bytes allocated by running VMs instances. | |||
|} | |||
Metric Values that are only useable in restricted views: | Metric Values that are only useable in restricted views: | ||
{|class="wikitable" | |||
!Metric | |||
!Name | |||
!Description | |||
|- | |||
|'''mixed''' | |||
|Combined Metrics | |||
|Number of discrete jobs launched, independent on completion status. | |||
|} | |||
Metric Values that are only useable in WLCG views: | Metric Values that are only useable in WLCG views: | ||
Metric | {|class="wikitable" | ||
Name | !Metric | ||
Description | !Name | ||
!Description | |||
Number of jobs | |- | ||
Number of discrete jobs launched, independent on completion status. | |'''number_of_jobs''' | ||
|Number of jobs | |||
Sum CPU Time Hours | |Number of discrete jobs launched, independent on completion status. | ||
Hours of CPU Time as reported to APEL by the batch system. | |- | ||
|'''sum_cpu''' | |||
Sum CPU Time Days | |Sum CPU Time Hours | ||
Days of CPU Time as reported to APEL by the batch system. | |Hours of CPU Time as reported to APEL by the batch system. | ||
|- | |||
Sum CPU Work HS06 Hours | |'''sum_cpu_days''' | ||
(Same as "Normalized Sum CPU"). Sum of CPU Time Hours as reported to APEL by the batch system and multiplied by benchmarked HEPSPEC06 power for a given resource. | |Sum CPU Time Days | ||
|Days of CPU Time as reported to APEL by the batch system. | |||
Sum CPU Work HS06 Days | |- | ||
(Same as "Normalized Sum CPU"). Sum of CPU Time Days as reported to APEL by the batch system and multiplied by benchmarked HEPSPEC06. | |'''sum_cpu_work''' | ||
|Sum CPU Work HS06 Hours | |||
Sum Wallclock Time Hours | |(Same as "Normalized Sum CPU"). Sum of CPU Time Hours as reported to APEL by the batch system and multiplied | ||
(Same as "Sum elapsed * Number of processors"). Wallclock time as reported by the batch system. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled wallclock multiplied by number of processors. | by benchmarked HEPSPEC06 power for a given resource. | ||
|- | |||
Sum Wallclock Time Days | |'''sum_cpu_work_days''' | ||
(Same as "Sum elapsed * Number of processors"). Wallclock time as reported by the batch system. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled wallclock multiplied by number of processors. | |Sum CPU Work HS06 Days | ||
|(Same as "Normalized Sum CPU"). Sum of CPU Time Days as reported to APEL by the batch system and multiplied by benchmarked HEPSPEC06. | |||
Sum Wallclock Work HS06 Hours | |- | ||
(Same as "Normalised Sum elapsed * Number of processors"). Wallclock time as reported by the batch system and multiplied by benchmarked HEPSPEC06 power of a given CPU resource and by number of processors. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled and Normalised Wallclock multiplied by Number of Processors | |'''sum_wallclock_time''' | ||
|Sum Wallclock Time Hours | |||
Sum Wallclock Work HS06 Days | |(Same as "Sum elapsed * Number of processors"). Wallclock time as reported by the batch system. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled wallclock multiplied by number of processors. | ||
(Same as "Normalised Sum elapsed * Number of processors"). Wallclock time as reported by the batch system and multiplied by benchmarked HEPSPEC06 power of a given CPU resource and by number of processors. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled and Normalised Wallclock multiplied by Number of Processors | |- | ||
|'''sum_wallclock_time_days''' | |||
CPU Efficiency | |Sum Wallclock Time Days | ||
100 * Sum CPU Time / Sum (Wallclock Time * Number of Processors). | |(Same as "Sum elapsed * Number of processors"). Wallclock time as reported by the batch system. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled wallclock multiplied by number of processors. | ||
|- | |||
|'''sum_wallclock_work''' | |||
|Sum Wallclock Work HS06 Hours | |||
|(Same as "Normalised Sum elapsed * Number of processors"). Wallclock time as reported by the batch system and multiplied by benchmarked HEPSPEC06 power of a given CPU resource and by number of processors. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled and Normalised Wallclock multiplied by Number of Processors | |||
|- | |||
|'''sum_wallclock_work_days''' | |||
|Sum Wallclock Work HS06 Days | |||
|(Same as "Normalised Sum elapsed * Number of processors"). Wallclock time as reported by the batch system and multiplied by benchmarked HEPSPEC06 power of a given CPU resource and by number of processors. For some sites represent raw wallclock time defined as end time stamp of a particular job minus start time stamp and multiplied by number of processors. There are other sites which do not report raw wallclock time, but rather report raw wallclock time scaled by a batch system to a reference node. For such sites this metric currently shows scaled and Normalised Wallclock multiplied by Number of Processors | |||
|- | |||
|'''cpu_efficiency''' | |||
|CPU Efficiency | |||
|100 * Sum CPU Time / Sum (Wallclock Time * Number of Processors). | |||
|} | |||
--> | |||
==Metric Unit Suffix== | |||
For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is | For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is | ||
<metric>-<unit_suffix> | |||
<pre><metric>-<unit_suffix></pre> | |||
For example: | For example: | ||
normcpu-month | |||
<pre>normcpu-month</pre> | |||
If no suffix is indicated, hours are selected by default: | If no suffix is indicated, hours are selected by default: | ||
Suffix | {|class="wikitable" | ||
Name | !Suffix | ||
Description | !Name | ||
hour | !Description | ||
Hours | |- | ||
Select the default unit of hours, equivalent to not indicating any suffix. | |'''hour''' | ||
day | |Hours | ||
Days | |Select the default unit of hours, equivalent to not indicating any suffix. | ||
Select a time of unit based on days (24 hours) | |- | ||
month | |'''day''' | ||
Months | |Days | ||
Select a time of unit based on months (720 hours) | |Select a time of unit based on days (24 hours) | ||
quarter | |- | ||
Quarters | |'''month''' | ||
Select a time of unit based on quarters (2160 hours) | |Months | ||
halfyear | |Select a time of unit based on months (720 hours) | ||
Half Years | |- | ||
Select a time of unit based on half years (4320 hours) | |'''quarter''' | ||
year | |Quarters | ||
Years | |Select a time of unit based on quarters (2160 hours) | ||
Select a time of unit based on years (8640 hours) | |- | ||
|'''halfyear''' | |||
|Half Years | |||
|Select a time of unit based on half years (4320 hours) | |||
|- | |||
|'''year''' | |||
|Years | |||
|Select a time of unit based on years (8640 hours) | |||
|} | |||
= Row and Column Variables = | = Row and Column Variables = | ||
Line 189: | Line 238: | ||
|} | |} | ||
<!-- | |||
Variable values that are only useable in cloud views: | |||
{|class="wikitable" | |||
Variable | !Variable | ||
Name | !Name | ||
Description | !Description | ||
STATUS | |- | ||
Status | |'''STATUS''' | ||
Grouping by VM Status | |Status | ||
vm_id | |Grouping by VM Status | ||
VM Id | |- | ||
Grouping by VM ID | |'''vm_id''' | ||
|VM Id | |||
|Grouping by VM ID | |||
|} | |||
Variable Values that are only useable in restricted views: | Variable Values that are only useable in restricted views: | ||
Variable | {|class="wikitable" | ||
Name | !Variable | ||
Description | !Name | ||
UserDN | !Description | ||
User DN | |- | ||
Grouping by UserDN | |'''UserDN''' | ||
UserRole | |User DN | ||
User Role | |Grouping by UserDN | ||
Grouping by User Role | |- | ||
|'''UserRole''' | |||
|User Role | |||
|Grouping by User Role | |||
|} | |||
Variable Values that are only useable in WLCG Tier views: | Variable Values that are only useable in WLCG Tier views: | ||
Variable | |||
Name | {|class="wikitable" | ||
Description | !Variable | ||
TIER1 | !Name | ||
TIER1 | !Description | ||
Grouping by Tier1 Node | |- | ||
FEDERATION | |'''TIER1''' | ||
Federation | |TIER1 | ||
Grouping by Tier2 Federation | |Grouping by Tier1 Node | ||
|- | |||
|'''FEDERATION''' | |||
|Federation | |||
|Grouping by Tier2 Federation | |||
|} | |||
--> | |||
= Dates = | = Dates = | ||
The URL components for dates are: | The URL components for dates are: | ||
<pre>/StartYear/StartMonth/EndYear/EndMonth/</pre> | |||
The date format is: | The date format is: | ||
<pre>/YYYY/MM/YYYY/MM/</pre> | |||
The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth. | The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth. | ||
Line 260: | Line 324: | ||
In the case of a “custom” selection, the used format is | In the case of a “custom” selection, the used format is | ||
<pre>custom-vo1,vo2,vo3,..,von</pre> | |||
This scales to infrastructure views with 200+ VOs. | This scales to infrastructure views with 200+ VOs. | ||
Line 292: | Line 356: | ||
|Returns data as a text file containing CSV (''Comma Separated Values'') data. | |Returns data as a text file containing CSV (''Comma Separated Values'') data. | ||
|} | |} | ||
= Output Example = | |||
<pre> | |||
https://accounting-pre.egi.cesga.es/egi/htc/operations_centres/number_of_jobs/REGION/DATE/2016/6/2017/6/egi/onlyinfrajobs/JSON/ | |||
</pre> | |||
<source lang="javascript"> | |||
[{"2016-06":219164,"2016-07":164009,"2016-08":246484,"2016-09":171441,"2016-10":110200,"2016-11":140790, | |||
"2016-12":54516,"2017-01":139085,"2017-02":30541,"id":"AfricaArabia","Total":1276230,"Percent":0.3}, {"2016- | |||
06":2075301,"2016-07":2343848,"2016-08":2165980,"2016-09":2455284,"2016-10":2699340,"2016-11":2676404,"2016- | |||
12":2649081,"2017-01":3022985,"2017-02":1717884,"id":"AsiaPacific","Total":21806107,"Percent":5.06}, {"2016- | |||
06":5168077,"2016-07":9503484,"2016-08":8512419,"2016-09":8729766,"2016-10":6092806,"2016-11":9139894,"2016- | |||
12":8713986,"2017-01":9729395,"2017-02":5508540,"id":"CERN","Total":71098367,"Percent":16.5}, {"2016- | |||
06":22966,"2016-07":25770,"2016-08":23796,"2016-09":25658,"2016-10":25232,"2016-11":24717,"2016- | |||
12":25744,"2017-01":25962,"2017-02":16267,"id":"NGI_AEGIS","Total":216112,"Percent":0.05}, {"2016- | |||
06":3249,"2016-07":1947,"2016-08":3218,"2016-09":3843,"2016-10":5204,"2016-11":3138,"2016-12":3245,"2017- | |||
01":3015,"2017-02":1121,"id":"NGI_ARMGRID","Total":27980,"Percent":0.01}, {"2016-06":36516,"2016- | |||
07":31644,"2016-08":31030,"2016-09":11946,"2016-10":26255,"2016-11":19714,"2016-12":14915,"2017- | |||
01":16721,"2017-02":7147,"id":"NGI_BG","Total":195888,"Percent":0.05}, {"2016-06":464959,"2016- | |||
07":496585,"2016-08":511609,"2016-09":457829,"2016-10":523571,"2016-11":590077,"2016-12":569312,"2017- | |||
01":422912,"2017-02":520081,"id":"NGI_CH","Total":4556935,"Percent":1.06}, {"2016-06":109807,"2016- | |||
07":152312,"2016-08":97788,"2016-09":157858,"2016-10":121591,"2016-11":140423,"2016-12":146933,"2017- | |||
01":163161,"2017-02":90258,"id":"NGI_CHINA","Total":1180131,"Percent":0.27}, {"2016-06":419707,"2016- | |||
07":622278,"2016-08":333445,"2016-09":324516,"2016-10":441747,"2016-11":523249,"2016-12":392839,"2017- | |||
01":738375,"2017-02":268361,"id":"NGI_CZ","Total":4064517,"Percent":0.94}, {"2016-06":6249765,"2016- | |||
07":7106409,"2016-08":5524328,"2016-09":5403365,"2016-10":6225585,"2016-11":5203500,"2016-12":4667233,"2017- | |||
01":4685814,"2017-02":3515577,"id":"NGI_DE","Total":48581576,"Percent":11.27}, {"2016-06":39302,"2016- | |||
07":43607,"2016-08":23509,"2016-09":21931,"2016-10":19937,"2016-11":31761,"2016-12":25990,"2017- | |||
01":17318,"2017-02":9092,"id":"NGI_FI","Total":232447,"Percent":0.05}, {"2016-06":5132661,"2016- | |||
07":5431276,"2016-08":4063110,"2016-09":3925907,"2016-10":4857282,"2016-11":5366228,"2016-12":5193655,"2017- | |||
01":4703217,"2017-02":2671748,"id":"NGI_FRANCE","Total":41345084,"Percent":9.59}, {"2016-06":8714,"2016- | |||
07":7582,"2016-08":8244,"2016-09":7563,"2016-10":8527,"2016-11":7806,"2016-12":8164,"2017-01":8470,"2017- | |||
02":7459,"id":"NGI_GE","Total":72529,"Percent":0.02}, {"2016-06":89845,"2016-07":76572,"2016-08":48624,"2016- | |||
09":64850,"2016-10":84817,"2016-11":103774,"2016-12":82437,"2017-01":86706,"2017- | |||
02":36689,"id":"NGI_GRNET","Total":674314,"Percent":0.16}, {"2016-06":10454,"2016-07":9428,"2016- | |||
08":6613,"2016-09":3274,"2016-10":5828,"2016-11":4741,"2016-12":5184,"2017-01":5598,"2017- | |||
02":5665,"id":"NGI_HR","Total":56785,"Percent":0.01}, {"2016-06":108292,"2016-07":121147,"2016- | |||
08":111935,"2016-09":131450,"2016-10":165201,"2016-11":147052,"2016-12":142005,"2017-01":123487,"2017- | |||
02":83204,"id":"NGI_HU","Total":1133773,"Percent":0.26}, {"2016-06":2307912,"2016-07":2948625,"2016- | |||
08":2295666,"2016-09":1798655,"2016-10":1739477,"2016-11":1931112,"2016-12":1735031,"2017-01":2036922,"2017- | |||
02":1046522,"id":"NGI_IBERGRID","Total":17839922,"Percent":4.14}, {"2016-06":519638,"2016-07":766188,"2016- | |||
08":759286,"2016-09":540829,"2016-10":733266,"2016-11":515202,"2016-12":356118,"2017-01":394209,"2017- | |||
02":286127,"id":"NGI_IL","Total":4870863,"Percent":1.13}, {"2016-06":6246608,"2016-07":6644789,"2016- | |||
08":6174721,"2016-09":4945670,"2016-10":5110056,"2016-11":5114330,"2016-12":4172565,"2017-01":4157040,"2017- | |||
02":2551246,"id":"NGI_IT","Total":45117025,"Percent":10.47}, {"2016-06":56,"2016-07":1,"2016-08":0,"2016- | |||
09":0,"2016-10":1,"2016-11":1,"2016-12":0,"2017-01":0,"2017-02":0,"id":"NGI_MD","Total":59,"Percent":0}, | |||
{"2016-06":591910,"2016-07":885542,"2016-08":626650,"2016-09":752257,"2016-10":538775,"2016-11":880280,"2016- | |||
12":986051,"2017-01":841145,"2017-02":342342,"id":"NGI_NDGF","Total":6444952,"Percent":1.5}, {"2016- | |||
06":1701275,"2016-07":2021610,"2016-08":1570774,"2016-09":1716250,"2016-10":1494287,"2016-11":2140621,"2016- | |||
12":1630570,"2017-01":2041008,"2017-02":974983,"id":"NGI_NL","Total":15291378,"Percent":3.55}, {"2016- | |||
06":456197,"2016-07":437478,"2016-08":361102,"2016-09":390919,"2016-10":448349,"2016-11":536580,"2016- | |||
12":385494,"2017-01":406321,"2017-02":208844,"id":"NGI_PL","Total":3631284,"Percent":0.84}, {"2016- | |||
06":1039451,"2016-07":978509,"2016-08":1025222,"2016-09":1000574,"2016-10":1108580,"2016-11":1125055,"2016- | |||
12":973822,"2017-01":1058285,"2017-02":682640,"id":"NGI_RO","Total":8992138,"Percent":2.09}, {"2016- | |||
06":323791,"2016-07":337088,"2016-08":466323,"2016-09":568205,"2016-10":460448,"2016-11":575114,"2016- | |||
12":507834,"2017-01":640344,"2017-02":291847,"id":"NGI_SI","Total":4170994,"Percent":0.97}, {"2016- | |||
06":318750,"2016-07":491367,"2016-08":464943,"2016-09":386949,"2016-10":233969,"2016-11":246260,"2016- | |||
12":215088,"2017-01":375396,"2017-02":225941,"id":"NGI_SK","Total":2958663,"Percent":0.69}, {"2016- | |||
06":191830,"2016-07":76867,"2016-08":95198,"2016-09":49766,"2016-10":76174,"2016-11":80574,"2016- | |||
12":49587,"2017-01":70533,"2017-02":52213,"id":"NGI_TR","Total":742742,"Percent":0.17}, {"2016- | |||
06":107158,"2016-07":127115,"2016-08":114712,"2016-09":126440,"2016-10":96335,"2016-11":104293,"2016- | |||
12":98550,"2017-01":81163,"2017-02":41632,"id":"NGI_UA","Total":897398,"Percent":0.21}, {"2016- | |||
06":8825217,"2016-07":10829331,"2016-08":7249724,"2016-09":6861761,"2016-10":7175525,"2016-11":7742905,"2016- | |||
12":8072590,"2017-01":7729046,"2017-02":4842588,"id":"NGI_UK","Total":69328687,"Percent":16.09}, {"2016- | |||
06":4689263,"2016-07":5032828,"2016-08":4627810,"2016-09":4020272,"2016-10":4835534,"2016-11":3838035,"2016- | |||
12":4118913,"2017-01":4614193,"2017-02":2104736,"id":"ROC_Canada","Total":37881584,"Percent":8.79}, {"2016- | |||
06":290699,"2016-07":578930,"2016-08":267672,"2016-09":237313,"2016-10":212189,"2016-11":276339,"2016- | |||
12":248714,"2017-01":260469,"2017-02":157167,"id":"ROC_LA","Total":2529492,"Percent":0.59}, {"2016- | |||
06":1342566,"2016-07":1471906,"2016-08":1316836,"2016-09":1510919,"2016-10":1597201,"2016-11":1833431,"2016- | |||
12":1708028,"2017-01":1971692,"2017-02":1018750,"id":"Russia","Total":13771329,"Percent":3.2}, {"2016- | |||
06":49111100,"2016-07":59766072,"2016-08":49128771,"2016-09":46803260,"2016-10":47273289,"2016- | |||
11":51063400,"2016-12":47954194,"2017-01":50569987,"2017-02":29317212,"id":"Total","Total":430987285,"Percent":""}, {"2016-06":"11.40%","2016-07":"13.87%","2016- | |||
08":"11.40%","2016-09":"10.86%","2016-10":"10.97%","2016-11":"11.85%","2016-12":"11.13%","2017- | |||
01":"11.73%","2017-02":"6.80%","id":"Percent","Percent":"","Total":""}, | |||
{"id":"xlegend","0":"AfricaArabia","1":"AsiaPacific","2":"CERN","3":"NGI_AEGIS","4":"NGI_ARMGRID","5":"NGI_BG", | |||
"6":"NGI_CH","7":"NGI_CHINA","8":"NGI_CZ","9":"NGI_DE","10":"NGI_FI","11":"NGI_FRANCE","12":"NGI_GE","13":"NGI_ | |||
GRNET","14":"NGI_HR","15":"NGI_HU","16":"NGI_IBERGRID","17":"NGI_IL","18":"NGI_IT","19":"NGI_MD","20":"NGI_NDGF | |||
","21":"NGI_NL","22":"NGI_PL","23":"NGI_RO","24":"NGI_SI","25":"NGI_SK","26":"NGI_TR","27":"NGI_UA","28":"NGI_U | |||
K","29":"ROC_Canada","30":"ROC_LA","31":"Russia"}, {"id":"ylegend","0":"2016-06","1":"2016-07","2":"2016- | |||
08","3":"2016-09","4":"2016-10","5":"2016-11","6":"2016-12","7":"2017-01","8":"2017-02","9":"id"}, | |||
{"id":"var","xrange":"DATE","yrange":"REGION","query":"njobs"}] | |||
</source> | |||
[[Accounting_Portal_API | Back to Accounting Portal API]] |
Latest revision as of 14:22, 12 June 2017
This view offers an overview of the EGI HTC infrastructure viewed by Operations Centres (NGIs).
Metric
The Row and Column variables define the dimensions in which the data are grouped, for each view usually the data are grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Metric | Name | Description |
---|---|---|
number_of_jobs | Number of jobs | Number of discrete jobs launched, independent on completion status. |
normalized_sum_cpu | Normalized Sum CPU | Hours of CPU time consumed in calculations by jobs, normalised using the HEPSPEC06 benchmark. |
sum_cpu | Sum CPU | Hours of not normalised CPU time consumed by jobs. |
normalized_sum_elapsed | Normalized Sum Elapsed | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the HEPSPEC06 benchmark. |
normalized_sum_elapsed_times_number_of_processors | Normalized Sum Elapsed * Number of Processors | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the multiplied by the number of logical cores used by each job. |
sum_elapsed_times_number_of_processors | Sum Elapsed * Number of Processors | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by the job. |
sum_elapsed | Unnormalized Wall CPU Hours | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks. |
cpu_efficiency | CPU Efficiency | Normalised Sum CPU / (Normalised Sum Elapsed * Number of Processors). Measures the percentage of running time of the process used to do actual computations. |
Metric Unit Suffix
For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is
<metric>-<unit_suffix>
For example:
normcpu-month
If no suffix is indicated, hours are selected by default:
Suffix | Name | Description |
---|---|---|
hour | Hours | Select the default unit of hours, equivalent to not indicating any suffix. |
day | Days | Select a time of unit based on days (24 hours) |
month | Months | Select a time of unit based on months (720 hours) |
quarter | Quarters | Select a time of unit based on quarters (2160 hours) |
halfyear | Half Years | Select a time of unit based on half years (4320 hours) |
year | Years | Select a time of unit based on years (8640 hours) |
Row and Column Variables
The Row and Column varaibles define the dimensions in which the data is grouped, for each view usually the data is grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Variable | Name | Description |
---|---|---|
DATE | Month | Grouping by Month |
Year | Year | Grouping by Year |
3M | Quarter | Grouping by Quarter |
6M | Half-year | Grouping by Half-year |
COUNTRY | Country | Grouping by Country |
REGION | Operations Centre | Grouping by Operations Centre |
SITE | Resource Centre | Grouping by Resource Centre |
VO | VO | Grouping by VO |
SubmitHost | SubmitHost | Grouping by SubmitHost |
Resource Centre | SITE | Group by Resource Centre |
NUMBER PROCESSORS | Number of Processors | Group by the number of Logical Cores. |
NODECOUNT | Nodes | Group by Nodes |
Dates
The URL components for dates are:
/StartYear/StartMonth/EndYear/EndMonth/
The date format is:
/YYYY/MM/YYYY/MM/
The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth.
VO Group Selector
VO Group selects which VOs are selected for the current view, there are five settings:
VO Group Selector | Meaning |
---|---|
lhc | The four VOs used on the WLCG Large Hadron Collider (ATLAS, ALICE, LHCb, CMS). |
top10 | Top 10 VOs that in the period and options selected have the highest unnormalised raw CPU value. |
all | All VOs are included. |
egi | VOs officially registered on the EGI Operations Portal. |
custom | Custom set of virtual organisations. |
In the case of a “custom” selection, the used format is
custom-vo1,vo2,vo3,..,von
This scales to infrastructure views with 200+ VOs.
Local Job Selector
The Local Job selector parameter filters accounting based on its use of local vs infrastructure reosurce
VO Group Selector | Meaning |
---|---|
onlyinfrajobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
localinfrajobs | All jobs executed by the EGI Federation, regardless of the submission mode. |
onlylocaljobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
Data Selector
Data Selector | Meaning |
---|---|
/JSON/ | Returns data as a text file containing JSON (Java Script Object Notation) data. |
/CSV/ | Returns data as a text file containing CSV (Comma Separated Values) data. |
Output Example
https://accounting-pre.egi.cesga.es/egi/htc/operations_centres/number_of_jobs/REGION/DATE/2016/6/2017/6/egi/onlyinfrajobs/JSON/
[{"2016-06":219164,"2016-07":164009,"2016-08":246484,"2016-09":171441,"2016-10":110200,"2016-11":140790,
"2016-12":54516,"2017-01":139085,"2017-02":30541,"id":"AfricaArabia","Total":1276230,"Percent":0.3}, {"2016-
06":2075301,"2016-07":2343848,"2016-08":2165980,"2016-09":2455284,"2016-10":2699340,"2016-11":2676404,"2016-
12":2649081,"2017-01":3022985,"2017-02":1717884,"id":"AsiaPacific","Total":21806107,"Percent":5.06}, {"2016-
06":5168077,"2016-07":9503484,"2016-08":8512419,"2016-09":8729766,"2016-10":6092806,"2016-11":9139894,"2016-
12":8713986,"2017-01":9729395,"2017-02":5508540,"id":"CERN","Total":71098367,"Percent":16.5}, {"2016-
06":22966,"2016-07":25770,"2016-08":23796,"2016-09":25658,"2016-10":25232,"2016-11":24717,"2016-
12":25744,"2017-01":25962,"2017-02":16267,"id":"NGI_AEGIS","Total":216112,"Percent":0.05}, {"2016-
06":3249,"2016-07":1947,"2016-08":3218,"2016-09":3843,"2016-10":5204,"2016-11":3138,"2016-12":3245,"2017-
01":3015,"2017-02":1121,"id":"NGI_ARMGRID","Total":27980,"Percent":0.01}, {"2016-06":36516,"2016-
07":31644,"2016-08":31030,"2016-09":11946,"2016-10":26255,"2016-11":19714,"2016-12":14915,"2017-
01":16721,"2017-02":7147,"id":"NGI_BG","Total":195888,"Percent":0.05}, {"2016-06":464959,"2016-
07":496585,"2016-08":511609,"2016-09":457829,"2016-10":523571,"2016-11":590077,"2016-12":569312,"2017-
01":422912,"2017-02":520081,"id":"NGI_CH","Total":4556935,"Percent":1.06}, {"2016-06":109807,"2016-
07":152312,"2016-08":97788,"2016-09":157858,"2016-10":121591,"2016-11":140423,"2016-12":146933,"2017-
01":163161,"2017-02":90258,"id":"NGI_CHINA","Total":1180131,"Percent":0.27}, {"2016-06":419707,"2016-
07":622278,"2016-08":333445,"2016-09":324516,"2016-10":441747,"2016-11":523249,"2016-12":392839,"2017-
01":738375,"2017-02":268361,"id":"NGI_CZ","Total":4064517,"Percent":0.94}, {"2016-06":6249765,"2016-
07":7106409,"2016-08":5524328,"2016-09":5403365,"2016-10":6225585,"2016-11":5203500,"2016-12":4667233,"2017-
01":4685814,"2017-02":3515577,"id":"NGI_DE","Total":48581576,"Percent":11.27}, {"2016-06":39302,"2016-
07":43607,"2016-08":23509,"2016-09":21931,"2016-10":19937,"2016-11":31761,"2016-12":25990,"2017-
01":17318,"2017-02":9092,"id":"NGI_FI","Total":232447,"Percent":0.05}, {"2016-06":5132661,"2016-
07":5431276,"2016-08":4063110,"2016-09":3925907,"2016-10":4857282,"2016-11":5366228,"2016-12":5193655,"2017-
01":4703217,"2017-02":2671748,"id":"NGI_FRANCE","Total":41345084,"Percent":9.59}, {"2016-06":8714,"2016-
07":7582,"2016-08":8244,"2016-09":7563,"2016-10":8527,"2016-11":7806,"2016-12":8164,"2017-01":8470,"2017-
02":7459,"id":"NGI_GE","Total":72529,"Percent":0.02}, {"2016-06":89845,"2016-07":76572,"2016-08":48624,"2016-
09":64850,"2016-10":84817,"2016-11":103774,"2016-12":82437,"2017-01":86706,"2017-
02":36689,"id":"NGI_GRNET","Total":674314,"Percent":0.16}, {"2016-06":10454,"2016-07":9428,"2016-
08":6613,"2016-09":3274,"2016-10":5828,"2016-11":4741,"2016-12":5184,"2017-01":5598,"2017-
02":5665,"id":"NGI_HR","Total":56785,"Percent":0.01}, {"2016-06":108292,"2016-07":121147,"2016-
08":111935,"2016-09":131450,"2016-10":165201,"2016-11":147052,"2016-12":142005,"2017-01":123487,"2017-
02":83204,"id":"NGI_HU","Total":1133773,"Percent":0.26}, {"2016-06":2307912,"2016-07":2948625,"2016-
08":2295666,"2016-09":1798655,"2016-10":1739477,"2016-11":1931112,"2016-12":1735031,"2017-01":2036922,"2017-
02":1046522,"id":"NGI_IBERGRID","Total":17839922,"Percent":4.14}, {"2016-06":519638,"2016-07":766188,"2016-
08":759286,"2016-09":540829,"2016-10":733266,"2016-11":515202,"2016-12":356118,"2017-01":394209,"2017-
02":286127,"id":"NGI_IL","Total":4870863,"Percent":1.13}, {"2016-06":6246608,"2016-07":6644789,"2016-
08":6174721,"2016-09":4945670,"2016-10":5110056,"2016-11":5114330,"2016-12":4172565,"2017-01":4157040,"2017-
02":2551246,"id":"NGI_IT","Total":45117025,"Percent":10.47}, {"2016-06":56,"2016-07":1,"2016-08":0,"2016-
09":0,"2016-10":1,"2016-11":1,"2016-12":0,"2017-01":0,"2017-02":0,"id":"NGI_MD","Total":59,"Percent":0},
{"2016-06":591910,"2016-07":885542,"2016-08":626650,"2016-09":752257,"2016-10":538775,"2016-11":880280,"2016-
12":986051,"2017-01":841145,"2017-02":342342,"id":"NGI_NDGF","Total":6444952,"Percent":1.5}, {"2016-
06":1701275,"2016-07":2021610,"2016-08":1570774,"2016-09":1716250,"2016-10":1494287,"2016-11":2140621,"2016-
12":1630570,"2017-01":2041008,"2017-02":974983,"id":"NGI_NL","Total":15291378,"Percent":3.55}, {"2016-
06":456197,"2016-07":437478,"2016-08":361102,"2016-09":390919,"2016-10":448349,"2016-11":536580,"2016-
12":385494,"2017-01":406321,"2017-02":208844,"id":"NGI_PL","Total":3631284,"Percent":0.84}, {"2016-
06":1039451,"2016-07":978509,"2016-08":1025222,"2016-09":1000574,"2016-10":1108580,"2016-11":1125055,"2016-
12":973822,"2017-01":1058285,"2017-02":682640,"id":"NGI_RO","Total":8992138,"Percent":2.09}, {"2016-
06":323791,"2016-07":337088,"2016-08":466323,"2016-09":568205,"2016-10":460448,"2016-11":575114,"2016-
12":507834,"2017-01":640344,"2017-02":291847,"id":"NGI_SI","Total":4170994,"Percent":0.97}, {"2016-
06":318750,"2016-07":491367,"2016-08":464943,"2016-09":386949,"2016-10":233969,"2016-11":246260,"2016-
12":215088,"2017-01":375396,"2017-02":225941,"id":"NGI_SK","Total":2958663,"Percent":0.69}, {"2016-
06":191830,"2016-07":76867,"2016-08":95198,"2016-09":49766,"2016-10":76174,"2016-11":80574,"2016-
12":49587,"2017-01":70533,"2017-02":52213,"id":"NGI_TR","Total":742742,"Percent":0.17}, {"2016-
06":107158,"2016-07":127115,"2016-08":114712,"2016-09":126440,"2016-10":96335,"2016-11":104293,"2016-
12":98550,"2017-01":81163,"2017-02":41632,"id":"NGI_UA","Total":897398,"Percent":0.21}, {"2016-
06":8825217,"2016-07":10829331,"2016-08":7249724,"2016-09":6861761,"2016-10":7175525,"2016-11":7742905,"2016-
12":8072590,"2017-01":7729046,"2017-02":4842588,"id":"NGI_UK","Total":69328687,"Percent":16.09}, {"2016-
06":4689263,"2016-07":5032828,"2016-08":4627810,"2016-09":4020272,"2016-10":4835534,"2016-11":3838035,"2016-
12":4118913,"2017-01":4614193,"2017-02":2104736,"id":"ROC_Canada","Total":37881584,"Percent":8.79}, {"2016-
06":290699,"2016-07":578930,"2016-08":267672,"2016-09":237313,"2016-10":212189,"2016-11":276339,"2016-
12":248714,"2017-01":260469,"2017-02":157167,"id":"ROC_LA","Total":2529492,"Percent":0.59}, {"2016-
06":1342566,"2016-07":1471906,"2016-08":1316836,"2016-09":1510919,"2016-10":1597201,"2016-11":1833431,"2016-
12":1708028,"2017-01":1971692,"2017-02":1018750,"id":"Russia","Total":13771329,"Percent":3.2}, {"2016-
06":49111100,"2016-07":59766072,"2016-08":49128771,"2016-09":46803260,"2016-10":47273289,"2016-
11":51063400,"2016-12":47954194,"2017-01":50569987,"2017-02":29317212,"id":"Total","Total":430987285,"Percent":""}, {"2016-06":"11.40%","2016-07":"13.87%","2016-
08":"11.40%","2016-09":"10.86%","2016-10":"10.97%","2016-11":"11.85%","2016-12":"11.13%","2017-
01":"11.73%","2017-02":"6.80%","id":"Percent","Percent":"","Total":""},
{"id":"xlegend","0":"AfricaArabia","1":"AsiaPacific","2":"CERN","3":"NGI_AEGIS","4":"NGI_ARMGRID","5":"NGI_BG",
"6":"NGI_CH","7":"NGI_CHINA","8":"NGI_CZ","9":"NGI_DE","10":"NGI_FI","11":"NGI_FRANCE","12":"NGI_GE","13":"NGI_
GRNET","14":"NGI_HR","15":"NGI_HU","16":"NGI_IBERGRID","17":"NGI_IL","18":"NGI_IT","19":"NGI_MD","20":"NGI_NDGF
","21":"NGI_NL","22":"NGI_PL","23":"NGI_RO","24":"NGI_SI","25":"NGI_SK","26":"NGI_TR","27":"NGI_UA","28":"NGI_U
K","29":"ROC_Canada","30":"ROC_LA","31":"Russia"}, {"id":"ylegend","0":"2016-06","1":"2016-07","2":"2016-
08","3":"2016-09","4":"2016-10","5":"2016-11","6":"2016-12","7":"2017-01","8":"2017-02","9":"id"},
{"id":"var","xrange":"DATE","yrange":"REGION","query":"njobs"}]