Difference between revisions of "Accounting Portal API Function EGI HTC"
Line 15: | Line 15: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Scope | !Accepted Values for Scope | ||
!Name | !Name | ||
!Description | !Description | ||
Line 48: | Line 48: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 87: | Line 87: | ||
Metric values that are only useable in cloud views: | Metric values that are only useable in cloud views: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 124: | Line 124: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 193: | Line 193: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Suffix | !Accepted Values for Suffix | ||
!Name | !Name | ||
!Description | !Description | ||
Line 225: | Line 225: | ||
Metric values that are only useable in cloud views: | Metric values that are only useable in cloud views: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 266: | Line 266: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 305: | Line 305: | ||
Metric values that are only useable in cloud views: | Metric values that are only useable in cloud views: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 342: | Line 342: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Metric | !Accepted Values for Metric | ||
!Name | !Name | ||
!Description | !Description | ||
Line 408: | Line 408: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Variable | !Accepted Values for Variable | ||
!Name | !Name | ||
!Description | !Description | ||
Line 465: | Line 465: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Variable | !Accepted Values for Variable | ||
!Name | !Name | ||
!Description | !Description | ||
Line 496: | Line 496: | ||
{|class="wikitable" | {|class="wikitable" | ||
!Variable | !Accepted Values for Variable | ||
!Name | !Name | ||
!Description | !Description | ||
Line 528: | Line 528: | ||
{|class="wikitable" | {|class="wikitable" | ||
!VO Group Selector | !Accepted Values for VO Group Selector | ||
!Meaning | !Meaning | ||
|- | |- | ||
Line 561: | Line 561: | ||
{|class="wikitable" | {|class="wikitable" | ||
!VO Group Selector | !Accepted Values for VO Group Selector | ||
!Meaning | !Meaning | ||
|- | |- | ||
Line 579: | Line 579: | ||
{|class=wikitable | {|class=wikitable | ||
!Data Selector | !Accepted Values for Data Selector | ||
!Meaning | !Meaning | ||
|- | |- |
Revision as of 12:33, 19 June 2017
EGI HTC API
This function returns data from the EGI HTC infrastructure.
/egi/htc/<scope>/<metric>/<row_variable>/<column_variable>/<start_date>/<end_date>/<VO_group>/<Local_Job_Filter>/<output_format>
All parameters except scope can be omitted and defaults will be selected, but all of them will need to be included otherwise.
Scope
Mandatory attribute
The Scope parameter limits the data that is displayed in the view to a concrete unit in a hierarchical organization.
Accepted Values for Scope | Name | Description |
---|---|---|
operations_centres | Operations Centres | All the HTC infrastructure by Operations Centre (NGI). |
countries | Countries | All the HTC infrastructure by Country. |
operation_centre/<operation_centre_name> | Individual Operation Centre | HTC accounting from a concrete Operations Centre (NGI). |
country/<country_name> | Individual Country | HTC accounting from a concrete Country. |
resource_centre/<resource_centre_name> | Individual Resource Centre (Site). | HTC accounting from a concrete Resource Centre (Site). |
Metric
Optional attribute
The Row and Column variables define the dimensions in which the data are grouped, for each view usually the data are grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Accepted Values for Metric | Name | Description |
---|---|---|
number_of_jobs | Number of jobs | Number of discrete jobs launched, independent on completion status. |
normalized_sum_cpu | Normalized Sum CPU | Hours of CPU time consumed in calculations by jobs, normalised using the HEPSPEC06 benchmark. |
sum_cpu | Sum CPU | Hours of not normalised CPU time consumed by jobs. |
normalized_sum_elapsed | Normalized Sum Elapsed | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the HEPSPEC06 benchmark. |
normalized_sum_elapsed_times_number_of_processors | Normalized Sum Elapsed * Number of Processors | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the multiplied by the number of logical cores used by each job. |
sum_elapsed_times_number_of_processors | Sum Elapsed * Number of Processors | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by the job. |
sum_elapsed | Unnormalized Wall CPU Hours | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks. |
cpu_efficiency | CPU Efficiency | Normalised Sum CPU / (Normalised Sum Elapsed * Number of Processors). Measures the percentage of running time of the process used to do actual computations. |
Metric Unit Suffix
For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is
<metric>-<unit_suffix>
For example:
normcpu-month
If no suffix is indicated, hours are selected by default:
Accepted Values for Suffix | Name | Description |
---|---|---|
hour | Hours | Select the default unit of hours, equivalent to not indicating any suffix. |
day | Days | Select a time of unit based on days (24 hours) |
month | Months | Select a time of unit based on months (720 hours) |
quarter | Quarters | Select a time of unit based on quarters (2160 hours) |
halfyear | Half Years | Select a time of unit based on half years (4320 hours) |
year | Years | Select a time of unit based on years (8640 hours) |
Row and Column Variables
Optional attribute
The Row and Column varaibles define the dimensions in which the data is grouped, for each view usually the data is grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Accepted Values for Variable | Name | Description |
---|---|---|
DATE | Month | Grouping by Month |
Year | Year | Grouping by Year |
3M | Quarter | Grouping by Quarter |
6M | Half-year | Grouping by Half-year |
COUNTRY | Country | Grouping by Country |
REGION | Operations Centre | Grouping by Operations Centre |
SITE | Resource Centre | Grouping by Resource Centre |
VO | VO | Grouping by VO |
SubmitHost | SubmitHost | Grouping by SubmitHost |
Resource Centre | SITE | Group by Resource Centre |
NUMBER PROCESSORS | Number of Processors | Group by the number of Logical Cores. |
NODECOUNT | Nodes | Group by Nodes |
Dates
Optional attribute
The URL components for dates are:
/StartYear/StartMonth/EndYear/EndMonth/
The date format is:
/YYYY/MM/YYYY/MM/
The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth.
VO Group Selector
VO Group selects which VOs are selected for the current view, there are five settings:
Accepted Values for VO Group Selector | Meaning |
---|---|
lhc | The four VOs used on the WLCG Large Hadron Collider (ATLAS, ALICE, LHCb, CMS). |
top10 | Top 10 VOs that in the period and options selected have the highest unnormalised raw CPU value. |
all | All VOs are included. |
egi | VOs officially registered on the EGI Operations Portal. |
custom | Custom set of virtual organisations. |
In the case of a “custom” selection, the used format is
custom-vo1,vo2,vo3,..,von
This scales to infrastructure views with 200+ VOs.
Local Job Selector
Optional attribute
The Local Job selector parameter filters accounting based on its use of local vs infrastructure reosurce
Accepted Values for VO Group Selector | Meaning |
---|---|
onlyinfrajobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
localinfrajobs | All jobs executed by the EGI Federation, regardless of the submission mode. |
onlylocaljobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
Data Selector
Optional attribute
Accepted Values for Data Selector | Meaning |
---|---|
/JSON/ | Returns data as a text file containing JSON (Java Script Object Notation) data. |
/CSV/ | Returns data as a text file containing CSV (Comma Separated Values) data. |
Output Example
https://accounting-pre.egi.cesga.es/egi/htc/countries/number_of_jobs/REGION/DATE/2016/6/2017/6/egi/onlyinfrajobs/JSON/
[{"2016-06":13661,"2016-07":2752,"2016-08":429,"2016-09":0,"2016-10":802,"2016-11":1914,"2016-12":0,"2017-01":3029,"2017-02":1374,"id":"Algeria","Total":23961,"Percent":0.01}, {"2016-06":3249,"2016-07":1947,"2016-
08":3218,"2016-09":3843,"2016-10":5204,"2016-11":3138,"2016-12":3245,"2017-01":3015,"2017-02":1121,"id":"Armenia","Total":27980,"Percent":0.01}, {"2016-06":291859,"2016-07":459222,"2016-08":373650,"2016-
09":366404,"2016-10":373292,"2016-11":405356,"2016-12":307101,"2017-01":346898,"2017-02":205973,"id":"Australia","Total":3129755,"Percent":0.73}, {"2016-06":80438,"2016-07":100363,"2016-08":135644,"2016-
09":124887,"2016-10":102366,"2016-11":115432,"2016-12":87951,"2017-01":95503,"2017-02":77308,"id":"Austria","Total":919892,"Percent":0.21}, {"2016-06":104782,"2016-07":92461,"2016-08":72188,"2016-09":89207,"2016-
10":114044,"2016-11":84188,"2016-12":65494,"2017-01":73037,"2017-02":43199,"id":"Belgium","Total":738600,"Percent":0.17}, {"2016-06":85973,"2016-07":93292,"2016-08":93644,"2016-09":88170,"2016-10":88410,"2016-
11":92791,"2016-12":91828,"2017-01":90197,"2017-02":51692,"id":"Brazil","Total":775997,"Percent":0.18}, {"2016-06":36516,"2016-07":31644,"2016-08":31030,"2016-09":11946,"2016-10":26255,"2016-11":19714,"2016-
12":14915,"2017-01":16721,"2017-02":7147,"id":"Bulgaria","Total":195888,"Percent":0.05}, {"2016-06":4689263,"2016-07":5032828,"2016-08":4627810,"2016-09":4020272,"2016-10":4835534,"2016-11":3838035,"2016-
12":4118913,"2017-01":4614193,"2017-02":2104736,"id":"Canada","Total":37881584,"Percent":8.79}, {"2016-06":136574,"2016-07":207197,"2016-08":126402,"2016-09":96322,"2016-10":82400,"2016-11":139013,"2016-
12":120527,"2017-01":136777,"2017-02":81318,"id":"Chile","Total":1126530,"Percent":0.26}, {"2016-06":114452,"2016-07":157259,"2016-08":102680,"2016-09":162608,"2016-10":126417,"2016-11":144989,"2016-
12":151508,"2017-01":166410,"2017-02":92223,"id":"China","Total":1218546,"Percent":0.28}, {"2016-06":10454,"2016-07":9428,"2016-08":6613,"2016-09":3274,"2016-10":5828,"2016-11":4741,"2016-12":5184,"2017-
01":5598,"2017-02":5665,"id":"Croatia","Total":56785,"Percent":0.01}, {"2016-06":419707,"2016-07":622278,"2016-08":333445,"2016-09":324516,"2016-10":441747,"2016-11":523249,"2016-12":392839,"2017-01":738375,"2017-
02":268361,"id":"Czech Republic","Total":4064517,"Percent":0.94}, {"2016-06":3414,"2016-07":2994,"2016-08":1863,"2016-09":1571,"2016-10":1099,"2016-11":2165,"2016-12":1852,"2017-01":1316,"2017-
02":671,"id":"Denmark","Total":16945,"Percent":0}, {"2016-06":40183,"2016-07":47781,"2016-08":34512,"2016-09":30190,"2016-10":54309,"2016-11":54000,"2016-12":117167,"2017-01":82523,"2017-
02":42673,"id":"Estonia","Total":503338,"Percent":0.12}, {"2016-06":68911,"2016-07":83530,"2016-08":66000,"2016-09":64274,"2016-10":57298,"2016-11":137065,"2016-12":105629,"2017-01":88186,"2017-
02":56777,"id":"Finland","Total":727670,"Percent":0.17}, {"2016-06":5132661,"2016-07":5431276,"2016-08":4063110,"2016-09":3925907,"2016-10":4857282,"2016-11":5366228,"2016-12":5193655,"2017-01":4703217,"2017-
02":2671748,"id":"France","Total":41345084,"Percent":9.59}, {"2016-06":8714,"2016-07":7582,"2016-08":8244,"2016-09":7563,"2016-10":8527,"2016-11":7806,"2016-12":8164,"2017-01":8470,"2017-
02":7459,"id":"Georgia","Total":72529,"Percent":0.02}, {"2016-06":6249765,"2016-07":7106409,"2016-08":5524328,"2016-09":5403365,"2016-10":6225585,"2016-11":5203500,"2016-12":4667233,"2017-01":4685814,"2017-
02":3515577,"id":"Germany","Total":48581576,"Percent":11.27}, {"2016-06":89845,"2016-07":76572,"2016-08":48624,"2016-09":64850,"2016-10":84817,"2016-11":103774,"2016-12":82437,"2017-01":86706,"2017-
02":36689,"id":"Greece","Total":674314,"Percent":0.16}, {"2016-06":108292,"2016-07":121147,"2016-08":111935,"2016-09":131450,"2016-10":165201,"2016-11":147052,"2016-12":142005,"2017-01":123487,"2017-
02":83204,"id":"Hungary","Total":1133773,"Percent":0.26}, {"2016-06":67810,"2016-07":101484,"2016-08":89384,"2016-09":198465,"2016-10":239001,"2016-11":36112,"2016-12":134808,"2017-01":123529,"2017-
02":59639,"id":"India","Total":1050232,"Percent":0.24}, {"2016-06":10214,"2016-07":14832,"2016-08":14350,"2016-09":11917,"2016-10":14721,"2016-11":21637,"2016-12":22400,"2017-01":20802,"2017-
02":12853,"id":"Iran","Total":143726,"Percent":0.03}, {"2016-06":519638,"2016-07":766188,"2016-08":759286,"2016-09":540829,"2016-10":733266,"2016-11":515202,"2016-12":356118,"2017-01":394209,"2017-
02":286127,"id":"Israel","Total":4870863,"Percent":1.13}, {"2016-06":6166170,"2016-07":6544426,"2016-08":6039077,"2016-09":4820783,"2016-10":5007690,"2016-11":4998898,"2016-12":4084614,"2017-01":4061537,"2017-
02":2473938,"id":"Italy","Total":44197133,"Percent":10.25}, {"2016-06":846750,"2016-07":961600,"2016-08":1033948,"2016-09":1117254,"2016-10":1060594,"2016-11":1251950,"2016-12":1387511,"2017-01":1583539,"2017-
02":812695,"id":"Japan","Total":10055841,"Percent":2.33}, {"2016-06":5785,"2016-07":8326,"2016-08":5749,"2016-09":4538,"2016-10":0,"2016-11":0,"2016-12":0,"2017-01":2849,"2017-
02":875,"id":"Malaysia","Total":28122,"Percent":0.01}, {"2016-06":68152,"2016-07":278441,"2016-08":47626,"2016-09":52821,"2016-10":41379,"2016-11":44535,"2016-12":36359,"2017-01":33495,"2017-
02":24157,"id":"Mexico","Total":626965,"Percent":0.15}, {"2016-06":56,"2016-07":1,"2016-08":0,"2016-09":0,"2016-10":1,"2016-11":1,"2016-12":0,"2017-01":0,"2017-02":0,"id":"Moldova","Total":59,"Percent":0}, {"2016-
06":2986,"2016-07":7497,"2016-08":6281,"2016-09":800,"2016-10":980,"2016-11":1722,"2016-12":0,"2017-01":4362,"2017-02":1640,"id":"Morocco","Total":26268,"Percent":0.01}, {"2016-06":1596493,"2016-07":1929149,"2016-
08":1498586,"2016-09":1627043,"2016-10":1380243,"2016-11":2056433,"2016-12":1565076,"2017-01":1967971,"2017-02":931784,"id":"Netherlands","Total":14552778,"Percent":3.38}, {"2016-06":45231,"2016-07":45607,"2016-
08":47860,"2016-09":45050,"2016-10":51944,"2016-11":52402,"2016-12":40473,"2017-01":56608,"2017-02":97712,"id":"Pakistan","Total":482887,"Percent":0.11}, {"2016-06":456197,"2016-07":437478,"2016-08":361102,"2016-
09":390919,"2016-10":448349,"2016-11":536580,"2016-12":385494,"2017-01":406321,"2017-02":208844,"id":"Poland","Total":3631284,"Percent":0.84}, {"2016-06":234085,"2016-07":181328,"2016-08":188300,"2016-
09":241098,"2016-10":137100,"2016-11":269543,"2016-12":203187,"2017-01":205961,"2017-02":154912,"id":"Portugal","Total":1815514,"Percent":0.42}, {"2016-06":1039451,"2016-07":978509,"2016-08":1025222,"2016-
09":1000574,"2016-10":1108580,"2016-11":1125055,"2016-12":973822,"2017-01":1058285,"2017-02":682640,"id":"Romania","Total":8992138,"Percent":2.09}, {"2016-06":1342566,"2016-07":1471906,"2016-08":1316836,"2016-
09":1510919,"2016-10":1597201,"2016-11":1833431,"2016-12":1708028,"2017-01":1971692,"2017-02":1018750,"id":"Russia","Total":13771329,"Percent":3.2}, {"2016-06":22966,"2016-07":25770,"2016-08":23796,"2016-
09":25658,"2016-10":25232,"2016-11":24717,"2016-12":25744,"2017-01":25962,"2017-02":16267,"id":"Serbia","Total":216112,"Percent":0.05}, {"2016-06":318750,"2016-07":491367,"2016-08":464943,"2016-09":386949,"2016-
10":233969,"2016-11":246260,"2016-12":215088,"2017-01":375396,"2017-02":225941,"id":"Slovakia","Total":2958663,"Percent":0.69}, {"2016-06":323791,"2016-07":337088,"2016-08":466323,"2016-09":568205,"2016-
10":460448,"2016-11":575114,"2016-12":507834,"2017-01":640344,"2017-02":291847,"id":"Slovenia","Total":4170994,"Percent":0.97}, {"2016-06":202517,"2016-07":153760,"2016-08":239774,"2016-09":170641,"2016-
10":108418,"2016-11":137154,"2016-12":54516,"2017-01":131694,"2017-02":27527,"id":"South Africa","Total":1226001,"Percent":0.28}, {"2016-06":229904,"2016-07":254695,"2016-08":222586,"2016-09":238886,"2016-
10":242506,"2016-11":219178,"2016-12":224613,"2017-01":205605,"2017-02":121551,"id":"South Korea","Total":1959524,"Percent":0.45}, {"2016-06":2073827,"2016-07":2767297,"2016-08":2107366,"2016-09":1557557,"2016-
10":1602377,"2016-11":1661569,"2016-12":1531844,"2017-01":1830961,"2017-02":891610,"id":"Spain","Total":16024408,"Percent":3.72}, {"2016-06":518704,"2016-07":794844,"2016-08":547784,"2016-09":678153,"2016-
10":446006,"2016-11":718811,"2016-12":787393,"2017-01":686438,"2017-02":251313,"id":"Sweden","Total":5429446,"Percent":1.26}, {"2016-06":5633036,"2016-07":10000069,"2016-08":9024028,"2016-09":9187595,"2016-
10":6616377,"2016-11":9729971,"2016-12":9283298,"2017-01":10152307,"2017-02":6028621,"id":"Switzerland","Total":75655302,"Percent":17.55}, {"2016-06":563484,"2016-07":483833,"2016-08":363070,"2016-09":457901,"2016-
10":700604,"2016-11":671881,"2016-12":515346,"2017-01":664942,"2017-02":401226,"id":"Taiwan","Total":4822287,"Percent":1.12}, {"2016-06":9619,"2016-07":9302,"2016-08":10491,"2016-09":10119,"2016-10":11852,"2016-
11":13322,"2016-12":12254,"2017-01":14964,"2017-02":3395,"id":"Thailand","Total":95318,"Percent":0.02}, {"2016-06":191830,"2016-07":76867,"2016-08":95198,"2016-09":49766,"2016-10":76174,"2016-11":80574,"2016-
12":49587,"2017-01":70533,"2017-02":52213,"id":"Turkey","Total":742742,"Percent":0.17}, {"2016-06":107158,"2016-07":127115,"2016-08":114712,"2016-09":126440,"2016-10":96335,"2016-11":104293,"2016-12":98550,"2017-
01":81163,"2017-02":41632,"id":"Ukraine","Total":897398,"Percent":0.21}, {"2016-06":8825217,"2016-07":10829331,"2016-08":7249724,"2016-09":6861761,"2016-10":7175525,"2016-11":7742905,"2016-12":8072590,"2017-
01":7729046,"2017-02":4842588,"id":"United Kingdom","Total":69328687,"Percent":16.09}, {"2016-06":49111100,"2016-07":59766072,"2016-08":49128771,"2016-09":46803260,"2016-10":47273289,"2016-11":51063400,"2016-
12":47954194,"2017-01":50569987,"2017-02":29317212,"id":"Total","Total":430987285,"Percent":""}, {"2016-06":"11.40%","2016-07":"13.87%","2016-08":"11.40%","2016-09":"10.86%","2016-10":"10.97%","2016-
11":"11.85%","2016-12":"11.13%","2017-01":"11.73%","2017-02":"6.80%","id":"Percent","Percent":"","Total":""},
{"id":"xlegend","0":"Algeria","1":"Armenia","2":"Australia","3":"Austria","4":"Belgium","5":"Brazil","6":"Bulgaria","7":"Canada","8":"Chile","9":"China","10":"Croatia","11":"Czech
Republic","12":"Denmark","13":"Estonia","14":"Finland","15":"France","16":"Georgia","17":"Germany","18":"Greece","19":"Hungary","20":"India","21":"Iran","22":"Israel","23":"Italy","24":"Japan","25":"Malaysia","26":
"Mexico","27":"Moldova","28":"Morocco","29":"Netherlands","30":"Pakistan","31":"Poland","32":"Portugal","33":"Romania","34":"Russia","35":"Serbia","36":"Slovakia","37":"Slovenia","38":"South Africa","39":"South
Korea","40":"Spain","41":"Sweden","42":"Switzerland","43":"Taiwan","44":"Thailand","45":"Turkey","46":"Ukraine","47":"United Kingdom"}, {"id":"ylegend","0":"2016-06","1":"2016-07","2":"2016-08","3":"2016-
09","4":"2016-10","5":"2016-11","6":"2016-12","7":"2017-01","8":"2017-02","9":"id"}, {"id":"var","xrange":"DATE","yrange":"REGION","query":"njobs"}]