Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Accounting Portal API Function EGI HTC

From EGIWiki
Revision as of 11:26, 19 June 2017 by Ivandiaz (talk | contribs)
Jump to navigation Jump to search

EGI HTC API

This function returns data from the EGI HTC infrastructure.

/egi/htc/<scope>/<metric>/<row_variable>/<column_variable>/<start_date>/<end_date>/<VO_group>/<Local_Job_Filter>/<output_format>

All parameters except can be omitted and defaults will be selected, but all of them will need to be included otherwise.

Scope

The Scope parameter limits the data that is displayed in the view to a concrete unit in a hierarchical organization. If no scope is present, the default is all of the infrastructure, with optional limitations depending on the view.

Scope Name Description
<None> Default Scope All the HTC infrastructure.
operations_centres Operations Centres All the HTC infrastructure by Operations Centre (NGI).
countries Countries All the HTC infrastructure by Country.
operation_centre/<operation_centre_name> Individual Operation Centre HTC accounting from a concrete Operations Centre (NGI).
country/<country_name> Individual Country HTC accounting from a concrete Country.
resource_centre/<resource_centre_name> Individual Resource Centre (Site). HTC accounting from a concrete Resource Centre (Site).

Metric Unit Suffix

For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is

<metric>-<unit_suffix>

For example:

normcpu-month

If no suffix is indicated, hours are selected by default:

Suffix Name Description
hour Hours Select the default unit of hours, equivalent to not indicating any suffix.
day Days Select a time of unit based on days (24 hours)
month Months Select a time of unit based on months (720 hours)
quarter Quarters Select a time of unit based on quarters (2160 hours)
halfyear Half Years Select a time of unit based on half years (4320 hours)
year Years Select a time of unit based on years (8640 hours)

Row and Column Variables

The Row and Column varaibles define the dimensions in which the data is grouped, for each view usually the data is grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:

Variable Name Description
DATE Month Grouping by Month
Year Year Grouping by Year
3M Quarter Grouping by Quarter
6M Half-year Grouping by Half-year
COUNTRY Country Grouping by Country
REGION Operations Centre Grouping by Operations Centre
SITE Resource Centre Grouping by Resource Centre
VO VO Grouping by VO
SubmitHost SubmitHost Grouping by SubmitHost
Resource Centre SITE Group by Resource Centre
NUMBER PROCESSORS Number of Processors Group by the number of Logical Cores.
NODECOUNT Nodes Group by Nodes


Dates

The URL components for dates are:

/StartYear/StartMonth/EndYear/EndMonth/

The date format is:

/YYYY/MM/YYYY/MM/

The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth.

VO Group Selector

VO Group selects which VOs are selected for the current view, there are five settings:

VO Group Selector Meaning
lhc The four VOs used on the WLCG Large Hadron Collider (ATLAS, ALICE, LHCb, CMS).
top10 Top 10 VOs that in the period and options selected have the highest unnormalised raw CPU value.
all All VOs are included.
egi VOs officially registered on the EGI Operations Portal.
custom Custom set of virtual organisations.


In the case of a “custom” selection, the used format is

custom-vo1,vo2,vo3,..,von

This scales to infrastructure views with 200+ VOs.

Local Job Selector

The Local Job selector parameter filters accounting based on its use of local vs infrastructure reosurce

VO Group Selector Meaning
onlyinfrajobs Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface.
localinfrajobs All jobs executed by the EGI Federation, regardless of the submission mode.
onlylocaljobs Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface.

Data Selector

Data Selector Meaning
/JSON/ Returns data as a text file containing JSON (Java Script Object Notation) data.
/CSV/ Returns data as a text file containing CSV (Comma Separated Values) data.

Output Example

https://accounting-pre.egi.cesga.es/egi/htc/countries/number_of_jobs/REGION/DATE/2016/6/2017/6/egi/onlyinfrajobs/JSON/
[{"2016-06":13661,"2016-07":2752,"2016-08":429,"2016-09":0,"2016-10":802,"2016-11":1914,"2016-12":0,"2017-01":3029,"2017-02":1374,"id":"Algeria","Total":23961,"Percent":0.01}, {"2016-06":3249,"2016-07":1947,"2016-
08":3218,"2016-09":3843,"2016-10":5204,"2016-11":3138,"2016-12":3245,"2017-01":3015,"2017-02":1121,"id":"Armenia","Total":27980,"Percent":0.01}, {"2016-06":291859,"2016-07":459222,"2016-08":373650,"2016-
09":366404,"2016-10":373292,"2016-11":405356,"2016-12":307101,"2017-01":346898,"2017-02":205973,"id":"Australia","Total":3129755,"Percent":0.73}, {"2016-06":80438,"2016-07":100363,"2016-08":135644,"2016-
09":124887,"2016-10":102366,"2016-11":115432,"2016-12":87951,"2017-01":95503,"2017-02":77308,"id":"Austria","Total":919892,"Percent":0.21}, {"2016-06":104782,"2016-07":92461,"2016-08":72188,"2016-09":89207,"2016-
10":114044,"2016-11":84188,"2016-12":65494,"2017-01":73037,"2017-02":43199,"id":"Belgium","Total":738600,"Percent":0.17}, {"2016-06":85973,"2016-07":93292,"2016-08":93644,"2016-09":88170,"2016-10":88410,"2016-
11":92791,"2016-12":91828,"2017-01":90197,"2017-02":51692,"id":"Brazil","Total":775997,"Percent":0.18}, {"2016-06":36516,"2016-07":31644,"2016-08":31030,"2016-09":11946,"2016-10":26255,"2016-11":19714,"2016-
12":14915,"2017-01":16721,"2017-02":7147,"id":"Bulgaria","Total":195888,"Percent":0.05}, {"2016-06":4689263,"2016-07":5032828,"2016-08":4627810,"2016-09":4020272,"2016-10":4835534,"2016-11":3838035,"2016-
12":4118913,"2017-01":4614193,"2017-02":2104736,"id":"Canada","Total":37881584,"Percent":8.79}, {"2016-06":136574,"2016-07":207197,"2016-08":126402,"2016-09":96322,"2016-10":82400,"2016-11":139013,"2016-
12":120527,"2017-01":136777,"2017-02":81318,"id":"Chile","Total":1126530,"Percent":0.26}, {"2016-06":114452,"2016-07":157259,"2016-08":102680,"2016-09":162608,"2016-10":126417,"2016-11":144989,"2016-
12":151508,"2017-01":166410,"2017-02":92223,"id":"China","Total":1218546,"Percent":0.28}, {"2016-06":10454,"2016-07":9428,"2016-08":6613,"2016-09":3274,"2016-10":5828,"2016-11":4741,"2016-12":5184,"2017-
01":5598,"2017-02":5665,"id":"Croatia","Total":56785,"Percent":0.01}, {"2016-06":419707,"2016-07":622278,"2016-08":333445,"2016-09":324516,"2016-10":441747,"2016-11":523249,"2016-12":392839,"2017-01":738375,"2017-
02":268361,"id":"Czech Republic","Total":4064517,"Percent":0.94}, {"2016-06":3414,"2016-07":2994,"2016-08":1863,"2016-09":1571,"2016-10":1099,"2016-11":2165,"2016-12":1852,"2017-01":1316,"2017-
02":671,"id":"Denmark","Total":16945,"Percent":0}, {"2016-06":40183,"2016-07":47781,"2016-08":34512,"2016-09":30190,"2016-10":54309,"2016-11":54000,"2016-12":117167,"2017-01":82523,"2017-
02":42673,"id":"Estonia","Total":503338,"Percent":0.12}, {"2016-06":68911,"2016-07":83530,"2016-08":66000,"2016-09":64274,"2016-10":57298,"2016-11":137065,"2016-12":105629,"2017-01":88186,"2017-
02":56777,"id":"Finland","Total":727670,"Percent":0.17}, {"2016-06":5132661,"2016-07":5431276,"2016-08":4063110,"2016-09":3925907,"2016-10":4857282,"2016-11":5366228,"2016-12":5193655,"2017-01":4703217,"2017-
02":2671748,"id":"France","Total":41345084,"Percent":9.59}, {"2016-06":8714,"2016-07":7582,"2016-08":8244,"2016-09":7563,"2016-10":8527,"2016-11":7806,"2016-12":8164,"2017-01":8470,"2017-
02":7459,"id":"Georgia","Total":72529,"Percent":0.02}, {"2016-06":6249765,"2016-07":7106409,"2016-08":5524328,"2016-09":5403365,"2016-10":6225585,"2016-11":5203500,"2016-12":4667233,"2017-01":4685814,"2017-
02":3515577,"id":"Germany","Total":48581576,"Percent":11.27}, {"2016-06":89845,"2016-07":76572,"2016-08":48624,"2016-09":64850,"2016-10":84817,"2016-11":103774,"2016-12":82437,"2017-01":86706,"2017-
02":36689,"id":"Greece","Total":674314,"Percent":0.16}, {"2016-06":108292,"2016-07":121147,"2016-08":111935,"2016-09":131450,"2016-10":165201,"2016-11":147052,"2016-12":142005,"2017-01":123487,"2017-
02":83204,"id":"Hungary","Total":1133773,"Percent":0.26}, {"2016-06":67810,"2016-07":101484,"2016-08":89384,"2016-09":198465,"2016-10":239001,"2016-11":36112,"2016-12":134808,"2017-01":123529,"2017-
02":59639,"id":"India","Total":1050232,"Percent":0.24}, {"2016-06":10214,"2016-07":14832,"2016-08":14350,"2016-09":11917,"2016-10":14721,"2016-11":21637,"2016-12":22400,"2017-01":20802,"2017-
02":12853,"id":"Iran","Total":143726,"Percent":0.03}, {"2016-06":519638,"2016-07":766188,"2016-08":759286,"2016-09":540829,"2016-10":733266,"2016-11":515202,"2016-12":356118,"2017-01":394209,"2017-
02":286127,"id":"Israel","Total":4870863,"Percent":1.13}, {"2016-06":6166170,"2016-07":6544426,"2016-08":6039077,"2016-09":4820783,"2016-10":5007690,"2016-11":4998898,"2016-12":4084614,"2017-01":4061537,"2017-
02":2473938,"id":"Italy","Total":44197133,"Percent":10.25}, {"2016-06":846750,"2016-07":961600,"2016-08":1033948,"2016-09":1117254,"2016-10":1060594,"2016-11":1251950,"2016-12":1387511,"2017-01":1583539,"2017-
02":812695,"id":"Japan","Total":10055841,"Percent":2.33}, {"2016-06":5785,"2016-07":8326,"2016-08":5749,"2016-09":4538,"2016-10":0,"2016-11":0,"2016-12":0,"2017-01":2849,"2017-
02":875,"id":"Malaysia","Total":28122,"Percent":0.01}, {"2016-06":68152,"2016-07":278441,"2016-08":47626,"2016-09":52821,"2016-10":41379,"2016-11":44535,"2016-12":36359,"2017-01":33495,"2017-
02":24157,"id":"Mexico","Total":626965,"Percent":0.15}, {"2016-06":56,"2016-07":1,"2016-08":0,"2016-09":0,"2016-10":1,"2016-11":1,"2016-12":0,"2017-01":0,"2017-02":0,"id":"Moldova","Total":59,"Percent":0}, {"2016-
06":2986,"2016-07":7497,"2016-08":6281,"2016-09":800,"2016-10":980,"2016-11":1722,"2016-12":0,"2017-01":4362,"2017-02":1640,"id":"Morocco","Total":26268,"Percent":0.01}, {"2016-06":1596493,"2016-07":1929149,"2016-
08":1498586,"2016-09":1627043,"2016-10":1380243,"2016-11":2056433,"2016-12":1565076,"2017-01":1967971,"2017-02":931784,"id":"Netherlands","Total":14552778,"Percent":3.38}, {"2016-06":45231,"2016-07":45607,"2016-
08":47860,"2016-09":45050,"2016-10":51944,"2016-11":52402,"2016-12":40473,"2017-01":56608,"2017-02":97712,"id":"Pakistan","Total":482887,"Percent":0.11}, {"2016-06":456197,"2016-07":437478,"2016-08":361102,"2016-
09":390919,"2016-10":448349,"2016-11":536580,"2016-12":385494,"2017-01":406321,"2017-02":208844,"id":"Poland","Total":3631284,"Percent":0.84}, {"2016-06":234085,"2016-07":181328,"2016-08":188300,"2016-
09":241098,"2016-10":137100,"2016-11":269543,"2016-12":203187,"2017-01":205961,"2017-02":154912,"id":"Portugal","Total":1815514,"Percent":0.42}, {"2016-06":1039451,"2016-07":978509,"2016-08":1025222,"2016-
09":1000574,"2016-10":1108580,"2016-11":1125055,"2016-12":973822,"2017-01":1058285,"2017-02":682640,"id":"Romania","Total":8992138,"Percent":2.09}, {"2016-06":1342566,"2016-07":1471906,"2016-08":1316836,"2016-
09":1510919,"2016-10":1597201,"2016-11":1833431,"2016-12":1708028,"2017-01":1971692,"2017-02":1018750,"id":"Russia","Total":13771329,"Percent":3.2}, {"2016-06":22966,"2016-07":25770,"2016-08":23796,"2016-
09":25658,"2016-10":25232,"2016-11":24717,"2016-12":25744,"2017-01":25962,"2017-02":16267,"id":"Serbia","Total":216112,"Percent":0.05}, {"2016-06":318750,"2016-07":491367,"2016-08":464943,"2016-09":386949,"2016-
10":233969,"2016-11":246260,"2016-12":215088,"2017-01":375396,"2017-02":225941,"id":"Slovakia","Total":2958663,"Percent":0.69}, {"2016-06":323791,"2016-07":337088,"2016-08":466323,"2016-09":568205,"2016-
10":460448,"2016-11":575114,"2016-12":507834,"2017-01":640344,"2017-02":291847,"id":"Slovenia","Total":4170994,"Percent":0.97}, {"2016-06":202517,"2016-07":153760,"2016-08":239774,"2016-09":170641,"2016-
10":108418,"2016-11":137154,"2016-12":54516,"2017-01":131694,"2017-02":27527,"id":"South Africa","Total":1226001,"Percent":0.28}, {"2016-06":229904,"2016-07":254695,"2016-08":222586,"2016-09":238886,"2016-
10":242506,"2016-11":219178,"2016-12":224613,"2017-01":205605,"2017-02":121551,"id":"South Korea","Total":1959524,"Percent":0.45}, {"2016-06":2073827,"2016-07":2767297,"2016-08":2107366,"2016-09":1557557,"2016-
10":1602377,"2016-11":1661569,"2016-12":1531844,"2017-01":1830961,"2017-02":891610,"id":"Spain","Total":16024408,"Percent":3.72}, {"2016-06":518704,"2016-07":794844,"2016-08":547784,"2016-09":678153,"2016-
10":446006,"2016-11":718811,"2016-12":787393,"2017-01":686438,"2017-02":251313,"id":"Sweden","Total":5429446,"Percent":1.26}, {"2016-06":5633036,"2016-07":10000069,"2016-08":9024028,"2016-09":9187595,"2016-
10":6616377,"2016-11":9729971,"2016-12":9283298,"2017-01":10152307,"2017-02":6028621,"id":"Switzerland","Total":75655302,"Percent":17.55}, {"2016-06":563484,"2016-07":483833,"2016-08":363070,"2016-09":457901,"2016-
10":700604,"2016-11":671881,"2016-12":515346,"2017-01":664942,"2017-02":401226,"id":"Taiwan","Total":4822287,"Percent":1.12}, {"2016-06":9619,"2016-07":9302,"2016-08":10491,"2016-09":10119,"2016-10":11852,"2016-
11":13322,"2016-12":12254,"2017-01":14964,"2017-02":3395,"id":"Thailand","Total":95318,"Percent":0.02}, {"2016-06":191830,"2016-07":76867,"2016-08":95198,"2016-09":49766,"2016-10":76174,"2016-11":80574,"2016-
12":49587,"2017-01":70533,"2017-02":52213,"id":"Turkey","Total":742742,"Percent":0.17}, {"2016-06":107158,"2016-07":127115,"2016-08":114712,"2016-09":126440,"2016-10":96335,"2016-11":104293,"2016-12":98550,"2017-
01":81163,"2017-02":41632,"id":"Ukraine","Total":897398,"Percent":0.21}, {"2016-06":8825217,"2016-07":10829331,"2016-08":7249724,"2016-09":6861761,"2016-10":7175525,"2016-11":7742905,"2016-12":8072590,"2017-
01":7729046,"2017-02":4842588,"id":"United Kingdom","Total":69328687,"Percent":16.09}, {"2016-06":49111100,"2016-07":59766072,"2016-08":49128771,"2016-09":46803260,"2016-10":47273289,"2016-11":51063400,"2016-
12":47954194,"2017-01":50569987,"2017-02":29317212,"id":"Total","Total":430987285,"Percent":""}, {"2016-06":"11.40%","2016-07":"13.87%","2016-08":"11.40%","2016-09":"10.86%","2016-10":"10.97%","2016-
11":"11.85%","2016-12":"11.13%","2017-01":"11.73%","2017-02":"6.80%","id":"Percent","Percent":"","Total":""}, 
{"id":"xlegend","0":"Algeria","1":"Armenia","2":"Australia","3":"Austria","4":"Belgium","5":"Brazil","6":"Bulgaria","7":"Canada","8":"Chile","9":"China","10":"Croatia","11":"Czech 
Republic","12":"Denmark","13":"Estonia","14":"Finland","15":"France","16":"Georgia","17":"Germany","18":"Greece","19":"Hungary","20":"India","21":"Iran","22":"Israel","23":"Italy","24":"Japan","25":"Malaysia","26":
"Mexico","27":"Moldova","28":"Morocco","29":"Netherlands","30":"Pakistan","31":"Poland","32":"Portugal","33":"Romania","34":"Russia","35":"Serbia","36":"Slovakia","37":"Slovenia","38":"South Africa","39":"South 
Korea","40":"Spain","41":"Sweden","42":"Switzerland","43":"Taiwan","44":"Thailand","45":"Turkey","46":"Ukraine","47":"United Kingdom"}, {"id":"ylegend","0":"2016-06","1":"2016-07","2":"2016-08","3":"2016-
09","4":"2016-10","5":"2016-11","6":"2016-12","7":"2017-01","8":"2017-02","9":"id"}, {"id":"var","xrange":"DATE","yrange":"REGION","query":"njobs"}]

Back to Accounting Portal API