Difference between revisions of "Accounting Portal API Function User HTC"
(→Scope) |
|||
Line 20: | Line 20: | ||
!Description | !Description | ||
|- | |- | ||
|''' | |'''operation_centres''' | ||
|Operations Centres | |Operations Centres | ||
|All the HTC infrastructure by Operations Centre (NGI). | |All the HTC infrastructure by Operations Centre (NGI). |
Revision as of 09:46, 1 August 2017
User HTC API
This function returns admin restricted data from concrete VOs.
/user/htc/<scope>/<metric>/<row_variable>/<column_variable>/<start_date>/<end_date>/<VO_group>/<Local_Job_Filter>/<output_format>
All parameters except scope can be omitted and defaults will be selected, but all of them will need to be included otherwise.
Scope
Mandatory attribute
Format:
<scope>
The Scope parameter limits the data that is displayed in the view to a concrete unit in a hierarchical organization.
Accepted Values for Scope | Name | Description |
---|---|---|
operation_centres | Operations Centres | All the HTC infrastructure by Operations Centre (NGI). |
countries | Countries | All the HTC infrastructure by Country. |
operation_centre/<operation_centre_name> | Individual Operation Centre | HTC accounting from a concrete Operations Centre (NGI). |
country/<country_name> | Individual Country | HTC accounting from a concrete Country. |
resource_centre/<resource_centre_name> | Individual Resource Centre (Site). | HTC accounting from a concrete Resource Centre (Site). |
Metric
Optional attribute
Format:
<metric>[-<metric_unit_suffix>]
The Row and Column variables define the dimensions in which the data are grouped, for each view usually the data are grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Accepted Values for Metric | Name | Description |
---|---|---|
number_of_jobs | Number of jobs | Number of discrete jobs launched, independent on completion status. |
normalized_sum_cpu | Normalized Sum CPU | Hours of CPU time consumed in calculations by jobs, normalised using the HEPSPEC06 benchmark. |
sum_cpu | Sum CPU | Hours of not normalised CPU time consumed by jobs. |
normalized_sum_elapsed | Normalized Sum Elapsed | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the HEPSPEC06 benchmark. |
normalized_sum_elapsed_times_number_of_processors | Normalized Sum Elapsed * Number of Processors | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the multiplied by the number of logical cores used by each job. |
sum_elapsed_times_number_of_processors | Sum Elapsed * Number of Processors | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by the job. |
sum_elapsed | Unnormalized Wall CPU Hours | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks. |
cpu_efficiency | CPU Efficiency | Normalised Sum CPU / (Normalised Sum Elapsed * Number of Processors). Measures the percentage of running time of the process used to do actual computations. |
Metric Unit Suffix
For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is
<metric>[-<metric_unit_suffix>]
For example:
normcpu-month
If no suffix is indicated, hours are selected by default:
Accepted Values for Suffix | Name | Description |
---|---|---|
hour | Hours | Select the default unit of hours, equivalent to not indicating any suffix. |
day | Days | Select a time of unit based on days (24 hours) |
month | Months | Select a time of unit based on months (720 hours) |
quarter | Quarters | Select a time of unit based on quarters (2160 hours) |
halfyear | Half Years | Select a time of unit based on half years (4320 hours) |
year | Years | Select a time of unit based on years (8640 hours) |
User HTC API
This function returns admin restricted data from their sent jobs.
/user/htc/<scope>/<metric>/<row_variable>/<column_variable>/<start_date>/<end_date>/<VO_group>/<Local_Job_Filter>/<output_format>
All parameters except scope can be omitted and defaults will be selected, but all of them will need to be included otherwise.
Scope
Mandatory attribute
Format:
<scope>
The Scope parameter limits the data that is displayed in the view to a concrete unit in a hierarchical organization.
Accepted Values for Scope | Name | Description |
---|---|---|
operations_centres | Operations Centres | All the HTC infrastructure by Operations Centre (NGI). |
countries | Countries | All the HTC infrastructure by Country. |
operation_centre/<operation_centre_name> | Individual Operation Centre | HTC accounting from a concrete Operations Centre (NGI). |
country/<country_name> | Individual Country | HTC accounting from a concrete Country. |
resource_centre/<resource_centre_name> | Individual Resource Centre (Site). | HTC accounting from a concrete Resource Centre (Site). |
Metric
Optional attribute
Format:
<metric>[-<metric_unit_suffix>]
The Row and Column variables define the dimensions in which the data are grouped, for each view usually the data are grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Accepted Values for Metric | Name | Description |
---|---|---|
number_of_jobs | Number of jobs | Number of discrete jobs launched, independent on completion status. |
normalized_sum_cpu | Normalized Sum CPU | Hours of CPU time consumed in calculations by jobs, normalised using the HEPSPEC06 benchmark. |
sum_cpu | Sum CPU | Hours of not normalised CPU time consumed by jobs. |
normalized_sum_elapsed | Normalized Sum Elapsed | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the HEPSPEC06 benchmark. |
normalized_sum_elapsed_times_number_of_processors | Normalized Sum Elapsed * Number of Processors | Hours of wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, normalised using the multiplied by the number of logical cores used by each job. |
sum_elapsed_times_number_of_processors | Sum Elapsed * Number of Processors | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks, multiplied by the number of logical cores used by the job. |
sum_elapsed | Unnormalized Wall CPU Hours | Hours of not normalised wall-clock time consumed by jobs, including I/O, OS and other housekeeping tasks. |
cpu_efficiency | CPU Efficiency | Normalised Sum CPU / (Normalised Sum Elapsed * Number of Processors). Measures the percentage of running time of the process used to do actual computations. |
Metric Unit Suffix
For all metrics with a unit based on hours, there is an optional suffix to indicate a longer time unit, the basic format is
<metric>[-<metric_unit_suffix>]
For example:
normcpu-month
If no suffix is indicated, hours are selected by default:
Accepted Values for Suffix | Name | Description |
---|---|---|
hour | Hours | Select the default unit of hours, equivalent to not indicating any suffix. |
day | Days | Select a time of unit based on days (24 hours) |
month | Months | Select a time of unit based on months (720 hours) |
quarter | Quarters | Select a time of unit based on quarters (2160 hours) |
halfyear | Half Years | Select a time of unit based on half years (4320 hours) |
year | Years | Select a time of unit based on years (8640 hours) |
Row and Column Variables
Format:
<row_variable>/<column_variable>
Optional attribute
The Row and Column varaibles define the dimensions in which the data is grouped, for each view usually the data is grouped bidimensionally using a row variable and a column variable, the name derived from appearing in the rows and columns of the table, respectively. These are the valid values for both of these:
Accepted Values for Variable | Name | Description |
---|---|---|
DATE | Month | Grouping by Month |
Year | Year | Grouping by Year |
3M | Quarter | Grouping by Quarter |
6M | Half-year | Grouping by Half-year |
COUNTRY | Country | Grouping by Country |
REGION | Operations Centre | Grouping by Operations Centre |
VO | VO | Grouping by VO |
SubmitHost | SubmitHost | Grouping by SubmitHost |
RESOURCE_CENTRE | SITE | Group by Resource Centre |
NUMBER PROCESSORS | Number of Processors | Group by the number of Logical Cores. |
NODECOUNT | Nodes | Group by Nodes |
Dates
Optional attribute
Format:
/StartYear/StartMonth/EndYear/EndMonth/
The date format is:
/YYYY/MM/YYYY/MM/
The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth.
VO Group Selector
Optional Attribute
Format:
<vo_group>
VO Group selects which VOs are selected for the current view, there are five settings:
Accepted Values for VO Group Selector | Meaning |
---|---|
lhc | The four VOs used on the WLCG Large Hadron Collider (ATLAS, ALICE, LHCb, CMS). |
top10 | Top 10 VOs that in the period and options selected have the highest unnormalised raw CPU value. |
all | All VOs are included. |
egi | VOs officially registered on the EGI Operations Portal. |
custom | Custom set of virtual organisations. |
In the case of a “custom” selection, the used format is
custom-vo1,vo2,vo3,..,von
This scales to infrastructure views with 200+ VOs.
Local Job Selector
Optional attribute Format:
<local_job_selector>
The Local Job selector parameter filters accounting based on its use of local vs infrastructure reosurce
Accepted Values for VO Group Selector | Meaning |
---|---|
onlyinfrajobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
localinfrajobs | All jobs executed by the EGI Federation, regardless of the submission mode. |
onlylocaljobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
Data Selector
Optional attribute
Accepted Values for Data Selector | Meaning |
---|---|
/JSON/ | Returns data as JSON (Java Script Object Notation) data. |
/CSV/ | Returns data as CSV (Comma Separated Values) data. |
Returns
In case of success: 200 OK HTTP Code and output.
In case of error: [HTTP Error Code]
Output Example
https://accounting-pre.egi.cesga.es/vo_admin/htc/cesga/njobs/REGION/DATE/2013/1/2017/6/egi/onlyinfrajobs/
[{"2013-10":16,"id":"NGI_IBERGRID","Total":16,"Percent":100}, {"2013-10":16,"id":"Total","Total":16,"Percent":""},
{"2013-10":"100.00%","id":"Percent","Percent":"","Total":""}, {"id":"xlegend","0":"NGI_IBERGRID"},
{"id":"ylegend","0":"2013-10","1":"id"}, {"id":"var","xrange":"DATE","yrange":"REGION","query":"njobs",
"user_dn":"\/DC=org\/DC=terena\/DC=tcs\/C=ES\/O=CESGA\/CN=Ivan Diaz Alvarez idiaz@cesga.es","granted_vos":["atlas","biomed","cesga","dteam","env.see-grid-
sci.eu","fedcloud.egi.eu","ific","israelvo.isragrid.org.il","isravo.isragrid.org.il","kzvo.isragrid.org.il","lhcb","pheno","prod.vo.eu-eela.eu"],"granted_sites":
["AEGIS04-KG","BG01-IPP","BG05-SUGrid","CESGA","EELA-UTFSM","IFIC-LCG2","INFN-ROMA1-CMS","JINR-LCG2","JINR-T1","NGI_IL","RO-11-NIPNE"]}]
Dates
Optional attribute
Format:
/StartYear/StartMonth/EndYear/EndMonth/
The date format is:
/YYYY/MM/YYYY/MM/
The date indicated by StartYear/StartMonth needs to be BEFORE EndYear/EndMonth.
VO Group Selector
Optional Attribute
Format:
<vo_group>
VO Group selects which VOs are selected for the current view, there are five settings:
Accepted Values for VO Group Selector | Meaning |
---|---|
lhc | The four VOs used on the WLCG Large Hadron Collider (ATLAS, ALICE, LHCb, CMS). |
top10 | Top 10 VOs that in the period and options selected have the highest unnormalised raw CPU value. |
all | All VOs are included. |
egi | VOs officially registered on the EGI Operations Portal. |
custom | Custom set of virtual organisations. |
In the case of a “custom” selection, the used format is
custom-vo1,vo2,vo3,..,von
This scales to infrastructure views with 200+ VOs.
Local Job Selector
Optional attribute Format:
<local_job_selector>
The Local Job selector parameter filters accounting based on its use of local vs infrastructure reosurce
Accepted Values for VO Group Selector | Meaning |
---|---|
onlyinfrajobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
localinfrajobs | All jobs executed by the EGI Federation, regardless of the submission mode. |
onlylocaljobs | Selects jobs that were successfully submitted and completed by a local cluster by submitting them directly to the the batch system of the computing cluster, i.e. bypassing the Compute Element interface. |
Data Selector
Optional attribute
Accepted Values for Data Selector | Meaning |
---|---|
/JSON/ | Returns data as JSON (Java Script Object Notation) data. |
/CSV/ | Returns data as CSV (Comma Separated Values) data. |
Returns
In case of success: 200 OK HTTP Code and output.
In case of error: [HTTP Error Code]
Output Example
https://accounting-pre.egi.cesga.es/user/htc/njobs/REGION/DATE/2013/1/2017/6/egi/onlyinfrajobs/
[{"2013-10":16,"id":"NGI_IBERGRID","Total":16,"Percent":100}, {"2013-10":16,"id":"Total","Total":16,"Percent":""},
{"2013-10":"100.00%","id":"Percent","Percent":"","Total":""}, {"id":"xlegend","0":"NGI_IBERGRID"},
{"id":"ylegend","0":"2013-10","1":"id"}, {"id":"var","xrange":"DATE","yrange":"REGION","query":"njobs",
"user_dn":"\/DC=org\/DC=terena\/DC=tcs\/C=ES\/O=CESGA\/CN=Ivan Diaz Alvarez idiaz@cesga.es","granted_vos":["atlas","biomed","cesga","dteam","env.see-grid-
sci.eu","fedcloud.egi.eu","ific","israelvo.isragrid.org.il","isravo.isragrid.org.il","kzvo.isragrid.org.il","lhcb","pheno","prod.vo.eu-eela.eu"],"granted_sites":
["AEGIS04-KG","BG01-IPP","BG05-SUGrid","CESGA","EELA-UTFSM","IFIC-LCG2","INFN-ROMA1-CMS","JINR-LCG2","JINR-T1","NGI_IL","RO-11-NIPNE"]}]