Difference between revisions of "GOCDB/Release4/Development/ExtensibilityMechanism"
(14 intermediate revisions by 2 users not shown) | |||
Line 8: | Line 8: | ||
==Introduction== | ==Introduction== | ||
* Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764 | * Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764 | ||
* Sites and | * Sites, Services and ServiceGroups can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech). | ||
* Extend GUI and PI to filter sites/services by required key | * Extend GUI and PI to filter sites/services by required key-value pair for applying restrictions to query results. | ||
** In the GUI filter sites/services via a key-value search. | ** In the GUI filter sites/services via a key-value search. | ||
** Support the 'extensions' URL param on PI methods | ** Support the 'extensions' URL param on PI methods: | ||
** Support the 'site_extensions' and 'service_extensions' URL params on PI methods 'get_downtime' and 'get_downtime_nested_services' | *** get_service_endpoint', 'get_site', 'get_site_list' 'get_service_group' | ||
** Support the 'site_extensions' and 'service_extensions' URL params on PI methods: | |||
*** 'get_downtime' and 'get_downtime_nested_services' | |||
* There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars. | * There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars. | ||
* The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries). | * The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries). | ||
* The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements. | * The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements. | ||
* The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets. (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT. If no operator is specified before the first (k=v) pair, then AND is assumed. | * The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets. | ||
< | ** (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT. | ||
<font color="red"> | ** If no operator is specified before the first (k=v) pair, then AND is assumed. | ||
< | ** A single operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered. | ||
** A sequence of multiple operators of the same type form a logical conjunction, while a different type of operator forms a logical disjunction with any previously specified restrictions. | |||
< | |||
<font color="red">( | |||
Examples: | |||
* equivalent (note no leading AND): | |||
** <font color="red">AND(key1=val)(key2=va2)</font><font color="green">OR(key3=val3)(key4=val4)</font><font color="blue">NOT(key5=val5)(key6=val6)</font> | |||
** <font color="red">(key1=val)(key2=va2)</font><font color="green">OR(key3=val3)(key4=val4)</font><font color="blue">NOT(key5=val5)(key6=val6)</font> | |||
* equivalent: | |||
** (VO2=bing)AND(VO2=baz)AND(VO=bar)OR(s1p1=v1) | |||
** <font color="red">(</font>(VO2=bing)AND(VO2=baz)AND(VO=bar)<font color="red">)</font>OR(s1p1=v1) | |||
* equivalent: | |||
** (VO=food)OR(VO2=bar)AND(s4p1=v1) | |||
** <font color="red">(</font>(VO=food)OR(VO2=bar)<font color="red">)</font>AND(s4p1=v1) | |||
* equivalent: | |||
** (VO=food)(s4p1=v1)OR(VO2=bar)(VO2=baz) | |||
** <font color="red">(</font>(VO=food)AND(s4p1=v1)<font color="red">)</font>OR(VO2=bar)OR(VO2=baz) | |||
* equivalent: | |||
** (VO=food)(s4p1=v1)OR(VO2=baz)AND(VO2=bling) | |||
** <font color="red">(</font><font color="blue">(</font>(VO=food)AND(s4p1=v1)<font color="blue">)</font>OR(VO2=baz)<font color="red">)</font>AND(VO2=bling) | |||
==Test Portal== | ==Test Portal== | ||
Line 56: | Line 76: | ||
The extensions parameter can also be used in conjunction with the existing parameters previously supported: | The extensions parameter can also be used in conjunction with the existing parameters previously supported: | ||
<pre>?method=get_site&extensions= | <pre>?method=get_site&extensions=(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK</pre> | ||
===get_downtime, get_downtime_nested_services=== | ===get_downtime, get_downtime_nested_services=== | ||
Line 76: | Line 96: | ||
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=) | https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=) | ||
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions= | https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)NOT(CPU_HS02_HOUR=) | ||
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5) | https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5) | ||
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions= | https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)OR(CPU_HS02_HOUR=5) | ||
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2 | https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2 | ||
Line 200: | Line 220: | ||
</SERVICE_GROUP> | </SERVICE_GROUP> | ||
</results> | </results> | ||
==Potential Use Cases== | ==Potential Use Cases== |
Latest revision as of 17:05, 13 February 2014
Main | EGI.eu operations services | Support | Documentation | Tools | Activities | Performance | Technology | Catch-all Services | Resource Allocation | Security |
GOC DB menu: | Home • | Documentation Index • |
<< Back to GOCDB/Release4/Development
Key-Value Pair Property Bag Extensiblity Mechanism
Introduction
- Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764
- Sites, Services and ServiceGroups can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech).
- Extend GUI and PI to filter sites/services by required key-value pair for applying restrictions to query results.
- In the GUI filter sites/services via a key-value search.
- Support the 'extensions' URL param on PI methods:
- get_service_endpoint', 'get_site', 'get_site_list' 'get_service_group'
- Support the 'site_extensions' and 'service_extensions' URL params on PI methods:
- 'get_downtime' and 'get_downtime_nested_services'
- There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars.
- The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries).
- The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements.
- The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets.
- (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT.
- If no operator is specified before the first (k=v) pair, then AND is assumed.
- A single operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered.
- A sequence of multiple operators of the same type form a logical conjunction, while a different type of operator forms a logical disjunction with any previously specified restrictions.
Examples:
- equivalent (note no leading AND):
- AND(key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
- (key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
- equivalent:
- (VO2=bing)AND(VO2=baz)AND(VO=bar)OR(s1p1=v1)
- ((VO2=bing)AND(VO2=baz)AND(VO=bar))OR(s1p1=v1)
- equivalent:
- (VO=food)OR(VO2=bar)AND(s4p1=v1)
- ((VO=food)OR(VO2=bar))AND(s4p1=v1)
- equivalent:
- (VO=food)(s4p1=v1)OR(VO2=bar)(VO2=baz)
- ((VO=food)AND(s4p1=v1))OR(VO2=bar)OR(VO2=baz)
- equivalent:
- (VO=food)(s4p1=v1)OR(VO2=baz)AND(VO2=bling)
- (((VO=food)AND(s4p1=v1))OR(VO2=baz))AND(VO2=bling)
Test Portal
- https://gocdb-test.esc.rl.ac.uk/v5
- https://gocdb-test.esc.rl.ac.uk/v5/index.php?Page_Type=Sites
- https://gocdb-test.esc.rl.ac.uk/v5/index.php?Page_Type=Services
PI Examples
get_site, get_site_list, get_service_endpoint, get_service_group
- The get_site, get_site_list, get_service_endpoints and get_service_group PI methods will support the 'extensions' URL param. This can be used to filter the Sites and Services by the specified key-value pairs.
To return all sites that define VO with a value of Alice:
?method=get_site&extensions=(VO=Alice)
Use no value to define a wildcard search, i.e. all sites that define the VO property regardless of value:
?method=get_site&extensions=(VO=)
Extensions also supports OR/AND/NOT operators. This can be used to search against multiple key values eg:
?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)(VO=LHCB)
These can be used together:
?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)NOT(VO=LHCB)
?method= get_service_endpoint&extensions=OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)
When no operator is specified the default is AND, therefore the following:
?method= get_service_endpoint&extensions=(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)
Is the same as:
?method= get_service_endpoint&extensions=AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)
The extensions parameter can also be used in conjunction with the existing parameters previously supported:
?method=get_site&extensions=(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK
get_downtime, get_downtime_nested_services
- The 'site_extensions' and 'service_extensions' can also be used on the 'get_downtime' and 'get_downtime_nested_services' methods using same logic described above. Note, the <EXTENSIONS> element is not rendered in the XML output for these queries.
?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
PI Examples
These are currently in testing and can be viewed on the GocDB test server:
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=AND(VO=Alice)(VO=CMS)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=AND(VO=Alice)(VO=CMS)
Note:
- The extent to which the ldap syntax query string would be supported will have to be explored. A simplified version including only exact pattern matches and simple wildcards may cover most use cases.
- The property bags may only be applied to a subset of entities and PI queries.
Sample PI XML Output
get_site (note new <EXTENSIONS>)
<?xml version="1.0" encoding="UTF-8"?>
<results>
<SITE ID="259" PRIMARY_KEY="458G0" NAME="NGS-HECTOR">
<PRIMARY_KEY>458G0</PRIMARY_KEY>
<SHORT_NAME>NGS-HECTOR</SHORT_NAME>
<OFFICIAL_NAME>HeCTOR, UK National Supercomputing Service
</OFFICIAL_NAME>
<SITE_DESCRIPTION>Pending</SITE_DESCRIPTION>
<GOCDB_PORTAL_URL>
https://127.0.0.1/DoctrineP/index.php?Page_Type=Siteampid=259
</GOCDB_PORTAL_URL>
<HOME_URL>www.hector.ac.uk</HOME_URL>
<CONTACT_EMAIL>support@hector.ac.uk</CONTACT_EMAIL>
<CONTACT_TEL>0131-650 5029</CONTACT_TEL>
<COUNTRY_CODE>GB</COUNTRY_CODE>
<COUNTRY>United Kingdom</COUNTRY>
<ROC>NGI_UK</ROC>
<SUBGRID>NGS</SUBGRID>
<PRODUCTION_INFRASTRUCTURE>Production</PRODUCTION_INFRASTRUCTURE>
<CERTIFICATION_STATUS>Uncertified</CERTIFICATION_STATUS>
<TIMEZONE>UTC</TIMEZONE>
<CSIRT_EMAIL>helpdesk@hector.ac.uk</CSIRT_EMAIL>
<DOMAIN>
<DOMAIN_NAME>hector.ac.uk</DOMAIN_NAME>
</DOMAIN>
<EXTENSIONS>
<EXTENSION>
<LOCAL_ID>24</LOCAL_ID>
<KEY>VO</KEY>
<VALUE>LHCB</VALUE>
</EXTENSION>
<EXTENSION>
<LOCAL_ID>21</LOCAL_ID>
<KEY>VO</KEY>
<VALUE>Alice</VALUE>
</EXTENSION>
<EXTENSION>
<LOCAL_ID>22</LOCAL_ID>
<KEY>VO</KEY>
<VALUE>Atlas</VALUE>
</EXTENSION>
<EXTENSION>
<LOCAL_ID>23</LOCAL_ID>
<KEY>VO</KEY>
<VALUE>CMS</VALUE>
</EXTENSION>
</EXTENSIONS>
</SITE>
</results>
get_service_endpoint (note new <EXTENSIONS>)
<?xml version="1.0" encoding="UTF-8"?>
<results>
<SERVICE_ENDPOINT PRIMARY_KEY="310G0">
<PRIMARY_KEY>310G0</PRIMARY_KEY>
<HOSTNAME>lcgui01.gridpp.rl.ac.uk</HOSTNAME>
<GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Serviceampid=310</GOCDB_PORTAL_URL>
<BETA>N</BETA>
<SERVICE_TYPE>UI</SERVICE_TYPE>
<HOST_IP>130.246.183.188</HOST_IP>
<CORE/>
<IN_PRODUCTION>N</IN_PRODUCTION>
<NODE_MONITORED>N</NODE_MONITORED>
<SITENAME>RAL-LCG2</SITENAME>
<COUNTRY_NAME>United Kingdom</COUNTRY_NAME>
<COUNTRY_CODE>GB</COUNTRY_CODE>
<ROC_NAME>NGI_UK</ROC_NAME>
<URL/>
<EXTENSIONS>
<EXTENSION>
<LOCAL_ID>23</LOCAL_ID>
<KEY>CPU_HS01_HOUR</KEY>
<VALUE>3</VALUE>
</EXTENSION>
</EXTENSIONS>
</SERVICE_ENDPOINT>
</results>
get_service_group (note new <EXTENSIONS>) <source lang="XML"> <?xml version="1.0" encoding="UTF-8"?>
<SERVICE_GROUP PRIMARY_KEY="689G0"> <NAME>DAVETESTSG</NAME> <DESCRIPTION>testing servcie group</DESCRIPTION> <MONITORED>N</MONITORED> <CONTACT_EMAIL>david.meredith@stfc.ac.uk</CONTACT_EMAIL> <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Service_Group amp id=689</GOCDB_PORTAL_URL> <EXTENSIONS> <EXTENSION> <LOCAL_ID>1</LOCAL_ID> <KEY>Test</KEY> <VALUE>Test</VALUE> </EXTENSION> </EXTENSIONS> </SERVICE_GROUP>
</results>
Potential Use Cases
A number of potential use cases have been reported (https://rt.egi.eu/rt/Ticket/Display.html?id=3764), including:
- Specifying service access protocol.
- Defining custom monitoring parameters.
- Defining custom functionality/capability flags supported by a service (this may be related to MPI in EGI requirements, see: https://documents.egi.eu/public/RetrieveFile?docid=1260&version=3&filename=VT-MPI_Report_v3_final.pdf and https://rt.egi.eu/rt/Ticket/Display.html?id=3396).
Issues
There is the potential for proliferation of key-value pairs that should instead be recorded in fields/properties as part of the data model.