Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Development/ExtensibilityMechanism"

From EGIWiki
Jump to navigation Jump to search
 
(26 intermediate revisions by 2 users not shown)
Line 8: Line 8:
==Introduction==
==Introduction==
* Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764  
* Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764  
* Sites and Services can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech).
* Sites, Services and ServiceGroups can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech).
* Extend GUI and PI to filter sites/services by required key/value pair.  
* Extend GUI and PI to filter sites/services by required key-value pair for applying restrictions to query results.  
** In the GUI filter sites/services via a key-value search.  
** In the GUI filter sites/services via a key-value search.  
** Support the 'extensions' URL param on PI methods 'get_service_endpoint', 'get_site' and 'get_site_list'  
** Support the 'extensions' URL param on PI methods:
** Support the 'site_extensions' and 'service_extensions' URL params on PI methods'get_downtime' and 'get_downtime_nested_services'
*** get_service_endpoint', 'get_site', 'get_site_list' 'get_service_group'  
* This is not yet in production and needs a thorough period of testing first.
** Support the 'site_extensions' and 'service_extensions' URL params on PI methods:
* Bugs are likely at this stage.  
*** 'get_downtime' and 'get_downtime_nested_services'
* There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars.  
* The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries).  
* The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries).  
* The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements.
* The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements.
* The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets. (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT. If no operator is specified before the first (k=v) pair, then AND is assumed. The operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered, for example:   
* The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets.
<br/>
** (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT.  
<font color="red">AND(key1=val)(key2=va2)</font><font color="green">OR(key3=val3)(key4=val4)</font><font color="blue">NOT(key5=val5)(key6=val6)</font>
** If no operator is specified before the first (k=v) pair, then AND is assumed.  
<br/>
** A single operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered.
is equivalent to: (note no leading AND which is assumed)
** A sequence of multiple operators of the same type form a logical conjunction, while a different type of operator forms a logical disjunction with any previously specified restrictions. 
<br/>
 
<font color="red">(key1=val)(key2=va2)</font><font color="green">OR(key3=val3)(key4=val4)</font><font color="blue">NOT(key5=val5)(key6=val6)</font>
 
Examples:   
* equivalent (note no leading AND):
** <font color="red">AND(key1=val)(key2=va2)</font><font color="green">OR(key3=val3)(key4=val4)</font><font color="blue">NOT(key5=val5)(key6=val6)</font>
** <font color="red">(key1=val)(key2=va2)</font><font color="green">OR(key3=val3)(key4=val4)</font><font color="blue">NOT(key5=val5)(key6=val6)</font>
 
* equivalent:
** (VO2=bing)AND(VO2=baz)AND(VO=bar)OR(s1p1=v1)
** <font color="red">(</font>(VO2=bing)AND(VO2=baz)AND(VO=bar)<font color="red">)</font>OR(s1p1=v1)
 
* equivalent:
** (VO=food)OR(VO2=bar)AND(s4p1=v1)
** <font color="red">(</font>(VO=food)OR(VO2=bar)<font color="red">)</font>AND(s4p1=v1)
 
* equivalent:
** (VO=food)(s4p1=v1)OR(VO2=bar)(VO2=baz)
** <font color="red">(</font>(VO=food)AND(s4p1=v1)<font color="red">)</font>OR(VO2=bar)OR(VO2=baz)
 
* equivalent:
** (VO=food)(s4p1=v1)OR(VO2=baz)AND(VO2=bling)
** <font color="red">(</font><font color="blue">(</font>(VO=food)AND(s4p1=v1)<font color="blue">)</font>OR(VO2=baz)<font color="red">)</font>AND(VO2=bling)
 
==Test Portal==
* https://gocdb-test.esc.rl.ac.uk/v5
* https://gocdb-test.esc.rl.ac.uk/v5/index.php?Page_Type=Sites
* https://gocdb-test.esc.rl.ac.uk/v5/index.php?Page_Type=Services


==PI Examples==
==PI Examples==
* The get_sites, get_site_list and get_service_endpoints PI methods will support the 'extensions' URL param. This can be used to filter the Sites and Services by the specified key-value pairs. To return all sites that define VO with a value of Alice:
===get_site, get_site_list, get_service_endpoint, get_service_group===
* The get_site, get_site_list, get_service_endpoints and get_service_group PI methods will support the 'extensions' URL param. This can be used to filter the Sites and Services by the specified key-value pairs.  
<br/>
To return all sites that define VO with a value of Alice:


<pre>?method=get_site&extensions=(VO=Alice)</pre>
<pre>?method=get_site&extensions=(VO=Alice)</pre>
Line 47: Line 76:
The extensions parameter can also be used in conjunction with the existing parameters previously supported:
The extensions parameter can also be used in conjunction with the existing parameters previously supported:


<pre>?method=get_site&extensions=OR(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK</pre>
<pre>?method=get_site&extensions=(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK</pre>
 
===get_downtime, get_downtime_nested_services===
* The 'site_extensions' and 'service_extensions' can also be used on the 'get_downtime' and 'get_downtime_nested_services' methods using same logic described above. Note, the <EXTENSIONS> element is not rendered in the XML output for these queries.
 
<pre>?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)</pre>
<pre>?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)</pre>


===PI Examples===


These are currently in testing and can be viewed on the GocDB test server:
These are currently in testing and can be viewed on the GocDB test server:
Line 60: Line 96:
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)


https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=OR(CPU_HS01_HOUR=)NOT(CPU_HS02_HOUR=)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)NOT(CPU_HS02_HOUR=)


https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5)


https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=OR(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)OR(CPU_HS02_HOUR=5)


https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2
Line 71: Line 107:


https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_group&scope=Local&extensions=(Test=)


Note:
Note:
Line 77: Line 119:


==Sample PI XML Output==
==Sample PI XML Output==
get_site (note new <EXTENSIONS>)
<source lang="XML">
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
Line 128: Line 171:
</results>
</results>
</source>
</source>
get_service_endpoint (note new <EXTENSIONS>)
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
<results>
    <SERVICE_ENDPOINT PRIMARY_KEY="310G0">
        <PRIMARY_KEY>310G0</PRIMARY_KEY>
        <HOSTNAME>lcgui01.gridpp.rl.ac.uk</HOSTNAME>
        <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Serviceampid=310</GOCDB_PORTAL_URL>
        <BETA>N</BETA>
        <SERVICE_TYPE>UI</SERVICE_TYPE>
        <HOST_IP>130.246.183.188</HOST_IP>
        <CORE/>
        <IN_PRODUCTION>N</IN_PRODUCTION>
        <NODE_MONITORED>N</NODE_MONITORED>
        <SITENAME>RAL-LCG2</SITENAME>
        <COUNTRY_NAME>United Kingdom</COUNTRY_NAME>
        <COUNTRY_CODE>GB</COUNTRY_CODE>
        <ROC_NAME>NGI_UK</ROC_NAME>
        <URL/>
        <EXTENSIONS>
            <EXTENSION>
                <LOCAL_ID>23</LOCAL_ID>
                <KEY>CPU_HS01_HOUR</KEY>
                <VALUE>3</VALUE>
            </EXTENSION>
        </EXTENSIONS>
    </SERVICE_ENDPOINT>
</results>
</source>
get_service_group (note new <EXTENSIONS>)
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
    <SERVICE_GROUP PRIMARY_KEY="689G0">
        <NAME>DAVETESTSG</NAME>
        <DESCRIPTION>testing servcie group</DESCRIPTION>
        <MONITORED>N</MONITORED>
        <CONTACT_EMAIL>david.meredith@stfc.ac.uk</CONTACT_EMAIL>
        <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Service_Group amp id=689</GOCDB_PORTAL_URL>
        <EXTENSIONS>
            <EXTENSION>
                <LOCAL_ID>1</LOCAL_ID>
                <KEY>Test</KEY>
                <VALUE>Test</VALUE>
            </EXTENSION>
        </EXTENSIONS>
    </SERVICE_GROUP>
</results>


==Potential Use Cases==
==Potential Use Cases==

Latest revision as of 16:05, 13 February 2014

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

Key-Value Pair Property Bag Extensiblity Mechanism

Introduction

  • Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764
  • Sites, Services and ServiceGroups can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech).
  • Extend GUI and PI to filter sites/services by required key-value pair for applying restrictions to query results.
    • In the GUI filter sites/services via a key-value search.
    • Support the 'extensions' URL param on PI methods:
      • get_service_endpoint', 'get_site', 'get_site_list' 'get_service_group'
    • Support the 'site_extensions' and 'service_extensions' URL params on PI methods:
      • 'get_downtime' and 'get_downtime_nested_services'
  • There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars.
  • The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries).
  • The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements.
  • The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets.
    • (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT.
    • If no operator is specified before the first (k=v) pair, then AND is assumed.
    • A single operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered.
    • A sequence of multiple operators of the same type form a logical conjunction, while a different type of operator forms a logical disjunction with any previously specified restrictions.


Examples:

  • equivalent (note no leading AND):
    • AND(key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
    • (key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
  • equivalent:
    • (VO2=bing)AND(VO2=baz)AND(VO=bar)OR(s1p1=v1)
    • ((VO2=bing)AND(VO2=baz)AND(VO=bar))OR(s1p1=v1)
  • equivalent:
    • (VO=food)OR(VO2=bar)AND(s4p1=v1)
    • ((VO=food)OR(VO2=bar))AND(s4p1=v1)
  • equivalent:
    • (VO=food)(s4p1=v1)OR(VO2=bar)(VO2=baz)
    • ((VO=food)AND(s4p1=v1))OR(VO2=bar)OR(VO2=baz)
  • equivalent:
    • (VO=food)(s4p1=v1)OR(VO2=baz)AND(VO2=bling)
    • (((VO=food)AND(s4p1=v1))OR(VO2=baz))AND(VO2=bling)

Test Portal

PI Examples

get_site, get_site_list, get_service_endpoint, get_service_group

  • The get_site, get_site_list, get_service_endpoints and get_service_group PI methods will support the 'extensions' URL param. This can be used to filter the Sites and Services by the specified key-value pairs.


To return all sites that define VO with a value of Alice:

?method=get_site&extensions=(VO=Alice)

Use no value to define a wildcard search, i.e. all sites that define the VO property regardless of value:

?method=get_site&extensions=(VO=)

Extensions also supports OR/AND/NOT operators. This can be used to search against multiple key values eg:

?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)(VO=LHCB)

These can be used together:

?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)NOT(VO=LHCB)
?method= get_service_endpoint&extensions=OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

When no operator is specified the default is AND, therefore the following:

?method= get_service_endpoint&extensions=(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

Is the same as:

?method= get_service_endpoint&extensions=AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

The extensions parameter can also be used in conjunction with the existing parameters previously supported:

?method=get_site&extensions=(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK

get_downtime, get_downtime_nested_services

  • The 'site_extensions' and 'service_extensions' can also be used on the 'get_downtime' and 'get_downtime_nested_services' methods using same logic described above. Note, the <EXTENSIONS> element is not rendered in the XML output for these queries.
?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

PI Examples

These are currently in testing and can be viewed on the GocDB test server:

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=AND(VO=Alice)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)NOT(CPU_HS02_HOUR=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)OR(CPU_HS02_HOUR=5)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=AND(VO=Alice)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_group&scope=Local&extensions=(Test=)

Note:

  • The extent to which the ldap syntax query string would be supported will have to be explored. A simplified version including only exact pattern matches and simple wildcards may cover most use cases.
  • The property bags may only be applied to a subset of entities and PI queries.

Sample PI XML Output

get_site (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
<results>
	<SITE ID="259" PRIMARY_KEY="458G0" NAME="NGS-HECTOR">
		<PRIMARY_KEY>458G0</PRIMARY_KEY>
		<SHORT_NAME>NGS-HECTOR</SHORT_NAME>
		<OFFICIAL_NAME>HeCTOR, UK National Supercomputing Service
		</OFFICIAL_NAME>
		<SITE_DESCRIPTION>Pending</SITE_DESCRIPTION>
		<GOCDB_PORTAL_URL>
			https://127.0.0.1/DoctrineP/index.php?Page_Type=Siteampid=259
		</GOCDB_PORTAL_URL>
		<HOME_URL>www.hector.ac.uk</HOME_URL>
		<CONTACT_EMAIL>support@hector.ac.uk</CONTACT_EMAIL>
		<CONTACT_TEL>0131-650 5029</CONTACT_TEL>
		<COUNTRY_CODE>GB</COUNTRY_CODE>
		<COUNTRY>United Kingdom</COUNTRY>
		<ROC>NGI_UK</ROC>
		<SUBGRID>NGS</SUBGRID>
		<PRODUCTION_INFRASTRUCTURE>Production</PRODUCTION_INFRASTRUCTURE>
		<CERTIFICATION_STATUS>Uncertified</CERTIFICATION_STATUS>
		<TIMEZONE>UTC</TIMEZONE>
		<CSIRT_EMAIL>helpdesk@hector.ac.uk</CSIRT_EMAIL>
		<DOMAIN>
			<DOMAIN_NAME>hector.ac.uk</DOMAIN_NAME>
		</DOMAIN>
		<EXTENSIONS>
			<EXTENSION>
				<LOCAL_ID>24</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>LHCB</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>21</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>Alice</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>22</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>Atlas</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>23</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>CMS</VALUE>
			</EXTENSION>
		</EXTENSIONS>
	</SITE>
</results>

get_service_endpoint (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
<results>
    <SERVICE_ENDPOINT PRIMARY_KEY="310G0">
        <PRIMARY_KEY>310G0</PRIMARY_KEY>
        <HOSTNAME>lcgui01.gridpp.rl.ac.uk</HOSTNAME>
        <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Serviceampid=310</GOCDB_PORTAL_URL>
        <BETA>N</BETA>
        <SERVICE_TYPE>UI</SERVICE_TYPE>
        <HOST_IP>130.246.183.188</HOST_IP>
        <CORE/>
        <IN_PRODUCTION>N</IN_PRODUCTION>
        <NODE_MONITORED>N</NODE_MONITORED>
        <SITENAME>RAL-LCG2</SITENAME>
        <COUNTRY_NAME>United Kingdom</COUNTRY_NAME>
        <COUNTRY_CODE>GB</COUNTRY_CODE>
        <ROC_NAME>NGI_UK</ROC_NAME>
        <URL/>
        <EXTENSIONS>
            <EXTENSION>
                <LOCAL_ID>23</LOCAL_ID>
                <KEY>CPU_HS01_HOUR</KEY>
                <VALUE>3</VALUE>
            </EXTENSION>
        </EXTENSIONS>
    </SERVICE_ENDPOINT>
</results>

get_service_group (note new <EXTENSIONS>) <source lang="XML"> <?xml version="1.0" encoding="UTF-8"?>

   <SERVICE_GROUP PRIMARY_KEY="689G0">
       <NAME>DAVETESTSG</NAME>
       <DESCRIPTION>testing servcie group</DESCRIPTION>
       <MONITORED>N</MONITORED>
       <CONTACT_EMAIL>david.meredith@stfc.ac.uk</CONTACT_EMAIL>
       <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Service_Group amp id=689</GOCDB_PORTAL_URL>
       <EXTENSIONS>
           <EXTENSION>
               <LOCAL_ID>1</LOCAL_ID>
               <KEY>Test</KEY>
               <VALUE>Test</VALUE>
           </EXTENSION>
       </EXTENSIONS>
   </SERVICE_GROUP>

</results>

Potential Use Cases

A number of potential use cases have been reported (https://rt.egi.eu/rt/Ticket/Display.html?id=3764), including:

Issues

There is the potential for proliferation of key-value pairs that should instead be recorded in fields/properties as part of the data model.