Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

GOCDB/Release4/Development/ExtensibilityMechanism

From EGIWiki
< GOCDB‎ | Release4‎ | Development
Revision as of 16:05, 13 February 2014 by Jamesm (talk | contribs) (→‎Sample PI XML Output)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

Key-Value Pair Property Bag Extensiblity Mechanism

Introduction

  • Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764
  • Sites, Services and ServiceGroups can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech).
  • Extend GUI and PI to filter sites/services by required key-value pair for applying restrictions to query results.
    • In the GUI filter sites/services via a key-value search.
    • Support the 'extensions' URL param on PI methods:
      • get_service_endpoint', 'get_site', 'get_site_list' 'get_service_group'
    • Support the 'site_extensions' and 'service_extensions' URL params on PI methods:
      • 'get_downtime' and 'get_downtime_nested_services'
  • There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars.
  • The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries).
  • The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements.
  • The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets.
    • (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT.
    • If no operator is specified before the first (k=v) pair, then AND is assumed.
    • A single operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered.
    • A sequence of multiple operators of the same type form a logical conjunction, while a different type of operator forms a logical disjunction with any previously specified restrictions.


Examples:

  • equivalent (note no leading AND):
    • AND(key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
    • (key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
  • equivalent:
    • (VO2=bing)AND(VO2=baz)AND(VO=bar)OR(s1p1=v1)
    • ((VO2=bing)AND(VO2=baz)AND(VO=bar))OR(s1p1=v1)
  • equivalent:
    • (VO=food)OR(VO2=bar)AND(s4p1=v1)
    • ((VO=food)OR(VO2=bar))AND(s4p1=v1)
  • equivalent:
    • (VO=food)(s4p1=v1)OR(VO2=bar)(VO2=baz)
    • ((VO=food)AND(s4p1=v1))OR(VO2=bar)OR(VO2=baz)
  • equivalent:
    • (VO=food)(s4p1=v1)OR(VO2=baz)AND(VO2=bling)
    • (((VO=food)AND(s4p1=v1))OR(VO2=baz))AND(VO2=bling)

Test Portal

PI Examples

get_site, get_site_list, get_service_endpoint, get_service_group

  • The get_site, get_site_list, get_service_endpoints and get_service_group PI methods will support the 'extensions' URL param. This can be used to filter the Sites and Services by the specified key-value pairs.


To return all sites that define VO with a value of Alice:

?method=get_site&extensions=(VO=Alice)

Use no value to define a wildcard search, i.e. all sites that define the VO property regardless of value:

?method=get_site&extensions=(VO=)

Extensions also supports OR/AND/NOT operators. This can be used to search against multiple key values eg:

?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)(VO=LHCB)

These can be used together:

?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)NOT(VO=LHCB)
?method= get_service_endpoint&extensions=OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

When no operator is specified the default is AND, therefore the following:

?method= get_service_endpoint&extensions=(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

Is the same as:

?method= get_service_endpoint&extensions=AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

The extensions parameter can also be used in conjunction with the existing parameters previously supported:

?method=get_site&extensions=(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK

get_downtime, get_downtime_nested_services

  • The 'site_extensions' and 'service_extensions' can also be used on the 'get_downtime' and 'get_downtime_nested_services' methods using same logic described above. Note, the <EXTENSIONS> element is not rendered in the XML output for these queries.
?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

PI Examples

These are currently in testing and can be viewed on the GocDB test server:

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=AND(VO=Alice)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)NOT(CPU_HS02_HOUR=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)OR(CPU_HS02_HOUR=5)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=AND(VO=Alice)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_group&scope=Local&extensions=(Test=)

Note:

  • The extent to which the ldap syntax query string would be supported will have to be explored. A simplified version including only exact pattern matches and simple wildcards may cover most use cases.
  • The property bags may only be applied to a subset of entities and PI queries.

Sample PI XML Output

get_site (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
<results>
	<SITE ID="259" PRIMARY_KEY="458G0" NAME="NGS-HECTOR">
		<PRIMARY_KEY>458G0</PRIMARY_KEY>
		<SHORT_NAME>NGS-HECTOR</SHORT_NAME>
		<OFFICIAL_NAME>HeCTOR, UK National Supercomputing Service
		</OFFICIAL_NAME>
		<SITE_DESCRIPTION>Pending</SITE_DESCRIPTION>
		<GOCDB_PORTAL_URL>
			https://127.0.0.1/DoctrineP/index.php?Page_Type=Siteampid=259
		</GOCDB_PORTAL_URL>
		<HOME_URL>www.hector.ac.uk</HOME_URL>
		<CONTACT_EMAIL>support@hector.ac.uk</CONTACT_EMAIL>
		<CONTACT_TEL>0131-650 5029</CONTACT_TEL>
		<COUNTRY_CODE>GB</COUNTRY_CODE>
		<COUNTRY>United Kingdom</COUNTRY>
		<ROC>NGI_UK</ROC>
		<SUBGRID>NGS</SUBGRID>
		<PRODUCTION_INFRASTRUCTURE>Production</PRODUCTION_INFRASTRUCTURE>
		<CERTIFICATION_STATUS>Uncertified</CERTIFICATION_STATUS>
		<TIMEZONE>UTC</TIMEZONE>
		<CSIRT_EMAIL>helpdesk@hector.ac.uk</CSIRT_EMAIL>
		<DOMAIN>
			<DOMAIN_NAME>hector.ac.uk</DOMAIN_NAME>
		</DOMAIN>
		<EXTENSIONS>
			<EXTENSION>
				<LOCAL_ID>24</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>LHCB</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>21</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>Alice</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>22</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>Atlas</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>23</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>CMS</VALUE>
			</EXTENSION>
		</EXTENSIONS>
	</SITE>
</results>

get_service_endpoint (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
<results>
    <SERVICE_ENDPOINT PRIMARY_KEY="310G0">
        <PRIMARY_KEY>310G0</PRIMARY_KEY>
        <HOSTNAME>lcgui01.gridpp.rl.ac.uk</HOSTNAME>
        <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Serviceampid=310</GOCDB_PORTAL_URL>
        <BETA>N</BETA>
        <SERVICE_TYPE>UI</SERVICE_TYPE>
        <HOST_IP>130.246.183.188</HOST_IP>
        <CORE/>
        <IN_PRODUCTION>N</IN_PRODUCTION>
        <NODE_MONITORED>N</NODE_MONITORED>
        <SITENAME>RAL-LCG2</SITENAME>
        <COUNTRY_NAME>United Kingdom</COUNTRY_NAME>
        <COUNTRY_CODE>GB</COUNTRY_CODE>
        <ROC_NAME>NGI_UK</ROC_NAME>
        <URL/>
        <EXTENSIONS>
            <EXTENSION>
                <LOCAL_ID>23</LOCAL_ID>
                <KEY>CPU_HS01_HOUR</KEY>
                <VALUE>3</VALUE>
            </EXTENSION>
        </EXTENSIONS>
    </SERVICE_ENDPOINT>
</results>

get_service_group (note new <EXTENSIONS>) <source lang="XML"> <?xml version="1.0" encoding="UTF-8"?>

   <SERVICE_GROUP PRIMARY_KEY="689G0">
       <NAME>DAVETESTSG</NAME>
       <DESCRIPTION>testing servcie group</DESCRIPTION>
       <MONITORED>N</MONITORED>
       <CONTACT_EMAIL>david.meredith@stfc.ac.uk</CONTACT_EMAIL>
       <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Service_Group amp id=689</GOCDB_PORTAL_URL>
       <EXTENSIONS>
           <EXTENSION>
               <LOCAL_ID>1</LOCAL_ID>
               <KEY>Test</KEY>
               <VALUE>Test</VALUE>
           </EXTENSION>
       </EXTENSIONS>
   </SERVICE_GROUP>

</results>

Potential Use Cases

A number of potential use cases have been reported (https://rt.egi.eu/rt/Ticket/Display.html?id=3764), including:

Issues

There is the potential for proliferation of key-value pairs that should instead be recorded in fields/properties as part of the data model.