Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

GOCDB/Release4/Development/ExtensibilityMechanism

From EGIWiki
< GOCDB‎ | Release4‎ | Development
Revision as of 16:35, 11 February 2014 by Davidm (talk | contribs)
Jump to navigation Jump to search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

Key-Value Pair Property Bag Extensiblity Mechanism

Introduction

  • Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764
  • Sites, Services and ServiceGroups can be extended so that they can define an optional set of custom key-value pairs (akin to GLUE2 ext mech).
  • Extend GUI and PI to filter sites/services by required key/value pair.
    • In the GUI filter sites/services via a key-value search.
    • Support the 'extensions' URL param on PI methods:
      • get_service_endpoint', 'get_site', 'get_site_list' 'get_service_group'
    • Support the 'site_extensions' and 'service_extensions' URL params on PI methods:
      • 'get_downtime' and 'get_downtime_nested_services'
  • There are a number of restrictions that could be relaxed if required, e.g. number of k=v pairs, and illegal chars.
  • The extensions parameters only support basic query syntax (e.g. no nesting of sub-queries).
  • The results of the get_site and get_service_endpoint PI queries nest new <EXTENSIONS> elements.
  • The format of the 'extensions' expression is one or more (key=value) pairs enclosed in brackets used to apply a restriction to the query results.
    • (K=v) pairs can be optionally prefixed with one of following operators: AND, OR, NOT.
    • If no operator is specified before the first (k=v) pair, then AND is assumed.
    • A single operator applies to ALL (k=v) pairs to the right of the operator until another operator is encountered.
    • Multiple operators of the same type form a logical conjunction with any previously specified restrictions, while a different type of operator forms a logical disjunction with any previously specified restrictions.

Examples:
AND(key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)
is equivalent to (note no leading AND):
(key1=val)(key2=va2)OR(key3=val3)(key4=val4)NOT(key5=val5)(key6=val6)


(VO2=bing)AND(VO2=baz)AND(VO=bar)OR(s1p1=v1) is equivalent to: ((VO2=bing)AND(VO2=baz)AND(VO=bar))OR(s1p1=v1)

(VO=food)OR(VO2=bar)AND(s4p1=v1) is equivalent to: ((VO=food)OR(VO2=bar))AND(s4p1=v1

(VO=food)(s4p1=v1)OR(VO2=bar)(VO2=baz) is equivalent to: ((VO=food)AND(s4p1=v1))OR(VO2=bar)OR(VO2=baz

Test Portal

PI Examples

get_site, get_site_list, get_service_endpoint, get_service_group

  • The get_site, get_site_list, get_service_endpoints and get_service_group PI methods will support the 'extensions' URL param. This can be used to filter the Sites and Services by the specified key-value pairs.


To return all sites that define VO with a value of Alice:

?method=get_site&extensions=(VO=Alice)

Use no value to define a wildcard search, i.e. all sites that define the VO property regardless of value:

?method=get_site&extensions=(VO=)

Extensions also supports OR/AND/NOT operators. This can be used to search against multiple key values eg:

?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)(VO=LHCB)

These can be used together:

?method=get_site&extensions=AND(VO=Alice)(VO=Atlas)NOT(VO=LHCB)
?method= get_service_endpoint&extensions=OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

When no operator is specified the default is AND, therefore the following:

?method= get_service_endpoint&extensions=(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

Is the same as:

?method= get_service_endpoint&extensions=AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2)

The extensions parameter can also be used in conjunction with the existing parameters previously supported:

?method=get_site&extensions=OR(VO=Alice)NOT(VO=LHCB)&scope=EGI&roc=NGI_UK

get_downtime, get_downtime_nested_services

  • The 'site_extensions' and 'service_extensions' can also be used on the 'get_downtime' and 'get_downtime_nested_services' methods using same logic described above. Note, the <EXTENSIONS> element is not rendered in the XML output for these queries.
?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)
?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

PI Examples

These are currently in testing and can be viewed on the GocDB test server:

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=AND(VO=Alice)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)NOT(CPU_HS02_HOUR=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=AND(CPU_HS01_HOUR=)(CPU_HS02_HOUR=5)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)OR(CPU_HS02_HOUR=5)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=)&sitename=RAL-LCG2

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=AND(VO=Alice)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site_list&extensions=(VO=LHCB)NOT(VO=Alice)(VO=Atlas)(VO=CMS)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime_nested_services&site_extensions=(eg.2=val.2)&service_extensions=(eg.2=)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_group&scope=Local&extensions=(Test=)

Note:

  • The extent to which the ldap syntax query string would be supported will have to be explored. A simplified version including only exact pattern matches and simple wildcards may cover most use cases.
  • The property bags may only be applied to a subset of entities and PI queries.

Sample PI XML Output

get_site (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
<results>
	<SITE ID="259" PRIMARY_KEY="458G0" NAME="NGS-HECTOR">
		<PRIMARY_KEY>458G0</PRIMARY_KEY>
		<SHORT_NAME>NGS-HECTOR</SHORT_NAME>
		<OFFICIAL_NAME>HeCTOR, UK National Supercomputing Service
		</OFFICIAL_NAME>
		<SITE_DESCRIPTION>Pending</SITE_DESCRIPTION>
		<GOCDB_PORTAL_URL>
			https://127.0.0.1/DoctrineP/index.php?Page_Type=Siteampid=259
		</GOCDB_PORTAL_URL>
		<HOME_URL>www.hector.ac.uk</HOME_URL>
		<CONTACT_EMAIL>support@hector.ac.uk</CONTACT_EMAIL>
		<CONTACT_TEL>0131-650 5029</CONTACT_TEL>
		<COUNTRY_CODE>GB</COUNTRY_CODE>
		<COUNTRY>United Kingdom</COUNTRY>
		<ROC>NGI_UK</ROC>
		<SUBGRID>NGS</SUBGRID>
		<PRODUCTION_INFRASTRUCTURE>Production</PRODUCTION_INFRASTRUCTURE>
		<CERTIFICATION_STATUS>Uncertified</CERTIFICATION_STATUS>
		<TIMEZONE>UTC</TIMEZONE>
		<CSIRT_EMAIL>helpdesk@hector.ac.uk</CSIRT_EMAIL>
		<DOMAIN>
			<DOMAIN_NAME>hector.ac.uk</DOMAIN_NAME>
		</DOMAIN>
		<EXTENSIONS>
			<EXTENSION>
				<LOCAL_ID>24</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>LHCB</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>21</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>Alice</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>22</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>Atlas</VALUE>
			</EXTENSION>
			<EXTENSION>
				<LOCAL_ID>23</LOCAL_ID>
				<KEY>VO</KEY>
				<VALUE>CMS</VALUE>
			</EXTENSION>
		</EXTENSIONS>
	</SITE>
</results>

get_service_endpoint (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
<results>
    <SERVICE_ENDPOINT PRIMARY_KEY="310G0">
        <PRIMARY_KEY>310G0</PRIMARY_KEY>
        <HOSTNAME>lcgui01.gridpp.rl.ac.uk</HOSTNAME>
        <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Serviceampid=310</GOCDB_PORTAL_URL>
        <BETA>N</BETA>
        <SERVICE_TYPE>UI</SERVICE_TYPE>
        <HOST_IP>130.246.183.188</HOST_IP>
        <CORE/>
        <IN_PRODUCTION>N</IN_PRODUCTION>
        <NODE_MONITORED>N</NODE_MONITORED>
        <SITENAME>RAL-LCG2</SITENAME>
        <COUNTRY_NAME>United Kingdom</COUNTRY_NAME>
        <COUNTRY_CODE>GB</COUNTRY_CODE>
        <ROC_NAME>NGI_UK</ROC_NAME>
        <URL/>
        <EXTENSIONS>
            <EXTENSION>
                <LOCAL_ID>23</LOCAL_ID>
                <KEY>CPU_HS01_HOUR</KEY>
                <VALUE>3</VALUE>
            </EXTENSION>
        </EXTENSIONS>
    </SERVICE_ENDPOINT>
</results>

get_service_group (note new <EXTENSIONS>)

<?xml version="1.0" encoding="UTF-8"?>
    <SERVICE_GROUP PRIMARY_KEY="689G0">
        <NAME>DAVETESTSG</NAME>
        <DESCRIPTION>testing servcie group</DESCRIPTION>
        <MONITORED>N</MONITORED>
        <CONTACT_EMAIL>david.meredith@stfc.ac.uk</CONTACT_EMAIL>
        <GOCDB_PORTAL_URL>https://localhost/portal/index.php?Page_Type=Service_Group amp id=689</GOCDB_PORTAL_URL>
        <EXTENSIONS>
            <EXTENSION>
                <LOCAL_ID>1</LOCAL_ID>
                <KEY>Test</KEY>
                <VALUE>Test</VALUE>
            </EXTENSION>
        </EXTENSIONS>
    </SERVICE_GROUP>
</results>

Potential Use Cases

A number of potential use cases have been reported (https://rt.egi.eu/rt/Ticket/Display.html?id=3764), including:

Issues

There is the potential for proliferation of key-value pairs that should instead be recorded in fields/properties as part of the data model.