Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Development/ExtensibilityMechanism"

From EGIWiki
Jump to navigation Jump to search
Line 16: Line 16:
This extensions parameter allows you to supply an LDAP styled query to further filter your site and endpoint results. For example:
This extensions parameter allows you to supply an LDAP styled query to further filter your site and endpoint results. For example:


?method=get_site&extensions=(VO=Alice)
<pre>?method=get_site&extensions=(VO=Alice)</pre>


Will return all sites that define VO with a value of Alice. A wildcard could also be used:
Will return all sites that define VO with a value of Alice. A wildcard could also be used:


?method=get_site&extensions=(VO=*)
<pre>?method=get_site&extensions=(VO=*)</pre>


Will return all sites that define VO as a key property with any value. Comma separated lists of values are also supported:
Will return all sites that define VO as a key property with any value. Comma separated lists of values are also supported:


?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=1,2,3,4,5)
<pre>?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=1,2,3,4,5)</pre>


This will return any service endpoints that define a CPU_HS01_HOUR with a value of 1, 2, 3, 4, or 5. Extensions also supports OR/AND operators. This can be used to search against multiple key values eg:
This will return any service endpoints that define a CPU_HS01_HOUR with a value of 1, 2, 3, 4, or 5. Extensions also supports OR/AND operators. This can be used to search against multiple key values eg:


?method= get_service_endpoint&extensions=(AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2))
<pre>?method= get_service_endpoint&extensions=(AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2))</pre>


This will return endpoints that define both CPU_HS01_HOUR with a value or 1 and CPU_HS02_HOUR with a value of 2. The same query can also be performed with OR:
This will return endpoints that define both CPU_HS01_HOUR with a value or 1 and CPU_HS02_HOUR with a value of 2. The same query can also be performed with OR:


?method= get_service_endpoint&extensions=(OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2))
<pre>?method= get_service_endpoint&extensions=(OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2))</pre>


This can then be used to find sites or service endpoints that all define the same group of key values:
This can then be used to find sites or service endpoints that all define the same group of key values:


?method=get_site&extensions=(AND(VO=Alice,Atlas,CMS,LHCB))
<pre>?method=get_site&extensions=(AND(VO=Alice,Atlas,CMS,LHCB))</pre>


The extensions parameter can also be used in conjunction with the existing parameters previously supported:
The extensions parameter can also be used in conjunction with the existing parameters previously supported:


?method=get_site&extensions=(OR(VO=Alice,Atlas,CMS,LHCB))&scope=EGI&roc=NGI_UK
<pre>?method=get_site&extensions=(OR(VO=Alice,Atlas,CMS,LHCB))&scope=EGI&roc=NGI_UK</pre>


These are currently in testing and can be viewed on the GocDB test server:
These are currently in testing and can be viewed on the GocDB test server:

Revision as of 18:00, 29 November 2013

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

Key-Value Pair Property Bag Extensiblity Mechanism

Introduction

  • Original RT request: https://rt.egi.eu/rt/Ticket/Display.html?id=3764
  • Simiar to the GLUE2 extensibility mechanim, the core GOCDB entities (NGIs, Sites, Services, ServiceGroups) will be extended so that they can define an optional set of custom key-value pairs.
  • In doing this, additional parameters could be supplied to the PI for selecting entities that define a particular set of properties. For example "select all Sites/Services that define the 'MPI=true' property."
  • As an extensibility mechanism, we do not intend to impose any restrictions on the values of key-value pairs (other than max value length, e.g. 255 chars ). Key-value pair meaninings should be determined as a community folksonomy.

PI Examples

The URL query could be parameterised with a known GET request paramter (e.g. 'extensions') with a LDAP syntax query string for the value, eg: This extensions parameter allows you to supply an LDAP styled query to further filter your site and endpoint results. For example:

?method=get_site&extensions=(VO=Alice)

Will return all sites that define VO with a value of Alice. A wildcard could also be used:

?method=get_site&extensions=(VO=*)

Will return all sites that define VO as a key property with any value. Comma separated lists of values are also supported:

?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=1,2,3,4,5)

This will return any service endpoints that define a CPU_HS01_HOUR with a value of 1, 2, 3, 4, or 5. Extensions also supports OR/AND operators. This can be used to search against multiple key values eg:

?method= get_service_endpoint&extensions=(AND(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2))

This will return endpoints that define both CPU_HS01_HOUR with a value or 1 and CPU_HS02_HOUR with a value of 2. The same query can also be performed with OR:

?method= get_service_endpoint&extensions=(OR(CPU_HS01_HOUR=1)(CPU_HS02_HOUR=2))

This can then be used to find sites or service endpoints that all define the same group of key values:

?method=get_site&extensions=(AND(VO=Alice,Atlas,CMS,LHCB))

The extensions parameter can also be used in conjunction with the existing parameters previously supported:

?method=get_site&extensions=(OR(VO=Alice,Atlas,CMS,LHCB))&scope=EGI&roc=NGI_UK

These are currently in testing and can be viewed on the GocDB test server:

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(VO=*)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(AND(VO=Alice,LHCB))

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&extensions=(OR(VO=CMS,LHCB,Atlas,Alice))

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=*)

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(OR(CPU_HS01_HOUR=*)(CPU_HS02_HOUR=5))

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(AND(CPU_HS01_HOUR=*)(CPU_HS02_HOUR=5))

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_service_endpoint&extensions=(CPU_HS01_HOUR=*)&sitename=RAL-LCG2


Note:

  • The extent to which the ldap syntax query string would be supported will have to be explored. A simplified version including only exact pattern matches and simple wildcards may cover most use cases.
  • The property bags may only be applied to a subset of entities and PI queries.

Sample PI XML Output

<?xml version="1.0" encoding="UTF-8"?>
<results>
   <SERVICE_ENDPOINT PRIMARY_KEY="50257G0">
      <PRIMARY_KEY>123456G0</PRIMARY_KEY>
      <HOSTNAME>some.host.ac.uk</HOSTNAME>
      <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/index.php?Page_Type=View_Object&amp;object_id=123456&amp;grid_id=0</GOCDB_PORTAL_URL>
      <HOST_OS>SL5</HOST_OS>
      <BETA>N</BETA>
      <SERVICE_TYPE>SomeComputeService</SERVICE_TYPE>
      <CORE></CORE>
      <IN_PRODUCTION>Y</IN_PRODUCTION>
      <NODE_MONITORED>Y</NODE_MONITORED>
      <SITENAME>DGIREF</SITENAME>
      <COUNTRY_NAME>Germany</COUNTRY_NAME>
      <COUNTRY_CODE>DE</COUNTRY_CODE>
      <ROC_NAME>NGI_DE</ROC_NAME>
      <ENDPOINT>
         <URL>ldap://ce-cms.vinca.rs:2170/mds-vo-name=AEGIS10-VINCA-CMS,o=grid</URL>
         <InterfaceName>RIS</InterfaceName>
      </ENDPOINT>   
      <!-- Key-Value Pair Property Bag attached to SERVICE_ENDPOINT entity -->
      <Extensions>
            <Extension>
                <LocalID>1234</LocalID>
                <Key>MPI</Key>
                <Value>True</Value> 
            </Extension>
            <Extension>
                <LocalID>12345</LocalID>
                <Key>somekey</Key>
                <Value>valueX</Value> 
            </Extension>
      </Extensions>
   </SERVICE_ENDPOINT>

</results>

Potential Use Cases

A number of potential use cases have been reported (https://rt.egi.eu/rt/Ticket/Display.html?id=3764), including:

Issues

There is the potential for proliferation of key-value pairs that should instead be recorded in fields/properties as part of the data model.