Difference between revisions of "GOCDB/Release4/Development/v5"

From EGIWiki
Jump to: navigation, search
(v5 Dropping PROM for Doctrine)
(v5 Release Summary)
 
(36 intermediate revisions by the same user not shown)
Line 6: Line 6:
 
<< Back to [[GOCDB/Release4/Development]] <br />
 
<< Back to [[GOCDB/Release4/Development]] <br />
  
= v5 Dropping PROM for Doctrine =
+
= v5 Release Summary =
 
* By replacing the PROM model with the [http://www.doctrine-project.org Doctrine ORM] GOCDB will support RDBMSs other than Oracle.  
 
* By replacing the PROM model with the [http://www.doctrine-project.org Doctrine ORM] GOCDB will support RDBMSs other than Oracle.  
* v5 scheduled for released early Sept (date to be TBA - prob 2nd Sept).
+
* v5 scheduled for release <u>2nd Oct</u>.
* Important changes to the Portal URLs and to the PI are detailed below.
+
* The v5 PI is largely backward compatible with the v4 PI. However, there are <b>some small but important changes to the Portal URLs and to the PI output</b> as described on this page.
 +
* Note, <b>the PI URLs will stay the same.</b>
 +
* If these changes affect a system that reads from the GOCDB-PI or links to the web portal please [mailto:gocdb-admins@mailman.egi.eu let us know].
 
** GOCDB v5 Test instance: https://gocdb-test.esc.rl.ac.uk/v5  
 
** GOCDB v5 Test instance: https://gocdb-test.esc.rl.ac.uk/v5  
 
** GOCDB v5 PI test: <pre>https://gocdb-test.esc.rl.ac.uk/v5_pi/public/<method_name></pre>
 
** GOCDB v5 PI test: <pre>https://gocdb-test.esc.rl.ac.uk/v5_pi/public/<method_name></pre>
 +
** Motivation, further details, data model changes: https://indico.egi.eu/indico/materialDisplay.py?contribId=71&sessionId=24&materialId=slides&confId=1417
  
= Benefits =
+
= v5 Benefits of changing the DB model (dropping PROM for doctrine) =
 
* Write code once, deploy to Oracle, MySQL, Postgres, SQLite.
 
* Write code once, deploy to Oracle, MySQL, Postgres, SQLite.
 
* Much simpler than the [[GOCDB/PROM]] model (see query comparison below)]
 
* Much simpler than the [[GOCDB/PROM]] model (see query comparison below)]
Line 19: Line 22:
 
* Easier for new developers to learn
 
* Easier for new developers to learn
 
* Better performance
 
* Better performance
** Reduced risk of failover performance issues
+
* Doctrine is de-facto object-relational-mapping library (ORM) for PHP.
 +
** Reduced risk of fail over performance issues
  
 
{|
 
{|
Line 36: Line 40:
  
 
== Portal URLs ==
 
== Portal URLs ==
 +
* This affects the Web Portal GUI URLs (not the PI URLs).
 
In PROM, each "entity" (site, service endpoint, downtime) was assigned an ID from a global pool. There is only one object 123, the GRIDOPS-GOCDB site. The following URL is used to access an entity regardless of type:  
 
In PROM, each "entity" (site, service endpoint, downtime) was assigned an ID from a global pool. There is only one object 123, the GRIDOPS-GOCDB site. The following URL is used to access an entity regardless of type:  
  
Line 53: Line 58:
  
 
== PI XML Output Values ==
 
== PI XML Output Values ==
In the GOCDB v4 PI there are a number of PROM specific XML values that will change when we move to Doctrine: PRIMARY_KEY, ID and GOCDB_PORTAL_URL. These are highlighted in RED below:
+
In the GOCDB v4 PI there are a number of PROM specific XML values that will change when we move to Doctrine: PRIMARY_KEY, ID and GOCDB_PORTAL_URL. These are highlighted in <font color="red">RED</font> below:
  
 
[[File:Get_Site_Fields.png]]
 
[[File:Get_Site_Fields.png]]
Line 66: Line 71:
 
<br/>
 
<br/>
 
* v4 <PRIMARY_KEY> element <u>values</u> are carried over for <b>Sites</b> and <b>Downtimes</b> to maintain backward compatiblity (get_site, get_site_list, get_downtime)
 
* v4 <PRIMARY_KEY> element <u>values</u> are carried over for <b>Sites</b> and <b>Downtimes</b> to maintain backward compatiblity (get_site, get_site_list, get_downtime)
* No preservation of v4 <PRIMARY_KEY> element values are planned for:
+
* No preservation of v4 <PRIMARY_KEY> element values for:
** get_service_endpoint (no reports that v4 SE PRIMARY_KEY values are required for carry over)
+
** get_service_endpoint  
 
** get_ngi
 
** get_ngi
 
** get_service_group
 
** get_service_group
** <b>Please check!</b>
+
** <b>Please check!</b> that this does not affect your tool.
  
 
== PI get_downtime method paging ==
 
== PI get_downtime method paging ==
* The ‘get_downtime’ PI method requires a new mandatory ‘&page’ parameter to limit the number of returned downtimes to a sensible limit (500 in each page).  
+
* The ‘get_downtime’ PI method supports a new ‘&page’ parameter to limit the number of returned downtimes to a manageable limit (1000 in each page).  
* Required to reduce server load and max query execution time (consider ~10,000 XML downtimes rendered in one page).  
+
* The <u><b>page parameter is optional</b></u>. If not specified, a query <u><b>returns all matching results</b></u>. 
* The 'get_downtime' method should be re-issued by the client and each time the page parameter should be incremented by one until an empty <results/> element is returned indicating no more results, e.g.  
+
* <b><u>Paging should be used with care</u></b> – since pending downtimes can be deleted, it is not guaranteed that all downtimes will be returned in a uninterrupted sequence if downtimes are actively deleted in the time it takes to issue a sequence of paged-queries. 
 +
* If you invoke ‘get_downtime’ with no query parameters to narrow your search, the query can potentially time-out/fail because too many downtimes are requested in one hit. In this scenario, consider narrowing your query with additional query parameters before using paging.  
 +
* The 'get_downtime' method can be re-issued by the client and each time the page parameter should be incremented by one until an empty <results/> element is returned indicating no more results, e.g.  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=1  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=1  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=2  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=2  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=3
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=3
* Paging is not required when issuing the 'all_lastmonth' param as below:
 
  
 
== PI get_downtime method 'all_lastmonth'==
 
== PI get_downtime method 'all_lastmonth'==
Line 85: Line 91:
 
* Results are not paged - *all* DTs returned for this window. ‘&page’ parameter can be excluded if specifying ‘all_lastmonth’ e.g.  
 
* Results are not paged - *all* DTs returned for this window. ‘&page’ parameter can be excluded if specifying ‘all_lastmonth’ e.g.  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&all_lastmonth  
 
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&all_lastmonth  
* WARNING:  The number of results should normally fall within the page limit, but since paging is turned off, this query could potentially be expensive/problematic if there are many more DTs than the page limit. Without paging, this query can theoretically fail!
+
<!--* WARNING:  The number of results should normally fall within the page limit, but since paging is turned off, this query could potentially be expensive/problematic if there are many more DTs than the page limit. Without paging, this query can theoretically fail!-->
 +
 
 +
== PI Scope Extensions ==
 +
* Scoping is being extended in v5 so that resources from multiple projects/infrastructures can be hosted in a single gocdb instance - see [[GOCDB/Release4/Development/conditionalCertificationStatusRules]]
 +
* <b>Important: the scope extensions are backward compatible and will not break clients' existing PI queries</b>.
 +
* The optional PI ‘&scope’ parameter will allow a comma-sep list of values rather than a single ‘EGI’ or ‘Local’ value.
 +
* An empty value can be used to ignore scope filtering: ‘&scope=’
 +
* If ‘&scope’ is not specified, ‘EGI’ is matched by default to <b>preserve v4 behaviour</b>.
 +
* New ‘&scope_match’ parameter for methods supporting &scope:
 +
** ‘&scope_match=any’ matches entities that define any of the specified scopes
 +
** ‘&scope_match=all’ matches entities that define all of the specified scopes
 +
 
 +
Examples:
 +
*Return sites that define either the EGI or PROJX scope tags: 
 +
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=any
 +
*Return sites that define both the EGI and PROJX scope tags:
 +
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=all
 +
*Return sites that define the default configured scope tag (‘EGI’) in our case (scope tag not specified):
 +
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site
 +
*Return sites irrespective of their scope tags (empty ‘scope=’):
 +
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=
 +
*Return all DTs for Services that define both PROJX+EGI scope (scope references the Service scope):
 +
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&scope=PROJX,EGI&scope_match=all
  
 
= Request for Comments =
 
= Request for Comments =
 
If these changes affect a system that reads from the GOCDB-PI or links to the web portal please [mailto:gocdb-admins@mailman.egi.eu let us know].
 
If these changes affect a system that reads from the GOCDB-PI or links to the web portal please [mailto:gocdb-admins@mailman.egi.eu let us know].

Latest revision as of 18:15, 30 September 2013

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

v5 Release Summary

v5 Benefits of changing the DB model (dropping PROM for doctrine)

  • Write code once, deploy to Oracle, MySQL, Postgres, SQLite.
  • Much simpler than the GOCDB/PROM model (see query comparison below)]
  • Easier for other projects to extend / modify
  • Easier for new developers to learn
  • Better performance
  • Doctrine is de-facto object-relational-mapping library (ORM) for PHP.
    • Reduced risk of fail over performance issues
PROM SQL
Doctrine SQL

Drawbacks

  • Data will be deleted rather than marked as 'off'. We therefore require an audit table to record who/when performed data deletions/updates (this is on the requirements roadmap ).
  • Loose some flexiblity as the DB will manage the relationships (FKs) between entities rather than the application (but in turn we gain performance/simplicity/db-agnostic).
  • Changes to some PI methods (paging and carrying over v4 PKs)
  • Changes to Portal URLs and object IDs (below)

Changes

The move to Doctrine will introduce some important changes. These are outlined below.

Portal URLs

  • This affects the Web Portal GUI URLs (not the PI URLs).

In PROM, each "entity" (site, service endpoint, downtime) was assigned an ID from a global pool. There is only one object 123, the GRIDOPS-GOCDB site. The following URL is used to access an entity regardless of type:

https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=123&grid_id=0

In v5 using Doctrine, each entity receives an ID unique only to that entity type. There is a site with ID 123, a service endpoint with ID 123 and a downtime with ID 123. The new URL will reflect the entity type:

https://goc.egi.eu/portal/index.php?Page_Type=Site&id=123

https://goc.egi.eu/portal/index.php?Page_Type=Service_Endpoint&id=123

https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=123

(...and so on for each GOCDB entity type)

GOCDB URLs.png

PI XML Output Values

In the GOCDB v4 PI there are a number of PROM specific XML values that will change when we move to Doctrine: PRIMARY_KEY, ID and GOCDB_PORTAL_URL. These are highlighted in RED below:

Get Site Fields.png

  • ID fields will NOT be consistent between v4 and v5 (each new object will get a new Doctrine ID value).
  • The GOCDB_PORTAL_URL will therefore NOT be consistent between v4 and v5.
  • The v4 PRIMARY_KEY value will be carried over only for Sites and Downtimes, see below:



Update Aug 2013 - Preserving v4 <PRIMARY_KEY> element values

  • v4 <PRIMARY_KEY> element values are carried over for Sites and Downtimes to maintain backward compatiblity (get_site, get_site_list, get_downtime)
  • No preservation of v4 <PRIMARY_KEY> element values for:
    • get_service_endpoint
    • get_ngi
    • get_service_group
    • Please check! that this does not affect your tool.

PI get_downtime method paging

  • The ‘get_downtime’ PI method supports a new ‘&page’ parameter to limit the number of returned downtimes to a manageable limit (1000 in each page).
  • The page parameter is optional. If not specified, a query returns all matching results.
  • Paging should be used with care – since pending downtimes can be deleted, it is not guaranteed that all downtimes will be returned in a uninterrupted sequence if downtimes are actively deleted in the time it takes to issue a sequence of paged-queries.
  • If you invoke ‘get_downtime’ with no query parameters to narrow your search, the query can potentially time-out/fail because too many downtimes are requested in one hit. In this scenario, consider narrowing your query with additional query parameters before using paging.
  • The 'get_downtime' method can be re-issued by the client and each time the page parameter should be incremented by one until an empty <results/> element is returned indicating no more results, e.g.

PI get_downtime method 'all_lastmonth'

PI Scope Extensions

  • Scoping is being extended in v5 so that resources from multiple projects/infrastructures can be hosted in a single gocdb instance - see GOCDB/Release4/Development/conditionalCertificationStatusRules
  • Important: the scope extensions are backward compatible and will not break clients' existing PI queries.
  • The optional PI ‘&scope’ parameter will allow a comma-sep list of values rather than a single ‘EGI’ or ‘Local’ value.
  • An empty value can be used to ignore scope filtering: ‘&scope=’
  • If ‘&scope’ is not specified, ‘EGI’ is matched by default to preserve v4 behaviour.
  • New ‘&scope_match’ parameter for methods supporting &scope:
    • ‘&scope_match=any’ matches entities that define any of the specified scopes
    • ‘&scope_match=all’ matches entities that define all of the specified scopes

Examples:

  • Return sites that define either the EGI or PROJX scope tags: 

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=any

  • Return sites that define both the EGI and PROJX scope tags:

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=all

  • Return sites that define the default configured scope tag (‘EGI’) in our case (scope tag not specified):

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site

  • Return sites irrespective of their scope tags (empty ‘scope=’):

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=

  • Return all DTs for Services that define both PROJX+EGI scope (scope references the Service scope):

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&scope=PROJX,EGI&scope_match=all

Request for Comments

If these changes affect a system that reads from the GOCDB-PI or links to the web portal please let us know.