Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Development/v5"

From EGIWiki
Jump to navigation Jump to search
 
(32 intermediate revisions by the same user not shown)
Line 6: Line 6:
<< Back to [[GOCDB/Release4/Development]] <br />
<< Back to [[GOCDB/Release4/Development]] <br />


= v5 Dropping PROM for Doctrine =
= v5 Release Summary =
* By replacing the PROM model with the [http://www.doctrine-project.org Doctrine ORM] GOCDB will support RDBMSs other than Oracle.  
* By replacing the PROM model with the [http://www.doctrine-project.org Doctrine ORM] GOCDB will support RDBMSs other than Oracle.  
* v5 scheduled for released early Sept (date to be TBA - prob 2nd Sept).
* v5 scheduled for release <u>2nd Oct</u>.
* Important changes to the Portal URLs and to the PI are detailed below.
* The v5 PI is largely backward compatible with the v4 PI. However, there are <b>some small but important changes to the Portal URLs and to the PI output</b> as described on this page.
* Note, <b>the PI URLs will stay the same.</b>
* If these changes affect a system that reads from the GOCDB-PI or links to the web portal please [mailto:gocdb-admins@mailman.egi.eu let us know].
** GOCDB v5 Test instance: https://gocdb-test.esc.rl.ac.uk/v5  
** GOCDB v5 Test instance: https://gocdb-test.esc.rl.ac.uk/v5  
** GOCDB v5 PI test: <pre>https://gocdb-test.esc.rl.ac.uk/v5_pi/public/<method_name></pre>
** GOCDB v5 PI test: <pre>https://gocdb-test.esc.rl.ac.uk/v5_pi/public/<method_name></pre>
** Motivation, further details, data model changes: https://indico.egi.eu/indico/materialDisplay.py?contribId=71&sessionId=24&materialId=slides&confId=1417


= Benefits =
= v5 Benefits of changing the DB model (dropping PROM for doctrine) =
* Write code once, deploy to Oracle, MySQL, Postgres, SQLite.
* Write code once, deploy to Oracle, MySQL, Postgres, SQLite.
* Much simpler than the [[GOCDB/PROM]] model (see query comparison below)]
* Much simpler than the [[GOCDB/PROM]] model (see query comparison below)]
Line 19: Line 22:
* Easier for new developers to learn
* Easier for new developers to learn
* Better performance
* Better performance
** Reduced risk of failover performance issues
* Doctrine is de-facto object-relational-mapping library (ORM) for PHP.
** Reduced risk of fail over performance issues


{|
{|
Line 36: Line 40:


== Portal URLs ==
== Portal URLs ==
* This affects the Web Portal GUI URLs (not the PI URLs).
In PROM, each "entity" (site, service endpoint, downtime) was assigned an ID from a global pool. There is only one object 123, the GRIDOPS-GOCDB site. The following URL is used to access an entity regardless of type:  
In PROM, each "entity" (site, service endpoint, downtime) was assigned an ID from a global pool. There is only one object 123, the GRIDOPS-GOCDB site. The following URL is used to access an entity regardless of type:  


Line 51: Line 56:


[[File:GOCDB_URLs.png|900px]]
[[File:GOCDB_URLs.png|900px]]


== PI XML Output Values ==
== PI XML Output Values ==
In the GOCDB v4 PI there are a number of PROM specific XML values that will change when we move to Doctrine: PRIMARY_KEY, ID and GOCDB_PORTAL_URL. These are highlighted in RED below:
In the GOCDB v4 PI there are a number of PROM specific XML values that will change when we move to Doctrine: PRIMARY_KEY, ID and GOCDB_PORTAL_URL. These are highlighted in <font color="red">RED</font> below:


[[File:Get_Site_Fields.png]]
[[File:Get_Site_Fields.png]]
Line 67: Line 71:
<br/>
<br/>
* v4 <PRIMARY_KEY> element <u>values</u> are carried over for <b>Sites</b> and <b>Downtimes</b> to maintain backward compatiblity (get_site, get_site_list, get_downtime)
* v4 <PRIMARY_KEY> element <u>values</u> are carried over for <b>Sites</b> and <b>Downtimes</b> to maintain backward compatiblity (get_site, get_site_list, get_downtime)
* No preservation of v4 <PRIMARY_KEY> element values are planned for:
* No preservation of v4 <PRIMARY_KEY> element values for:
** get_service_endpoint (no reports that v4 SE PRIMARY_KEY values are required for carry over)
** get_service_endpoint  
** get_ngi
** get_ngi
** get_service_group
** get_service_group
** <b>Please check!</b>
** <b>Please check!</b> that this does not affect your tool.
 


== PI get_downtime method paging ==
== PI get_downtime method paging ==
* The ‘get_downtime’ PI method requires a new mandatory ‘&page’ parameter to limit the number of returned downtimes to a sensible limit (500 in each page).  
* The ‘get_downtime’ PI method supports a new ‘&page’ parameter to limit the number of returned downtimes to a manageable limit (1000 in each page).  
* Required to reduce server load and max query execution time (consider ~10,000 XML downtimes rendered in one page).  
* The <u><b>page parameter is optional</b></u>. If not specified, a query <u><b>returns all matching results</b></u>. 
* The 'get_downtime' method should be re-issued by the client and each time the page parameter should be incremented by one until an empty <results/> element is returned indicating no more results, e.g.  
* <b><u>Paging should be used with care</u></b> – since pending downtimes can be deleted, it is not guaranteed that all downtimes will be returned in a uninterrupted sequence if downtimes are actively deleted in the time it takes to issue a sequence of paged-queries. 
* If you invoke ‘get_downtime’ with no query parameters to narrow your search, the query can potentially time-out/fail because too many downtimes are requested in one hit. In this scenario, consider narrowing your query with additional query parameters before using paging.  
* The 'get_downtime' method can be re-issued by the client and each time the page parameter should be incremented by one until an empty <results/> element is returned indicating no more results, e.g.  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=1  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=1  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=2  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=2  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=3
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&page=3
* Paging is not required when issuing the 'all_lastmonth' param as below:


== PI get_downtime method 'all_lastmonth'==
== PI get_downtime method 'all_lastmonth'==
Line 88: Line 91:
* Results are not paged - *all* DTs returned for this window. ‘&page’ parameter can be excluded if specifying ‘all_lastmonth’ e.g.  
* Results are not paged - *all* DTs returned for this window. ‘&page’ parameter can be excluded if specifying ‘all_lastmonth’ e.g.  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&all_lastmonth  
** https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&all_lastmonth  
* WARNING:  The number of results should normally fall within the page limit, but since paging is turned off, this query could potentially be expensive/problematic if there are many more DTs than the page limit. Without paging, this query can theoretically fail!
<!--* WARNING:  The number of results should normally fall within the page limit, but since paging is turned off, this query could potentially be expensive/problematic if there are many more DTs than the page limit. Without paging, this query can theoretically fail!-->
 


== PI Scope Extensions ==
== PI Scope Extensions ==
* Scoping is being extended in v5 so that resources from multiple projects/infrastructures can be hosted in a single gocdb instance - see [[GOCDB/Release4/Development/conditionalCertificationStatusRules]]
* Scoping is being extended in v5 so that resources from multiple projects/infrastructures can be hosted in a single gocdb instance - see [[GOCDB/Release4/Development/conditionalCertificationStatusRules]]
* <b>Important: the scope extensions are backward compatible and will not break clients existing PI queries</b>.  
* <b>Important: the scope extensions are backward compatible and will not break clients' existing PI queries</b>.  
* The optional PI ‘&scope’ parameter will allow a comma-sep list of values rather than a single ‘EGI’ or ‘Local’ value.  
* The optional PI ‘&scope’ parameter will allow a comma-sep list of values rather than a single ‘EGI’ or ‘Local’ value.  
* An empty value can be used to ignore scope filtering: ‘&scope=’  
* An empty value can be used to ignore scope filtering: ‘&scope=’  
Line 101: Line 103:
** ‘&scope_match=all’ matches entities that define all of the specified scopes
** ‘&scope_match=all’ matches entities that define all of the specified scopes


* Examples:  
Examples:  
** Return sites that define either the EGI or PROJX scope tags:   
*Return sites that define either the EGI or PROJX scope tags:   
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=any  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=any  
** Return sites that define both the EGI and PROJX scope tags:  
*Return sites that define both the EGI and PROJX scope tags:  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=all  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=all  
** Return sites that define the default configured scope tag (‘EGI’) in our case (scope tag not specified):  
*Return sites that define the default configured scope tag (‘EGI’) in our case (scope tag not specified):  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site  
** Return sites irrespective of their scope tags (empty ‘scope=’):  
*Return sites irrespective of their scope tags (empty ‘scope=’):  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=  
** Return all DTs for Services that define both PROJX+EGI scope (scope references the Service scope):  
*Return all DTs for Services that define both PROJX+EGI scope (scope references the Service scope):  
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime &scope=PROJX,EGI&scope_match=all
https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&scope=PROJX,EGI&scope_match=all


= Request for Comments =
= Request for Comments =
If these changes affect a system that reads from the GOCDB-PI or links to the web portal please [mailto:gocdb-admins@mailman.egi.eu let us know].
If these changes affect a system that reads from the GOCDB-PI or links to the web portal please [mailto:gocdb-admins@mailman.egi.eu let us know].

Latest revision as of 17:15, 30 September 2013

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

v5 Release Summary

v5 Benefits of changing the DB model (dropping PROM for doctrine)

  • Write code once, deploy to Oracle, MySQL, Postgres, SQLite.
  • Much simpler than the GOCDB/PROM model (see query comparison below)]
  • Easier for other projects to extend / modify
  • Easier for new developers to learn
  • Better performance
  • Doctrine is de-facto object-relational-mapping library (ORM) for PHP.
    • Reduced risk of fail over performance issues
PROM SQL
Doctrine SQL

Drawbacks

  • Data will be deleted rather than marked as 'off'. We therefore require an audit table to record who/when performed data deletions/updates (this is on the requirements roadmap ).
  • Loose some flexiblity as the DB will manage the relationships (FKs) between entities rather than the application (but in turn we gain performance/simplicity/db-agnostic).
  • Changes to some PI methods (paging and carrying over v4 PKs)
  • Changes to Portal URLs and object IDs (below)

Changes

The move to Doctrine will introduce some important changes. These are outlined below.

Portal URLs

  • This affects the Web Portal GUI URLs (not the PI URLs).

In PROM, each "entity" (site, service endpoint, downtime) was assigned an ID from a global pool. There is only one object 123, the GRIDOPS-GOCDB site. The following URL is used to access an entity regardless of type:

https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=123&grid_id=0

In v5 using Doctrine, each entity receives an ID unique only to that entity type. There is a site with ID 123, a service endpoint with ID 123 and a downtime with ID 123. The new URL will reflect the entity type:

https://goc.egi.eu/portal/index.php?Page_Type=Site&id=123

https://goc.egi.eu/portal/index.php?Page_Type=Service_Endpoint&id=123

https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=123

(...and so on for each GOCDB entity type)

GOCDB URLs.png

PI XML Output Values

In the GOCDB v4 PI there are a number of PROM specific XML values that will change when we move to Doctrine: PRIMARY_KEY, ID and GOCDB_PORTAL_URL. These are highlighted in RED below:

Get Site Fields.png

  • ID fields will NOT be consistent between v4 and v5 (each new object will get a new Doctrine ID value).
  • The GOCDB_PORTAL_URL will therefore NOT be consistent between v4 and v5.
  • The v4 PRIMARY_KEY value will be carried over only for Sites and Downtimes, see below:



Update Aug 2013 - Preserving v4 <PRIMARY_KEY> element values

  • v4 <PRIMARY_KEY> element values are carried over for Sites and Downtimes to maintain backward compatiblity (get_site, get_site_list, get_downtime)
  • No preservation of v4 <PRIMARY_KEY> element values for:
    • get_service_endpoint
    • get_ngi
    • get_service_group
    • Please check! that this does not affect your tool.

PI get_downtime method paging

  • The ‘get_downtime’ PI method supports a new ‘&page’ parameter to limit the number of returned downtimes to a manageable limit (1000 in each page).
  • The page parameter is optional. If not specified, a query returns all matching results.
  • Paging should be used with care – since pending downtimes can be deleted, it is not guaranteed that all downtimes will be returned in a uninterrupted sequence if downtimes are actively deleted in the time it takes to issue a sequence of paged-queries.
  • If you invoke ‘get_downtime’ with no query parameters to narrow your search, the query can potentially time-out/fail because too many downtimes are requested in one hit. In this scenario, consider narrowing your query with additional query parameters before using paging.
  • The 'get_downtime' method can be re-issued by the client and each time the page parameter should be incremented by one until an empty <results/> element is returned indicating no more results, e.g.

PI get_downtime method 'all_lastmonth'

PI Scope Extensions

  • Scoping is being extended in v5 so that resources from multiple projects/infrastructures can be hosted in a single gocdb instance - see GOCDB/Release4/Development/conditionalCertificationStatusRules
  • Important: the scope extensions are backward compatible and will not break clients' existing PI queries.
  • The optional PI ‘&scope’ parameter will allow a comma-sep list of values rather than a single ‘EGI’ or ‘Local’ value.
  • An empty value can be used to ignore scope filtering: ‘&scope=’
  • If ‘&scope’ is not specified, ‘EGI’ is matched by default to preserve v4 behaviour.
  • New ‘&scope_match’ parameter for methods supporting &scope:
    • ‘&scope_match=any’ matches entities that define any of the specified scopes
    • ‘&scope_match=all’ matches entities that define all of the specified scopes

Examples:

  • Return sites that define either the EGI or PROJX scope tags: 

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=any

  • Return sites that define both the EGI and PROJX scope tags:

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=EGI,PROJX&scope_match=all

  • Return sites that define the default configured scope tag (‘EGI’) in our case (scope tag not specified):

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site

  • Return sites irrespective of their scope tags (empty ‘scope=’):

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_site&scope=

  • Return all DTs for Services that define both PROJX+EGI scope (scope references the Service scope):

https://gocdb-test.esc.rl.ac.uk/v5_pi/public/?method=get_downtime&scope=PROJX,EGI&scope_match=all

Request for Comments

If these changes affect a system that reads from the GOCDB-PI or links to the web portal please let us know.