Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Regionalisation"

From EGIWiki
Jump to navigation Jump to search
 
(26 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Template:Op menubar}}
{{Template:Op menubar}}
{{Template:Tools menubar}}
{{Template:GOCDB_menubar}}
Back to [[GOCDB/Documentation_Index]] <br/>
{{TOC_right}}
[[Category:GOCDB]]
<< Back to [[GOCDB/Release4/Development|GOCDB4 Development Plans]] - Current GOCDB4 developments  <br/>
__TOC__
__TOC__
== GOCDB Regionalisation - latest status ==
== GOCDB Regionalisation - latest status ==


The last released version of GOCDB regional module dates from October 2010 (gocdb4.0d7). It is a fully functional tool, and all the basic components are included so that initial deployment can be tested using that version. Links:
*GOCDB is available at: https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.0d7_distrib/RPM/
*SRC is in the SVN at: http://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb/
*Documentation on the wiki [[GOCDB/Regional_Module_Technical_Documentation | GOCDB Regional Module Technical Documentation]].


'''Important note: GOCDB regional modules can be operated in full production but can't synchronise to the central GOCDB (yet).'''
 
'''Important note: The Regional-Publishing GOCDB has been dropped since the EGI Tech Forum 2012 OTAG'''


For all other GOCDB development plans/improvements see [[GOCDB/Release4/Development]]
For all other GOCDB development plans/improvements see [[GOCDB/Release4/Development]]
Line 16: Line 15:
= Regionalisation Plans =
= Regionalisation Plans =
== Introduction ==
== Introduction ==
Steps 1) to 3) below are required for a [[#3) Regional-Publishing GOCDB]] that can publish its 'EGI.EU' scoped data to the Central GOCDB.
Production of a Regional-Publishing GOCDB has been dropped. This page is for archive only.
Steps 1) to 3) below are required for a <strike>[[#3) Regional-Publishing GOCDB]] that can publish its 'EGI' scoped data to the Central GOCDB.</strike><br />
For other current developments see: [[GOCDB/Release4/Development|GOCDB4 Development Plans]]


== 1) Data Tagging/Scoping  ==
== 1) Data Tagging/Scoping  ==
Line 22: Line 24:
'''(Separate EGI and Non-EGI Data)'''  
'''(Separate EGI and Non-EGI Data)'''  


*Confirmed Development. Details: [[GOCDB/Release4/Regionalisation/Data_Scoping]]
*Details: [[GOCDB/Release4/Regionalisation/Data_Scoping]]


A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data. To do this, new tagging logic is required so that new data scoping rules can be applied to GOCDB entities, allowing Sites, Services and other data to be identified as either ‘EGI.EU,’ ‘Local’ or ‘some other’ scoped data. This is a regionalisation prerequisite. The tags will be created along with GUI support for (de)associating tags to gocdb data. This would facilitate:
A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data. To do this, new tagging logic has been introduced so that new data scoping rules can be applied to GOCDB entities, allowing Sites, Services and other data to be identified as either ‘EGI,’ ‘Local’ or ‘some other’ scoped data. This is a regionalisation prerequisite. This would facilitate:
*Storing both local and EGI scoped data in the central GOCDB (for those NGIs who do not have resource to host their own regional instance). This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051.  
*Storing both local and EGI scoped data in the central GOCDB (for those NGIs who do not have resource to host their own regional instance). This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051.  
*Regional instances can manage their own local data. Here, only the EGI.EU scoped data would be copied to the central instance.  
*Regional instances can manage their own local data. Here, only the EGI scoped data would be copied to the central instance.  
*To cater for future requirements, new tags can be created for special use-cases. Although there are no concrete use-cases for this (yet), consider a local site declared in a regional instance.  It emerges that the NGI has a special requirement that this particular site is made available in the central PI, but that it also needs to be out of 'EGI.EU' scope (e.g. for monitoring exclusion purposes). Here, the regional instance would assign the 'REGIONAL' tag to the relevant data. In doing this, the central instance can query the regional GOCDB for its data tagged with 'EGI.EU' and 'REGIONAL' tags (two separate query invocations with different tag parameters). This produces the same result as those NGIs who don’t use a regional instance and declare their local data with the same 'REGIONAL' tag in the central instance (its just a different input mechanism).  
*To cater for future requirements, new tags can be created for special use-cases. Although there are no concrete use-cases for this (yet), consider a local site declared in a regional instance.  It emerges that the NGI has a special requirement that this particular site is made available in the central PI, but that it also needs to be out of 'EGI' scope (e.g. for monitoring exclusion purposes). Here, the regional instance would assign the 'Local' tag to the relevant data. In doing this, the central instance can query the regional GOCDB for its data tagged with 'EGI' and 'Local' tags (two separate query invocations with different tag parameters). This produces the same result as those NGIs who don’t use a regional instance and declare their local data with the same 'Local' tag in the central instance (its just a different input mechanism).  
<!--*For special use-cases, it would also be possible to publish 'Local' scoped data from regional to central; Consider a local site declared in a regional instance.  It emerges that the NGI has a special requirement that this site needs to be made available in the central PI, but that it also needs to be out of 'EGI.EU' scope. Here, the published XML would define the LOCAL tag. This produces the same result as those NGIs who don’t use a regional instance and declare their local sites in the central (just a different input mechanism). -->
<!--*For special use-cases, it would also be possible to publish 'Local' scoped data from regional to central; Consider a local site declared in a regional instance.  It emerges that the NGI has a special requirement that this site needs to be made available in the central PI, but that it also needs to be out of 'EGI.EU' scope. Here, the published XML would define the LOCAL tag. This produces the same result as those NGIs who don’t use a regional instance and declare their local sites in the central (just a different input mechanism). -->
*GOCDB admins (central or regional) can define different tags as required for specific purposes (e.g. virtual groupings).  
*GOCDB admins (central or regional) can define different tags as required for specific purposes.  


Queries to the gocdb PI (central or regional) can restrict to the appropriate scope using new PI parameters, e.g:
Queries to the gocdb PI (central or regional) can restrict to the appropriate scope using new PI parameters, e.g:
* https://goc.egi.eu/gocdbpi/public/?method=get_site_list&tag=EGI.EU
* https://goc.egi.eu/gocdbpi/public/?method=get_site_list&scope=EGI
* https://goc.egi.eu/gocdbpi/public/?method=get_site_list&tag=REGIONAL
* https://goc.egi.eu/gocdbpi/public/?method=get_site_list&scope=Local
* https://ngix.dl.ac.uk/gocdbpi/public/?method=get_site_list&tag=NGI_LOCAL
* or “&amp;scope=VirtualSiteA_WhateverTag”
* or “&amp;tag=VirtualSiteA_WhateverTag”


*More details: [[GOCDB/Release4/Regionalisation/Data_Scoping]]
*More details: [[GOCDB/Release4/Regionalisation/Data_Scoping]]
Line 59: Line 60:
<br/>
<br/>


== 3) Regional-Publishing GOCDB ==
== <strike>3) Regional-Publishing GOCDB</strike> ==
* Long Term Development
 
A regional-standalone GOCDB instance will publish is EGI.EU scoped regional data to the central instance. This would allow NGIs to deploy their own GOCDB, customise it and publish EGI data (constrained by XML schema) to the central portal. The central portal would be the definitive source for all EGI information.
* <b>Support for a Regional Publishing GOCDB has been dropped</b>. This page is for archive purposes only.
<strike>A regional-standalone GOCDB instance could publish is EGI scoped regional data to the central instance (thus becoming a regional-publishing GOCDB). The aim is to allow NGIs to deploy their own GOCDB, customise it and publish EGI data (constrained by XML schema) to the central portal. Therefore, the central portal would still be the definitive source for all EGI information. Note, the developers have some major concerns about the regional-publishing gocdb, see Developers Concerns below.</strike>


'''Notes:'''
'''Notes:'''
Line 78: Line 80:
* Central operators would need to override data published by regional GOCDBs (for https://rt.egi.eu/rt/Ticket/Display.html?id=1094)
* Central operators would need to override data published by regional GOCDBs (for https://rt.egi.eu/rt/Ticket/Display.html?id=1094)


'''Concerns'''
'''Internal Tasks to Complete'''
* Create transfer mechanism
** Identify changed data
** Expose changed data through the PI
** Authenticate data sources
** Validate exposed data
** Add / Edit Objects
** Confirm data has been successfully synchronized
* Native downtime viewing mechanism (Ops portal solution won't work for regional GOCDB)
* Create standard package creation process
* Define release process
* Documentation
** Installation
** Common maintenance tasks
** Answers to common queries
** NGI Specific Modification of a Regional GOCDB
*** Modifying the front end
*** Querying the database
 
== Developers Concerns regarding a Regional-Publishing GOCDB ==


A publishing GOCDB exists to solve three problems:
A publishing GOCDB exists to solve four problems:
* Resilience - EGI data is stored in the central and in regional GOCDB instances
* 1) Resilience - EGI data is stored in both the central and regional GOCDB instances,
* Store all Regional data in the central GOCDB
* 2) Store (publish) all or selected Regional data in the central GOCDB.
** As the ATP system needs all regional data in the central GOCDB instance
* 3) Provide a single point for NGIs to both input and query topology data (no need for NGIs to use or query central instance).
* Provide a single point for NGIs to input topology data
* 4) Customization (of an NGIs regional instance).


These problems could be better addressed if the central GOCDB stores all regional and EGI data and the regional GOCDB shows view and edit forms produced by the central GOCDB alongside the private NGI only data.
Given available effort, we believe these problems could be better addressed if the central GOCDB stores regional and EGI data, while the regional GOCDB presents view and edit forms produced by the central GOCDB alongside its private NGI specific data.


Development for the above proposal would require much less effort than producing and supporting a synchronized system. Some of this extra effort could be used to increase the resilience of the central GOCDB thus addressing the first problem. The remaining extra effort can then be used to solve other issues such as re-working of the user role system and researching and adding support for virtual sites.
Development for the above proposal would require much less effort than producing and supporting a publishing GOCDB. Some of this extra effort could be used to increase the resilience of the central GOCDB thus addressing the first problem. The remaining extra effort can then be used to solve other issues such as re-working of the user role system and researching and adding support for virtual sites.  


Another barrier for a syncing GOCDB is that it reduces the benefit of NGI customizations for the following reasons:
Another barrier for a regional-publishing GOCDB is that it would be far less customizable compared to a regional-standalone gocdb for the following reasons:


* A synchronizing regional GOCDB will have to be updated to the latest version as soon as the central instance updates
* A regional-publishing GOCDB will always have to be updated to the latest version as soon as the central instance updates
** If this doesn't happen we can't guarantee the synchronization mechanism will work
** If this doesn't happen we can't guarantee the synchronization mechanism will work
* Central updates will introduce new features such as certification status rules, new user role management and virtual sites that change fundamental GOCDB behavior and can break compatibility with NGI specific customizations
* Central updates will inevitably introduce new features such as certification status rules, new user role management and virtual sites. This will change fundamental GOCDB behavior and will almost certainly break compatibility with NGI specific customizations
* For each GOCDB update the NGIs will have to provide effort to test and fix their customizations
* For each GOCDB update, the NGIs will have to provide effort to test and fix their customizations
 
* The central GOCDB team don't want to be restricted in development by trying to support NGI customization.
== Our notes / some issues ==
* Implicit PK problems (e.g. sites's SHORT_NAME).
* Transport mechanism (XML REST currently preferred over async message brokers because we need request/response).
* Publisher table - i.e. record every individual object update/insert/delete so that PUT messages can be constructed at a later time.
* Do we publish individual/any objects for insert/update or strict object graphs and their dependencies? (prefer known types 'udateEndpoint' rather than 'updateObject').


== Regional plans per country/region ==
== Regional plans per country/region ==

Latest revision as of 12:32, 18 December 2012

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB4 Development Plans - Current GOCDB4 developments

GOCDB Regionalisation - latest status

Important note: The Regional-Publishing GOCDB has been dropped since the EGI Tech Forum 2012 OTAG

For all other GOCDB development plans/improvements see GOCDB/Release4/Development

Regionalisation Plans

Introduction

Production of a Regional-Publishing GOCDB has been dropped. This page is for archive only.

Steps 1) to 3) below are required for a #3) Regional-Publishing GOCDB that can publish its 'EGI' scoped data to the Central GOCDB.
For other current developments see: GOCDB4 Development Plans

1) Data Tagging/Scoping

(Separate EGI and Non-EGI Data)

A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data. To do this, new tagging logic has been introduced so that new data scoping rules can be applied to GOCDB entities, allowing Sites, Services and other data to be identified as either ‘EGI,’ ‘Local’ or ‘some other’ scoped data. This is a regionalisation prerequisite. This would facilitate:

  • Storing both local and EGI scoped data in the central GOCDB (for those NGIs who do not have resource to host their own regional instance). This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051.
  • Regional instances can manage their own local data. Here, only the EGI scoped data would be copied to the central instance.
  • To cater for future requirements, new tags can be created for special use-cases. Although there are no concrete use-cases for this (yet), consider a local site declared in a regional instance. It emerges that the NGI has a special requirement that this particular site is made available in the central PI, but that it also needs to be out of 'EGI' scope (e.g. for monitoring exclusion purposes). Here, the regional instance would assign the 'Local' tag to the relevant data. In doing this, the central instance can query the regional GOCDB for its data tagged with 'EGI' and 'Local' tags (two separate query invocations with different tag parameters). This produces the same result as those NGIs who don’t use a regional instance and declare their local data with the same 'Local' tag in the central instance (its just a different input mechanism).
  • GOCDB admins (central or regional) can define different tags as required for specific purposes.

Queries to the gocdb PI (central or regional) can restrict to the appropriate scope using new PI parameters, e.g:


Use Cases:

  • Users can centrally include or exclude sites/SEs from the from the EGI infrastructure as required (provides extensible data visibility/scoping according to tag groupings).
  • Whether using Regional or Central instance, a single point is provided to input and query data for both EGI and local scoped data.
  • PI queries are the same across central and regional gocdbs.
  • Regional GOCDB installation is a choice (e.g. for NGIs without effort to host a regional gocdb).



2) Regional-Standalone GOCDB

  • Confirmed Development

We will continue to release a standalone instance of the GOCDB for regional NGI use. This regional instance will be supported regardless of other developments. We will ensure that future developments don't conflict with requirements from NGIs by capturing the NGIs requirements.

Use Cases:

  • NGIs can fully customise their own GOCDB as required (e.g. extend DDL/GUI/logic/SQL/whatever)
  • Support available centrally



3) Regional-Publishing GOCDB

  • Support for a Regional Publishing GOCDB has been dropped. This page is for archive purposes only.

A regional-standalone GOCDB instance could publish is EGI scoped regional data to the central instance (thus becoming a regional-publishing GOCDB). The aim is to allow NGIs to deploy their own GOCDB, customise it and publish EGI data (constrained by XML schema) to the central portal. Therefore, the central portal would still be the definitive source for all EGI information. Note, the developers have some major concerns about the regional-publishing gocdb, see Developers Concerns below.

Notes:

  • It may be necessary to have two-way synchronisation, where the central instance would also update regional instances: https://rt.egi.eu/rt/Ticket/Display.html?id=1094.
  • NGI defined extensions or customizations to the standard GOCDB data types would not be published to the central GOCDB (such customizations, which can be thought of as adding extra columns to the db tables, would not be supported by the strongly typed XSD schema used when publishing data to the central instance);
  • #1) Data Grouping/Scoping is necessary in order to publish the appropriate data.

Details:

Use Cases:

  • NGIs can extend their own GOCDB as required (e.g. extend DDL/GUI/logic/SQL)
    • (note, these customizations should extend gocdb, leaving the core tables/queries/schema in tact in order to publish the core EGI data, e.g. custom data should be defined in a different table space).
  • Single access point to input both EGI and non-EGI data, and query data.
  • Central operators would need to override data published by regional GOCDBs (for https://rt.egi.eu/rt/Ticket/Display.html?id=1094)

Internal Tasks to Complete

  • Create transfer mechanism
    • Identify changed data
    • Expose changed data through the PI
    • Authenticate data sources
    • Validate exposed data
    • Add / Edit Objects
    • Confirm data has been successfully synchronized
  • Native downtime viewing mechanism (Ops portal solution won't work for regional GOCDB)
  • Create standard package creation process
  • Define release process
  • Documentation
    • Installation
    • Common maintenance tasks
    • Answers to common queries
    • NGI Specific Modification of a Regional GOCDB
      • Modifying the front end
      • Querying the database

Developers Concerns regarding a Regional-Publishing GOCDB

A publishing GOCDB exists to solve four problems:

  • 1) Resilience - EGI data is stored in both the central and regional GOCDB instances,
  • 2) Store (publish) all or selected Regional data in the central GOCDB.
  • 3) Provide a single point for NGIs to both input and query topology data (no need for NGIs to use or query central instance).
  • 4) Customization (of an NGIs regional instance).

Given available effort, we believe these problems could be better addressed if the central GOCDB stores regional and EGI data, while the regional GOCDB presents view and edit forms produced by the central GOCDB alongside its private NGI specific data.

Development for the above proposal would require much less effort than producing and supporting a publishing GOCDB. Some of this extra effort could be used to increase the resilience of the central GOCDB thus addressing the first problem. The remaining extra effort can then be used to solve other issues such as re-working of the user role system and researching and adding support for virtual sites.

Another barrier for a regional-publishing GOCDB is that it would be far less customizable compared to a regional-standalone gocdb for the following reasons:

  • A regional-publishing GOCDB will always have to be updated to the latest version as soon as the central instance updates
    • If this doesn't happen we can't guarantee the synchronization mechanism will work
  • Central updates will inevitably introduce new features such as certification status rules, new user role management and virtual sites. This will change fundamental GOCDB behavior and will almost certainly break compatibility with NGI specific customizations
  • For each GOCDB update, the NGIs will have to provide effort to test and fix their customizations
  • The central GOCDB team don't want to be restricted in development by trying to support NGI customization.

Regional plans per country/region

Initial plans (as of March 2009)

This page gives an overview of regional plans as discussed and presented during SA1 face to face coordination meeting in Catania, 4 march 2009. See slides available on http://indico.cern.ch/conferenceDisplay.py?confId=51626 for more details on each region/country plans.

NGI plans matrix

  • Region 1 means deploying and using a distributed, customisable version of current GOCDB, provided by RAL-STFC
  • Region 2 means continuing using central GOCDB
  • Region 3 means deploying/using another tool than GOCDB, interfaced with central GOCDB for interoperability purposes
NGI Region1 Region2 Region3 Timeline Notes
Asia Pacific ROC X
Benelux X
Czech Republic X Tests started early 2010
Denmark X
Finland X
France X Discussions engaged Nov 2009
Germany X Tests started Jul 2009
Greece X Potentially interested
Ireland X Second region 1 pilot
Italy X
Norway X
Poland X
Russia X
South East Europe ROC X Tests started Will use HGSM - First region 3 pilot
South West Europe ROC X Start integration by Feb 2010 Will use HGSM - Second region 3 pilot
Sweden X
United Kingdom X Tests started Jun 2009 First region 1 pilot
Others X "Catch-all" usecase for all countries/regions not planning for regionalisation yet.

Specific regional usecases - discussions, requests and ideas

UK NGI

Discussion has first started with NGS. A first summary of GOCDB possible use is available on [NGS wiki] .

  • idea from RHUL site: we could have differenciated roles for site admins, e.g. a "primary site admin" which is the one who is contacted when needed. Additional roles can be region specific if we don't need them at a central level
  • Current testing of a UK GOCDB module is done at RAL

South East Europe

  • SEE ROC plans are to use HGSM as their production regional repository.
  • Integration work between HGSM and GOCDB has started. Exchange format is being defined and will be tested during October 2009. Further testing will done in November and December.
  • HGSM will be official production repository for SEE by January 2010

Grid-Ireland

Grid-Ireland needs to deploy a standalone GOCDB to be integrated to their certification and test infrastructure.

Italy

  • Italy would be in favour of a MySQL version of regional GOCDB.
  • Italian model might become hierarchical: one regional GOCDB with 3 "sub-regional" GOCDB below that.

DECH

  • DECH ROC (for Germany/Switzerland) has agreed to be the 4th pilot distribution instance
  • Current testing of a UK GOCDB module is done at KIT

Serbia

  • Serbia would like to try to install and use GOCDB4 as Serbian Grid (AEGIS)information service.

Czech NGI

  • The Czech NGI is currently deploying the test regional GOCDB package for evaluation.