GOCDB/Release4/Regionalisation
Main | EGI.eu operations services | Support | Documentation | Tools | Activities | Performance | Technology | Catch-all Services | Resource Allocation | Security |
Tools menu: | • Main page | • Instructions for developers | • AAI Proxy | • Accounting Portal | • Accounting Repository | • AppDB | • ARGO | • GGUS | • GOCDB |
• Message brokers | • Licenses | • OTAGs | • Operations Portal | • Perun | • EGI Collaboration tools | • LToS | • EGI Workload Manager |
Back to GOCDB/Documentation_Index
GOCDB Regionalisation - latest status
The last released version of GOCDB regional module dates from October 2010 (gocdb4.0d7). It is a fully functional tool, and all the basic components are included so that initial deployment can be tested using that version. It is available from https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.0d7_distrib/RPM/
The code of the latest development version of the module is available from the SVN repository at http://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb/
The documentation is available on the wiki page GOCDB Regional Module Technical Documentation.
Important note: GOCDB regional modules can be operated in full production but can't synchronise to the central GOCDB.
For all GOCDB development plans/improvements see GOCDB/Release4/Development
Regionalisation Plans
Introduction
Steps 1) to 3) below are required for a #3) Regional-Publishing GOCDB that can publish its 'EGI.EU' scoped data to the Central GOCDB. These are long term plans into EGI-Y3.
1) Data Grouping/Scoping
(Separate EGI and Non-EGI Data when publishing data from regional to central and when querying the PI)
- Confirmed Development
A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data. To do this, new tagging logic is required so that new data scoping rules can be applied to GOCDB entities, allowing Sites, Services and other data to be identified as either ‘EGI.EU,’ ‘Local’ or ‘some other’ scoped data. This is a regionalisation prerequisite. The tags will be created along with GUI support for (de)assigning tags to gocdb data. This would facilitate:
- Storing both local and EGI scoped data in the central GOCDB (for those NGIs who do not have resource to host their own regional instance). This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051.
- Regional instances can manage their own local data. Here, only the EGI.EU scoped data would be copied to the central instance.
- If required (no use-case yet, but possible), new tags can be created for special use-cases; Consider a local site declared in a regional instance. It emerges that the NGI has a special requirement that this site is made available in the central PI, but that it also needs to be out of 'EGI.EU' scope. Here, the regional instance would assign the 'CENTRAL_GOCDB.LOCAL' tag to the relevant data so that the central instance can query the regional GOCDB for data tagged with either 'EGI.EU' and/or 'CENTRAL_GOCDB.LOCAL' tags. This produces the same result as those NGIs who don’t use a regional instance and declare their data with the same 'CENTRAL_GOCDB.LOCAL' tag in the central instance (its just a different input mechanism).
- GOCDB admins (central or regional) can define different tags as required for specific purposes (e.g. virtual groupings).
Queries to the gocdb PI (central or regional) can restrict to the appropriate scope using new PI parameters, e.g:
- https://goc.egi.eu/gocdbpi/public/?method=get_site_list&tags=EGI.EU
- or“&tags=Local.EGI” or “&tags=EGI.EU,VirtualSiteA”
- Details:
Use Cases:
- Users can centrally include or exclude sites/SEs from the from the EGI infrastructure as required (provides extensible data visibility/scoping according to tag groupings).
- Whether using Regional or Central instance, a single point is provided to input and query data for both EGI and local scoped data.
- Regional GOCDB installation is a choice (e.g. for NGIs without effort to host a regional gocdb).
2) Regional-Standalone GOCDB
- Confirmed Development
We will continue to release a standalone instance of the GOCDB for regional NGI use. This regional instance will be supported regardless of other developments. We will ensure that future developments don't conflict with requirements from NGIs by capturing the NGIs requirements.
Use Cases:
- NGIs can fully customise their own GOCDB as required (e.g. extend DDL/GUI/logic/SQL/whatever)
- Support available centrally
3) Regional-Publishing GOCDB
- Long Term Development
A regional-standalone GOCDB instance will publish is EGI scoped regional data to the central instance.
This would allow NGIs to deploy their own GOCDB, customise it and publish EGI data (constrained by XML schema) to the central portal. Publishing of this data would have to be transactional, most probably via a schema constrained WS/REST interface (preferable over asynch messaging which would require response queues + ACKs). The central portal would be the definitive source for all EGI information.
Notes:
- It may be necessary to have two-way synchronisation, where the central instance would also update regional instances: https://rt.egi.eu/rt/Ticket/Display.html?id=1094.
- NGI extensions and custom data would not be published to the central GOCDB, therefore;
- #1) Data Grouping/Scoping is necessary in order to publish the appropriate data.
Use Cases:
- NGIs can extend their own GOCDB as required (e.g. extend DDL/GUI/logic/SQL)
- (note, these customizations should extend gocdb, leaving the core tables/queries/schema in tact in order to publish the core EGI data, e.g. custom data should be defined in a different table space).
- Single access point to input both EGI and non-EGI data, and query data.
- Central operators would need to override data published by regional GOCDBs (for https://rt.egi.eu/rt/Ticket/Display.html?id=1094)
Our notes / some issues
- Implicit PK problems (e.g. sites's SHORT_NAME).
- Transport mechanism (XML REST currently preferred over async message brokers because we need request/response).
- Publisher table - i.e. record every individual object update/insert/delete so that PUT messages can be constructed at a later time.
- Do we publish individual/any objects for insert/update or strict object graphs and their dependencies? (prefer known types 'udateEndpoint' rather than 'updateObject').
Regional plans per country/region
Initial plans (as of March 2009)
This page gives an overview of regional plans as discussed and presented during SA1 face to face coordination meeting in Catania, 4 march 2009. See slides available on http://indico.cern.ch/conferenceDisplay.py?confId=51626 for more details on each region/country plans.
NGI plans matrix
- Region 1 means deploying and using a distributed, customisable version of current GOCDB, provided by RAL-STFC
- Region 2 means continuing using central GOCDB
- Region 3 means deploying/using another tool than GOCDB, interfaced with central GOCDB for interoperability purposes
NGI | Region1 | Region2 | Region3 | Timeline | Notes |
---|---|---|---|---|---|
Asia Pacific ROC | X | ||||
Benelux | X | ||||
Czech Republic | X | Tests started early 2010 | |||
Denmark | X | ||||
Finland | X | ||||
France | X | Discussions engaged Nov 2009 | |||
Germany | X | Tests started Jul 2009 | |||
Greece | X | Potentially interested | |||
Ireland | X | Second region 1 pilot | |||
Italy | X | ||||
Norway | X | ||||
Poland | X | ||||
Russia | X | ||||
South East Europe ROC | X | Tests started | Will use HGSM - First region 3 pilot | ||
South West Europe ROC | X | Start integration by Feb 2010 | Will use HGSM - Second region 3 pilot | ||
Sweden | X | ||||
United Kingdom | X | Tests started Jun 2009 | First region 1 pilot | ||
Others | X | "Catch-all" usecase for all countries/regions not planning for regionalisation yet. |
Specific regional usecases - discussions, requests and ideas
UK NGI
Discussion has first started with NGS. A first summary of GOCDB possible use is available on [NGS wiki] .
- idea from RHUL site: we could have differenciated roles for site admins, e.g. a "primary site admin" which is the one who is contacted when needed. Additional roles can be region specific if we don't need them at a central level
- Current testing of a UK GOCDB module is done at RAL
South East Europe
- SEE ROC plans are to use HGSM as their production regional repository.
- Integration work between HGSM and GOCDB has started. Exchange format is being defined and will be tested during October 2009. Further testing will done in November and December.
- HGSM will be official production repository for SEE by January 2010
Grid-Ireland
Grid-Ireland needs to deploy a standalone GOCDB to be integrated to their certification and test infrastructure.
Italy
- Italy would be in favour of a MySQL version of regional GOCDB.
- Italian model might become hierarchical: one regional GOCDB with 3 "sub-regional" GOCDB below that.
DECH
- DECH ROC (for Germany/Switzerland) has agreed to be the 4th pilot distribution instance
- Current testing of a UK GOCDB module is done at KIT
Serbia
- Serbia would like to try to install and use GOCDB4 as Serbian Grid (AEGIS)information service.
Czech NGI
- The Czech NGI is currently deploying the test regional GOCDB package for evaluation.