Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Regionalisation"

From EGIWiki
Jump to navigation Jump to search
Line 19: Line 19:
Steps 1) to 3) below are required for a [[#3) Regional-Synchronised GOCDB]] that can publish its 'EGI scoped' data to the Central GOCDB. These are long term plans into EGI-Y3.
Steps 1) to 3) below are required for a [[#3) Regional-Synchronised GOCDB]] that can publish its 'EGI scoped' data to the Central GOCDB. These are long term plans into EGI-Y3.


== 1) Data Grouping/Scoping ==  
== 1) Data Grouping/Scoping ==
 
'''(Separate EGI and Non-EGI Data)'''  
'''(Separate EGI and Non-EGI Data)'''  
* Confirmed Development
A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data.
To do this, new ‘tagging’ logic is required so that new data-scoping rules can be applied to GOCDB entities, allowing Sites, Services and other GOCDB data to be identified as either ‘EGI,’ ‘Local’ or ‘some other’ scoped data. This is a non-trivial regionalisation pre-requisite. The tags/groups will be created along with tools to associate / remove tags. With this functionality in place NGIs will be able to store and query for both local and EGI sites/services in the central GOCDB. (This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051).


'''Notes:'''
*Confirmed Development
* This will require GUI enhancements, allowing users to associate tags with gocdb objects, and new request parameters for restricting the PI queries based on tag/group memberships (e.g. “method=get_site_list_by_group&group=EGI” or “&group=local” or “&group=any”).
 
* This development is a pre-requisite for regionalisation so that only the appropriate data can be published in a: [[#3) Regional-Synchronised GOCDB]].  
A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data. To do this, new ‘tagging’ logic is required so that new data-scoping rules can be applied to GOCDB entities, allowing Sites, Services and other GOCDB data to be identified as either ‘EGI,’ ‘Local’ or ‘some other’ scoped data. This is a non-trivial regionalisation pre-requisite. The tags/groups will be created along with tools to associate / remove tags. With this functionality in place NGIs will be able to store and query for both local and EGI sites/services in the central GOCDB. (This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051).
 
'''Notes:'''  
 
*This will require GUI enhancements, allowing users to associate tags with gocdb objects, and new request parameters for restricting the PI queries based on tag/group memberships (e.g. “method=get_site_list_by_group&group=EGI” or “&group=local” or “&group=any”).  
*This development is a pre-requisite for regionalisation so that only the appropriate data can be published in a: [[#3.29_Regional-Synchronised_GOCDB]].
 
'''Use Cases:'''
 
*Users can centrally include/exclude sites/SEs from the from the whole EGI infrastructure as required (provides extensible data visibility/scoping according to tags/groupings).
*Single point to input/query data for EGI and local/regional data.
*Regional GOCDB installation not required (e.g. for NGIs without effort to install/maintain a regional gocdb).


'''Use Cases:'''
'''Our Notes (scopes = tag cloud):'''  
* Users can centrally include/exclude sites/SEs from the from the whole EGI infrastructure as required (provides extensible data visibility/scoping according to tags/groupings).
* Single point to input/query data for EGI and local/regional data.
* Regional GOCDB installation not required (e.g. for NGIs without effort to install/maintain a regional gocdb).


'''Our Notes (scopes = tag cloud):'''
*Many to many relationship between 'tag' object and data objects (i.e. producing few tag instances, but many links to those tags). A single data object can be linked to multiple (parent) tags while an individual tag can link to many (child) objects (site, SE, downtime, user).  
* Many to many relationship between 'tag' object and data objects (i.e. producing few tag instances, but many links to those tags). A single data object can be linked to multiple (parent) tags while an individual tag can link to many (child) objects (site, SE, downtime, user).  
*In the relationship between Tags and gocdb data entities, Tag is always the parent of the data objects for permissions reasons (e.g. we want users to be able to link/un-link to an existing tag, but not be able to change/re-name the tag).  
* In the relationship between Tags and gocdb data entities, Tag is always the parent of the data objects for permissions reasons (e.g. we want users to be able to link/un-link to an existing tag, but not be able to change/re-name the tag).  
*Tag instances can only be added / removed by gocdb admins and users can choose to link/un-link their data objects with that tag.  
* Tag instances can only be added / removed by gocdb admins and users can choose to link/un-link their data objects with that tag.  
*Tags that are linked to a parent object will also be linked to child objects by default when creating the object (i.e. cascading the same tags to children). For example, when adding a new SE to a site, the site's tags are also linked to the SE object. However, the user can always make subsequent modifications if required (e.g. un-link the SE's 'EGI tag' to exclude the SE from the EGI scoped data).  
* Tags that are linked to a parent object will also be linked to child objects by default when creating the object (i.e. cascading the same tags to children). For example, when adding a new SE to a site, the site's tags are also linked to the SE object. However, the user can always make subsequent modifications if required (e.g. un-link the SE's 'EGI tag' to exclude the SE from the EGI scoped data).
*Internal design: [[GOCDB/Data Scoping]


== 2) Regional-Standalone GOCDB ==
== 2) Regional-Standalone GOCDB ==

Revision as of 16:34, 15 June 2011

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Tools menu: Main page Instructions for developers AAI Proxy Accounting Portal Accounting Repository AppDB ARGO GGUS GOCDB
Message brokers Licenses OTAGs Operations Portal Perun EGI Collaboration tools LToS EGI Workload Manager


Back to GOCDB/Documentation_Index

GOCDB Regionalisation - latest status

The last released version of GOCDB regional module dates from October 2010 (gocdb4.0d7). It is a fully functional tool, and all the basic components are included so that initial deployment can be tested using that version. It is available from https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.0d7_distrib/RPM/

The code of the latest development version of the module is available from the SVN repository at http://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb/

The documentation is available on the wiki page GOCDB Regional Module Technical Documentation.

Important note: GOCDB regional modules can be operated in full production but can't synchronise to the central GOCDB.

For all GOCDB development plans/improvements see GOCDB/Release4/Development

Regionalisation Plans

Introduction

Steps 1) to 3) below are required for a #3) Regional-Synchronised GOCDB that can publish its 'EGI scoped' data to the Central GOCDB. These are long term plans into EGI-Y3.

1) Data Grouping/Scoping

(Separate EGI and Non-EGI Data)

  • Confirmed Development

A GOCDB instance (whether central or regional) needs to differentiate between EGI and non-EGI data. To do this, new ‘tagging’ logic is required so that new data-scoping rules can be applied to GOCDB entities, allowing Sites, Services and other GOCDB data to be identified as either ‘EGI,’ ‘Local’ or ‘some other’ scoped data. This is a non-trivial regionalisation pre-requisite. The tags/groups will be created along with tools to associate / remove tags. With this functionality in place NGIs will be able to store and query for both local and EGI sites/services in the central GOCDB. (This plan is from "Design A" in https://rt.egi.eu/rt/Ticket/Display.html?id=943#txn-52051).

Notes:

  • This will require GUI enhancements, allowing users to associate tags with gocdb objects, and new request parameters for restricting the PI queries based on tag/group memberships (e.g. “method=get_site_list_by_group&group=EGI” or “&group=local” or “&group=any”).
  • This development is a pre-requisite for regionalisation so that only the appropriate data can be published in a: #3.29_Regional-Synchronised_GOCDB.

Use Cases:

  • Users can centrally include/exclude sites/SEs from the from the whole EGI infrastructure as required (provides extensible data visibility/scoping according to tags/groupings).
  • Single point to input/query data for EGI and local/regional data.
  • Regional GOCDB installation not required (e.g. for NGIs without effort to install/maintain a regional gocdb).

Our Notes (scopes = tag cloud):

  • Many to many relationship between 'tag' object and data objects (i.e. producing few tag instances, but many links to those tags). A single data object can be linked to multiple (parent) tags while an individual tag can link to many (child) objects (site, SE, downtime, user).
  • In the relationship between Tags and gocdb data entities, Tag is always the parent of the data objects for permissions reasons (e.g. we want users to be able to link/un-link to an existing tag, but not be able to change/re-name the tag).
  • Tag instances can only be added / removed by gocdb admins and users can choose to link/un-link their data objects with that tag.
  • Tags that are linked to a parent object will also be linked to child objects by default when creating the object (i.e. cascading the same tags to children). For example, when adding a new SE to a site, the site's tags are also linked to the SE object. However, the user can always make subsequent modifications if required (e.g. un-link the SE's 'EGI tag' to exclude the SE from the EGI scoped data).
  • Internal design: [[GOCDB/Data Scoping]

2) Regional-Standalone GOCDB

  • Confirmed Development

We will continue to release a standalone instance of the GOCDB for regional NGI use. This regional instance will be supported regardless of other developments. We will ensure that future developments don't conflict with requirements from NGIs by capturing the NGIs requirements.

Use Cases:

  • NGIs can fully customise their own GOCDB as required (e.g. extend DDL/GUI/logic/SQL/whatever)
  • Support available centrally


3) Regional-Publishing GOCDB

  • Long Term Development

A regional-standalone GOCDB instance will publish is EGI scoped regional data to the central instance.

This would allow NGIs to deploy their own GOCDB, customise it and publish EGI data (constrained by XML schema) to the central portal. Publishing of this data would have to be transactional, most probably via a schema constrained WS/REST interface (preferable over asynch messaging which would require response queues + ACKs). The central portal would be the definitive source for all EGI information.

Notes:

Use Cases:

  • NGIs can extend their own GOCDB as required (e.g. extend DDL/GUI/logic/SQL)
    • (note, these customizations should extend gocdb, leaving the core tables/queries/schema in tact in order to publish the core EGI data, e.g. custom data should be defined in a different table space).
  • Single access point to input both EGI and non-EGI data, and query data.
  • Central operators would need to override data published by regional GOCDBs (for https://rt.egi.eu/rt/Ticket/Display.html?id=1094)

Issues

  • Implicit PK problems (e.g. sites's SHORT_NAME).
  • Currently PK's are created by '<objID>G<gridID>' which mandates gridID support (are UUIDs better?). Our assumption is that object ids are created by the regional portal (not central).
  • Authenticate each/every XML submission (regional instance will require host cert)
  • Authorize inserts/updates of the request against their unique gridID.
  • Distinguish between data inserts and updates. Before updating, perform implicit selects to determine that the data does already exist and belongs to the caller.
  • Should we support different GridIDs? on the central portal?
  • XML Well formed, valid + extra logical checks of the element/attribute values for 'logical' correctness.
  • Transport mechanism (XML REST currently preferred over async message brokers because we need request/response).
  • Publisher table - i.e. record every individual object update/insert/delete so that PUT messages can be constructed at a later time.
  • Do we publish individual/any objects for insert/update or strict object graphs and their dependencies? (prefer known types 'udateEndpoint' rather than 'updateObject').


Regional plans per country/region

Initial plans (as of March 2009)

This page gives an overview of regional plans as discussed and presented during SA1 face to face coordination meeting in Catania, 4 march 2009. See slides available on http://indico.cern.ch/conferenceDisplay.py?confId=51626 for more details on each region/country plans.

NGI plans matrix

  • Region 1 means deploying and using a distributed, customisable version of current GOCDB, provided by RAL-STFC
  • Region 2 means continuing using central GOCDB
  • Region 3 means deploying/using another tool than GOCDB, interfaced with central GOCDB for interoperability purposes
NGI Region1 Region2 Region3 Timeline Notes
Asia Pacific ROC X
Benelux X
Czech Republic X Tests started early 2010
Denmark X
Finland X
France X Discussions engaged Nov 2009
Germany X Tests started Jul 2009
Greece X Potentially interested
Ireland X Second region 1 pilot
Italy X
Norway X
Poland X
Russia X
South East Europe ROC X Tests started Will use HGSM - First region 3 pilot
South West Europe ROC X Start integration by Feb 2010 Will use HGSM - Second region 3 pilot
Sweden X
United Kingdom X Tests started Jun 2009 First region 1 pilot
Others X "Catch-all" usecase for all countries/regions not planning for regionalisation yet.

Specific regional usecases - discussions, requests and ideas

UK NGI

Discussion has first started with NGS. A first summary of GOCDB possible use is available on [NGS wiki] .

  • idea from RHUL site: we could have differenciated roles for site admins, e.g. a "primary site admin" which is the one who is contacted when needed. Additional roles can be region specific if we don't need them at a central level
  • Current testing of a UK GOCDB module is done at RAL

South East Europe

  • SEE ROC plans are to use HGSM as their production regional repository.
  • Integration work between HGSM and GOCDB has started. Exchange format is being defined and will be tested during October 2009. Further testing will done in November and December.
  • HGSM will be official production repository for SEE by January 2010

Grid-Ireland

Grid-Ireland needs to deploy a standalone GOCDB to be integrated to their certification and test infrastructure.

Italy

  • Italy would be in favour of a MySQL version of regional GOCDB.
  • Italian model might become hierarchical: one regional GOCDB with 3 "sub-regional" GOCDB below that.

DECH

  • DECH ROC (for Germany/Switzerland) has agreed to be the 4th pilot distribution instance
  • Current testing of a UK GOCDB module is done at KIT

Serbia

  • Serbia would like to try to install and use GOCDB4 as Serbian Grid (AEGIS)information service.

Czech NGI

  • The Czech NGI is currently deploying the test regional GOCDB package for evaluation.