Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

GOCDB/Transfer Mechanism

From EGIWiki
Jump to navigation Jump to search

Introduction

The mechanism to transfer data from a regional GOCDB to a central GOCDB is separated into two parts: regional instance tasks and central instance tasks.

Regional Instance Tasks

Whenever a CRUD (create, replace, update, delete) operation is performed on a fundamental GOCDB object (site, service endpoint, downtime, user, user role) the regional GOCDB will record the event. This record will include the object Id, the type of operation (CRUD) and the date and time at which the object was modified. The record will also show whether the object has been successfully synchronised with the central GOCDB via a synchronisation boolean. By default the synchronisation field will be false.

This table will be used to create a PI query exposing EGI data that has changed in the regional GOCDB:

  • gocdbpi/?method=synchronise_egi_data

The central GOCDB will periodically read from this queries and acknowledge successfully synchronised data by posting the time of the last successfully synchronised object back to the regional portal.

e.g. the central GOCDB will perform an HTTP GET of https://regional.gocdb.ngi.com/gocdbpi/confirm_sync.php?last_date=11.12.01 10-07-2012

When the regional GOCDB receives a request for this URL it will change the synchronised boolean in the synchronisation table as true provided that the modification date was before or at the time provided in the last_date field. This URL will be secured to only accept requests if the request has identified itself with the central GOCDB certificate.

Central Instance Tasks

  • Periodically poll all known regional instances
  • Check remote certificate is who we think it should be
  • Retrieve all unsynchronised data
  • (Read this in chunks to avoid loading all XML into memory at once)
  • Validate Data
    • Check that each object's primary_key (e.g. 212G3) has a grid id (G3) that matches the cert they present (e.g. grid ID 3 = france = in2p3 GOCDB cert)