GOCDB/Transfer Mechanism

Introduction

The mechanism to transfer data from a regional GOCDB to a central GOCDB is separated into two parts: regional instance tasks and central instance tasks.

Regional Instance Tasks

Whenever a CRUD (create, replace, update, delete) operation is performed on a fundamental GOCDB object (site, service endpoint, downtime, user, user role) the regional GOCDB will record the event. This record will include the object Id, the type of operation (CRUD) and the date and time at which the object was modified. The record will also show whether the object has been successfully synchronised with the central GOCDB via a synchronisation boolean. By default the synchronisation field will be false.

This table will be used to create a PI query exposing EGI data that has changed in the regional GOCDB including the date the change was made. This data will be in chronological order (oldest first):

• gocdbpi/?method=synchronise_egi_data

The central GOCDB will periodically read from this queries and acknowledge successfully synchronised data by posting the time of the last successfully synchronised object back to the regional portal.

e.g. the central GOCDB will perform an HTTP GET of https://regional.gocdb.ngi.com/gocdbpi/confirm_sync.php?last_date=11.12.01 10-07-2012

When the regional GOCDB receives a request for this URL it will change the synchronised boolean in the synchronisation table as true provided that the modification date was before or at the time provided in the last_date field. This URL will be secured to only accept requests if the request has identified itself with the central GOCDB certificate.

Central Instance Tasks

• Periodically poll all known regional instances
• Check remote certificate is who we think it should be
• Retrieve all unsynchronised data
• (Read this in chunks to avoid loading all XML into memory at once)
• Validate Data
  • Check that each object's primary_key (e.g. 212G3) has a grid id (G3) that matches the cert they present (e.g. grid ID 3 = france = in2p3 GOCDB cert)
  • Check each item (site, SE, downtime, user, user role) is formatted correctly as per XML schema
  • Validate fields against gocdb_schema.xml (potentially we can do this using the XML schema)
• Insert each item (site, se, downtime, user, user role) atomically
  • Note the modification date of the object we're inserting
    
  • Roll operation back if it fails
    • Stop the automatic sync for this regional GOCDB
    • Send alert e-mail
• HTTP GET https://regional.gocdb.ngi.com/gocdbpi/confirm_sync.php?last_date=11.12.01 (with the modification date of the last successfully inserted item)