From EGIWiki
Jump to: navigation, search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security

Tools menu: Main page Instructions for developers AAI Proxy Accounting Portal Accounting Repository AppDB ARGO GGUS GOCDB
Message brokers Licenses OTAGs Operations Portal Perun EGI Collaboration tools LToS EGI Workload Manager


Patches, Bug Fixes and Documentation Changes

  • Trim whitespace from text entry form fields
  • Add a "read-only" check to the Write API
  • Add support for EGI Dev IdP "out of the box"
  • Use new EGI Check-in entitlement to access GOCDB
  • Write API code refactors to improve maintainability
  • Use consistent wording in portal when describing downtimes
  • Update to README.md in GitHub repository

Full details can be found here: https://github.com/GOCDB/gocdb/milestone/4?closed=1


Patches, Bug Fixes and Documentation Changes

  • Replace icons used on GOCDB pages with new icons of known source and add a 'delete' icon.
  • Allow "org.squid-cache.Squid" ServiceType to be production and not monitored (https://ggus.eu/index.php?mode=ticket_info&ticket_id=139924).
  • Fix GOCDB_monitor checks on SL7
  • Add LICENSE file to GitHub repository


Patches, Bug Fixes and Documentation Changes

Full details can be found here: https://github.com/GOCDB/gocdb/milestone/2?closed=1

1. this may require a change in your config/local_info.xml as follows:


<google> <!-- Show google map on start page -->






  • Released: December 7th
  • This release adds the following changes
  • Please report issues to gocdb-admins at egi.eu

New Write API for Custom Properties

New attributes added on ServiceEndpoint

  • New boolean 'monitored' flag on SE used for ARGO monitoring.
    • Adds a new 'ENDPOINT_MONITORED' element to the XML output of each endpoint in the 'get_service' method.
  • New 'email' field on SE for adding contact details for that service endpoint (rather than always relying on the parent Site).
    • The SE email is NOT currently rendered in the output of the 'get_service' method - this method has a Level 1 protection level which is considered a public method containing no critical information and no personal email/details: https://wiki.egi.eu/wiki/GOCDB/PI/Technical_Documentation#Protection_levels If emails are needed, we can either:
      • Elevate the protection level of the get_service method to Level 2 (which means a certificate would be needed to query the method)
      • Relax EGI/GOCDB protection level rules so that Level 1 can include site/service emails but no personal emails, note however there would still be a risk that personal emails would still be added.

Optional Cursor Paging on Read API

  • This release applies optional CURSOR-based paging to the majority of the API methods and includes XML output changes !
  • Cursor paging allows clients to iterate through massive result-sets which are split into one or more smaller pages. This allows result-sets to be processed that would otherwise be too large or memory-expensive to process in a single query.
  • Paging may not always be necessary, we suggest you first try to execute your query without paging. If the query fails due to breaching of the execution time or memory limit, then try adding the next_cursor=0.
  • It replaces the OFFSET-based paging that was previously available on the ‘get_downtime*’ methods due to possible data inconsistency issues when offset-paging over real-time data,
  • Most of the API methods now support optional 'next_cursor' and 'prev_cursor' URL params.
    • The following methods will not support paging: get_site_list, get_roc_list, get_service_types, get_site_count_per_country.
  • For backward compatibility, pagination is disabled by default and will only be enabled when passing one of the new cursor query params.
    • If neither the 'next_cursor' or 'prev_cursor' URL params are specified, the method behaves as before by fetching/rendering the requested result-set in one query, and will not add the <meta> element as detailed below.
    • A non-paged query can therefore fail/timout if either the execution time or memory limits are exceeded (e.g. if you make a request for every downtime without narrowing the search with extra filter params).
  • Paged XML results introduce the <meta> element which contain links to the current/self, next, previous and first page of results.
  • When using paging, you need to move forward through the results by reading the next links until the ‘<count>’ element hits zero which indicates no more results are available at that particular moment in time.
  • To start at the beginning of a result-set, specify 'next_cursor=0'
  • The max page size has been initially set to 400 100 but this is subject to change/refinement. NB: the max page size value is provided in the <max_page_size> element.
  • The <count> element shows how many results are on the current page (note, if this figure is less then <max_page_size> then you have come to the end of the results, and at that moment in time, the next page will show no results with a count of zero).

Changes to the Read API:

  • Support for the 'page' URL parameter used for offset-paging on the 'get_downtime*' methods has been removed.
  • All results are now ordered by ascending ID. NB: this changes the ‘get_downtime*’ methods which previously ordered downtimes by descending start date.
  • The <meta> element is added to the XML results when paging via the 'next_cursor' or 'prev_cursor' URL params.

Here are some API queries to get you started (note that the gocdb-test data is a few weeks stale and will be updated periodically):


Use of eppn in EGI

Data Privacy and Code of Conduct Policy Document Under Construction: GOCDB/data privacy

Hi Ops/all,

Apologies for the lengthy text:

With the plans to include federated identity, services like GocDB (which falls under the R&S REFEDS SP Category) will need to use an attribute supplied by an IdP to create/identify a user account, usually ‘eduPersonPrincipleName’ (eppn) (or less likely ‘eduPersonTargetedID’, eptid, see discussion below). These attributes will be used instead of an x509 DN and will require changes to the output of the PI, changing CERTDN element to PRINCIPAL element in queries like get_user and get_site_contacts - GocDB needs to republish these attributes in its PI for use by the rest of the infrastructure (e.g. queried by Ops portal, used in accounting and so on).

Now, according to the rules of the UK Access Fed (similar rules apply in other feds), “The Service Provider must not disclose to third parties any Attributes other than to any data processor of the service provider or where the relevant end user has give its prior informed consent to such disclosure. see [note2]” . (Where [note2] states: “The basic Rule is that attributes may only be used by the service requested by the user and only for the specified purposes. Service Providers that wish to use attributes in other ways (for example to provide direct user support) can arrange this either by obtaining positive informed consent from each individual End User, or by contract with Identity Providers who are then responsible for informing their End User.”)

Therefore, to use Attributes like the eppn in EGI we need to obtain ‘positive informed consent’ from the user that we can publish and make visible their data/eppn to authenticated users and client-services of GOCDB, including those trusted by UK Access Management Federation, EGI-SSO or IGTF (I think we must explicitly state the different authentication/trust realms, e.g. any person/host with a valid IGTF cert can query the GocDB API for this info).

  • Q. Any experts out there know if the sample notification/approval-dialog shown in the screen grab below provides adequate positive consent?
  • Q. Assuming user clicks OK, do you know if there are any issues for EGI and for the hosting institution (STFC) in re-publishing the eppn in the PI? (it then becomes readable to any other service/user who has a valid IGTF cert)
    • Update 20/08/2015: Re-publishing the eppn looks like a no-go: The SP needs to first take account of each/every IdP's policy on whether the eppn can be reused. This is does not scale. However, hashing the ePPN or using the ePUID may be ok (see below).

Using ePTID without ePPN (not suitable)

GocDB could use the ‘eduPersonTargetedID’ (eptid) and re-publish that in the PI – the eptid is less of a security concern because its anonymous/opaque. However, since the eptid will be different for the same user across different SAML Service Providers, this provides little use when there is requirement to identify a particular accountID across different SPs and organisations (this is the EGI requirement). Note, if EGI had a single centralised/proxy IdP, it could be used to re-issue the same eptid consistently to all the underlying EGI services/systems. I think this would be far preferable compared to services individually registering with federations and receiving different eptids per service.

  • Update 20/08/2015, using the eptid is not suitable as it does not match the EGI requirement

Proposed Solution Requires a) and b)

a) Hashing the ePPN using a project-owned secret salt; a way to securely re-publish and correlate an ID dervied from the ePPN within a project

A group of related Service Providers, e.g. those belonging to a particular project like EGI, may concatenate a secret ‘salt’ value to the value of an attribute before hashing the result, for example; hash(‘secretSaltValue + eduPersonPrincipleName’). In doing this, the same hash value can only be reproduced if: a) an SP has access to the same secret salt value and, b) the SPs agree to use the same hashing algorithm. If the salted and hashed attribute is the ePPN, this produces an opaque and persistent identifier that can only be reproduced and correlated across the SPs within the project (i.e. those that have secure access to the salt). Provided the salt value is not disclosed to any third parties, the resulting hash string can be safely republished, for example, for use in correlating account IDs across project-specific SPs in the same project.

b) Requesting both ePPN and ePTID to determine ePPN re-allocation

The ePPN is re-assignable, and re-assignment practices depend on the federation/country; you cannot always rely on a federation re-assigning an ePPN after an agreed period of inactivity (e.g. 2yrs with the UKAMF). Therefore, as suggested by the R&S REFEDS SP Category, its good practice to also request/release both ePPN and ePTID together. This is useful because the SP can also record an ePTID change (by definition the ePTID can never be re-assigned). By doing this, the SP knows that the holder of the ePPN has changed. Note, the ePTID would only be used internally by a particular SP to check that the user has not changed; it wouldn't be used when hashing the ePPN with a secret salt for use in republishing as described above.

  • Q. I think this could be used by SPs within EGI allowing accounts to be safely correlated across SPs.

I’d be grateful for any thoughts/guidance on these matters (many thanks to Mikael Linden and Andrew Cormack for their help).




v5.5 Release

  • Test instance: https://gocdb-test.esc.rl.ac.uk/portal
  • Release date was Wed 2nd Dec.
  • The 'get_site_security_info' method will be moved from 'private,' which requires the DN of each calling client to be manually registered in an ACL (access control list), to 'protected,' which allows anyone with a valid IGTF certificate to query the method. The only info regarded as potentially sensitive is each NGI's CSIRT email and Tel. The reasons why this method was originally made private were a) because it collates all the contacts into a single query and so was largely a spam concern, b) inherited legacy - it's simply always been this way. I also understand that the original intention was that the generic CSIRT email addresses/Tels were never intended to be restricted.
  • Adding <SCOPES/> element to the XML output of the following PI queries: get_site, get_service, get_service_endpoint, get_ngi, get_service_group
  • Adding new scope tags for WLCG VOs, including ‘wlcg, atlas, cms, lhcb, alice, tier1, tier2’ for finer-grained resource filtering.
  • Large scale refactoring of the role model to cater for multi-tenant (multiple projects) and to more easily add/refine new roles and actions/permissions per-project. The main change is that roles apply to a particular project – this is shown in the output of the ‘get_user’ method that qualifies which user-roles apply to which project using the ‘<RECOGNISED_IN_PROJECTS>’ element.

    <EGEE_USER ID="1484G0" PRIMARY_KEY="1484G0">
	<TEL>+44 1925 603762</TEL>
	<CERTDN>/C=UK/O=eScience/OU=CLRC/L=DL/CN=david meredith</CERTDN>
	    <USER_ROLE>Service Group Administrator</USER_ROLE>
	    <RECOGNISED_IN_PROJECTS/>            <!-- ***new element (ServiceGroup role is Project agnostic)*** -->
	    <USER_ROLE>Chief Operations Officer</USER_ROLE>
	    <RECOGNISED_IN_PROJECTS>                    <!-- ***new element*** -->
		<PROJECT ID="1505">StandaloneProjectX</PROJECT>
	    <USER_ROLE>Site Operations Manager</USER_ROLE>
	    <RECOGNISED_IN_PROJECTS>                  <!-- ***new element*** -->