Difference between revisions of "GOCDB/PI/Technical Documentation"
(43 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
{{Template:Op menubar}} | {{Template:Op menubar}} | ||
{{Template: | {{Template:GOCDB_menubar}} | ||
{{TOC_top}} | |||
[[Category:GOCDB]] | |||
== GOCDB Programmatic Interface == | == GOCDB Programmatic Interface == | ||
Line 8: | Line 8: | ||
==== Releases, changes and announcements ==== | ==== Releases, changes and announcements ==== | ||
All client tool using GOCDB should subscribe to the GOCDB discussion mailing list: [mailto:gocdb-discuss@mailman.egi.eu gocdb-discuss_at_mailman.egi.eu]. Releases, new features, changes in methods and general announcements related to this interface are made on this list. | |||
If your application is using the GOCDB Programmatic interface please [https://mailman.egi.eu/mailman/listinfo/gocdb-discuss subscribe to gocdb-discuss] or send your subscription request by mail to [mailto:gocdb-admins@mailman.egi.eu gocdb-admins_at_mailman.egi.eu]. | |||
=== Interface description === | === Interface description === | ||
Line 24: | Line 20: | ||
==== Entry point ==== | ==== Entry point ==== | ||
Enter GOCDB here https://goc.egi.eu/ or directly at https://goc.egi.eu/portal/ to browse the global EGI information | Enter GOCDB here https://goc.egi.eu/ or directly at https://goc.egi.eu/portal/ to browse and edit the global EGI information. | ||
Entry point for every single method is given on method definition pages listed below. | |||
Entry point for every single method is given on method definition pages listed below. | |||
==== Outputs ==== | ==== Outputs ==== | ||
Output format is '''XML'''. Calling any implemented method will return you a valid DOM document that can be parsed by any tool on client side. | Output format is '''XML'''. Calling any implemented method will return you a valid DOM document that can be parsed by any tool on client side. | ||
<!--- Removed from V5 | |||
==== Schemas ==== | ==== Schemas ==== | ||
The associated '''XML schema''' for each method is available by calling the method with "&output=xmlschema", e.g. https://goc.egi.eu/gocdbpi/private/?method=get_site&output=xmlschema | The associated '''XML schema''' for each method is available by calling the method with "&output=xmlschema", e.g. https://goc.egi.eu/gocdbpi/private/?method=get_site&output=xmlschema | ||
--> | |||
=== Data protection and access === | === Data protection and access === | ||
Line 42: | Line 37: | ||
There are currently 3 protection levels for all methods of the interface | There are currently 3 protection levels for all methods of the interface | ||
* '''Level 1''': public methods (no critical information, no mail | * '''Level 1''': public methods (no critical information, no personal mail/details) | ||
* '''Level 2''': protected methods (contain mails or personal details) | * '''Level 2''': protected methods (contain mails or personal details) | ||
* '''Level 3''': private methods (contains security or critical information) | * '''Level 3''': private methods (contains security or critical information) | ||
Line 50: | Line 45: | ||
==== How to gain access? ==== | ==== How to gain access? ==== | ||
If you need to access protected methods, a valid certificate will be enough. If you need to access private methods, please send a mail to [mailto:gocdb- | If you need to access protected methods, a valid certificate will be enough. If you need to access private methods, please send a mail to [mailto:gocdb-admins@mailman.egi.eu gocdb-admins_at_mailman.egi.eu], indicating: | ||
* Your name and affiliation | * Your name and affiliation | ||
Line 63: | Line 58: | ||
https://goc.egi.eu/gocdbpi/private/?method=get_site&scope=Local | https://goc.egi.eu/gocdbpi/private/?method=get_site&scope=Local | ||
Our user facing documentation for this feature is available at [[GOCDB/ | Our user facing documentation for this feature is available at [[GOCDB/Input_System_User_Documentation#Data_Visibility_.2F_Scopes]]. | ||
=== Available methods === | === Available methods === | ||
{| | {| {{egi-table}} | ||
|- | |- | ||
! Method name | ! Method name | ||
Line 76: | Line 71: | ||
| [[GOCDB/PI/get site method|get_site]] | | [[GOCDB/PI/get site method|get_site]] | ||
| read - generic | | read - generic | ||
| Returns site information | | Returns site information, grouped by site | ||
| 2 | | 2 | ||
|- | |- | ||
Line 92: | Line 87: | ||
| read - generic | | read - generic | ||
| Returns security contact information for sites | | Returns security contact information for sites | ||
| | | 2 | ||
|- | |- | ||
| [[GOCDB/PI/get roc list method|get_roc_list]] | | [[GOCDB/PI/get roc list method|get_roc_list]] | ||
Line 99: | Line 94: | ||
| 1 | | 1 | ||
|- | |- | ||
| [[GOCDB/PI/get subgrid list method|get_subgrid_list]] | | <strike>[[GOCDB/PI/get subgrid list method|get_subgrid_list]]</strike> | ||
| read - generic | | read - generic | ||
| Returns a list of Subgrids (i.e. registered sub-parts of an NGI) with minimal associated information | | <strike>Returns a list of Subgrids (i.e. registered sub-parts of an NGI) with minimal associated information</strike> | ||
| 1 | | 1 | ||
|- | |- | ||
Line 109: | Line 104: | ||
| 2 | | 2 | ||
|- | |- | ||
| [[GOCDB/PI/get egee contacts method|get_egee_contacts]] | | <strike>[[GOCDB/PI/get egee contacts method|get_egee_contacts]]</strike> | ||
| read - generic | | read - generic | ||
| Returns a list of contacts for staff that have a role a EGI level | | <strike>Returns a list of contacts for staff that have a role a EGI level</strike> | ||
| 2 | | 2 | ||
|- | |- | ||
| [[GOCDB/PI/get downtime method|get_downtime]] | | [[GOCDB/PI/get downtime method|get_downtime]] | ||
| read - generic | | read - generic | ||
| Returns a list of | | Returns a list of service downtimes (downtimes are repeated for each affected service) | ||
| 1 | |||
|- | |||
| [[GOCDB/PI/get downtime nested services method|get_downtime_nested_services]] | |||
| read - generic | |||
| Returns a list of downtimes with affected services nested as child elements of the downtime | |||
| 1 | | 1 | ||
|- | |- | ||
| [[GOCDB/PI/get service endpoint method|get_service_endpoint]] | | [[GOCDB/PI/get service endpoint method|get_service_endpoint]] | ||
| read - generic | | read - generic | ||
| Returns a list of service | | Returns a list of services (single node x single service) and associated information (same as get_service) | ||
| 1 | |||
|- | |||
| [[GOCDB/PI/get service endpoint method|get_service]] | |||
| read - generic | |||
| Returns a list of services (single node x single service) and associated information (same as get_service_endpoint) | |||
| 1 | | 1 | ||
|- | |- | ||
Line 137: | Line 142: | ||
| read - dedicated | | read - dedicated | ||
| Returns the list of downtimes recently declared with notification settings for CIC portal downtime notification service | | Returns the list of downtimes recently declared with notification settings for CIC portal downtime notification service | ||
| | | 1 | ||
|- | |- | ||
| [[GOCDB/PI/get cert status changes|get_cert_status_changes]] | | [[GOCDB/PI/get cert status changes|get_cert_status_changes]] | ||
Line 154: | Line 159: | ||
| 2 | | 2 | ||
|- | |- | ||
| [[GOCDB/PI/ | | [[GOCDB/PI/get service group role|get_service_group_role]] | ||
| read - generic | | read - generic | ||
| Returns a list of service groups and the roles held by users over these groups. | | Returns a list of service groups and the roles held by users over these groups. | ||
| 2 | | 2 | ||
|- | |||
| [[GOCDB/PI/get ngi|get_ngi]] | |||
| read - generic | |||
| Returns a list of NGIs with contact information | |||
| 2 | |||
|- | |||
| [[GOCDB/PI/get_project_contacts|get_project_contacts]] | |||
| read - generic | |||
| Returns a list of persons (and associated info) having a role over a project. | |||
| 2 | |||
|- | |||
| [[GOCDB/PI/get_site_count_per_country|get_site_count_per_country]] | |||
| read - generic | |||
| Returns a count of sites per country (not per NGI) | |||
| 1 | |||
|} | |} | ||
'''Note about parameters:''' | '''Note about parameters:''' | ||
*Unless otherwise stated, all parameters are optional. | *Unless otherwise stated, all parameters are optional. | ||
*Unless otherwise state, all parameters are <b>case sensitive</b>. | |||
*It is possible to combine parameters, for example: ''?method=get_site&country=Italy&certification_status=Certified&production_status=Production'' | *It is possible to combine parameters, for example: ''?method=get_site&country=Italy&certification_status=Certified&production_status=Production'' | ||
=== Client code examples === | === Client code examples === | ||
==== Simple data retrieval using wget ==== | ==== Simple data retrieval using wget/curl ==== | ||
If getting the data locally is what you're after, a simple cron job retrieving data via wget might do the trick. | If getting the data locally is what you're after, a simple cron job retrieving data via wget might do the trick. | ||
Line 175: | Line 196: | ||
$ wget --no-check-certificate -O result.xml <nowiki>'https://goc.egi.eu/gocdbpi/public/?method=get_site_list&sitename=RAL-LCG2'</nowiki> | $ wget --no-check-certificate -O result.xml <nowiki>'https://goc.egi.eu/gocdbpi/public/?method=get_site_list&sitename=RAL-LCG2'</nowiki> | ||
''--no-check-certificate'' option | ''--no-check-certificate'' option (like the -k given for curl below) bypasses server certificate validation; this is handy if you don't have access to a directory containing the appropriate CA certificates, but is not generally recommended; providing the CA directory as in the example below provides the information it needs for such validation. Note also that the URL must be encapsulated in single quotes to protect the ampersand from the shell. | ||
* Protected and private methods: Here you will need authentication, which can be done through wget as shown in next example: | * Protected and private methods: Here you will need authentication, which can be done through wget as shown in next example: | ||
Line 181: | Line 202: | ||
$ wget --certificate=cert.pem --private-key=cert.key.pem --private-key-type=PEM --ca-directory=/etc/grid-security/certificates/ -O result.xml <nowiki>'https://goc.egi.eu/gocdbpi/private/?method=get_site&sitename=RAL-LCG2'</nowiki> | $ wget --certificate=cert.pem --private-key=cert.key.pem --private-key-type=PEM --ca-directory=/etc/grid-security/certificates/ -O result.xml <nowiki>'https://goc.egi.eu/gocdbpi/private/?method=get_site&sitename=RAL-LCG2'</nowiki> | ||
Note that if you don't specify the --ca-directory option | Note that if you don't specify the --ca-directory option and you haven't asked for --no-check-certificate then the SSL handshake will fail as the server cert check can't be performed. | ||
'''Note:''' examples above are shown using wget-10.2. Some discrepencies are observed between version so check wget help for correct options. Bugs and weird behaviour have also been observed for wget with a version older than 10 and using wget on cygwin. | |||
$ curl -k --cert ./cert.pem --key ./key.pem <nowiki>'https://goc.egi.eu/gocdbpi/public/?method=get_site&sitename=RAL-LCG2'</nowiki> | |||
'''Note:''' curl on SL6 uses NSS rather than openssl and this can cause some issues. One problem is that NSS would think "cert.pem" was an alias so you have to turn it into a relative path hence these examples use "./" in front of each filename. Another problem is that in the example above a passphrase isn't always requested for key.pem so one has to be passed in using the --pass option. One method of doing this without leaving your passphrase in your command line history file is as follows: | |||
$ curl -k --cert ./cert.pem --key ./key.pem --pass $(echo -n "enter passphrase: ">&2 ; read -s a ; echo $a) <nowiki>'https://goc.egi.eu/gocdbpi/public/?method=get_site&sitename=RAL-LCG2'</nowiki> | |||
==== Perl example ==== | ==== Perl example ==== | ||
Line 217: | Line 244: | ||
* [https://documents.egi.eu/secure/RetrieveFile?docid=471&version=2&filename=Get_downtimes_python_example.tar.gz Get_downtimes_python_example.tar.gz] | * [https://documents.egi.eu/secure/RetrieveFile?docid=471&version=2&filename=Get_downtimes_python_example.tar.gz Get_downtimes_python_example.tar.gz] | ||
==== Java example ==== | |||
Provided by David Meredith, STFC | |||
<Source lang="java"> | |||
import java.io.File; | |||
import java.net.URL; | |||
import javax.net.ssl.HttpsURLConnection; | |||
import javax.net.ssl.SSLContext; | |||
import javax.net.ssl.TrustManager; | |||
import javax.net.ssl.X509TrustManager; | |||
import org.junit.Test; | |||
public class DownloadFileSample { | |||
// trust all certificates (don't do this is production) | |||
TrustManager[] trustAllCerts = new TrustManager[]{ | |||
new X509TrustManager() { | |||
public java.security.cert.X509Certificate[] getAcceptedIssuers() { | |||
return null; | |||
} | |||
public void checkClientTrusted( | |||
java.security.cert.X509Certificate[] certs, String authType) { | |||
} | |||
public void checkServerTrusted( | |||
java.security.cert.X509Certificate[] certs, String authType) { | |||
} | |||
} | |||
}; | |||
@Test | |||
public void testIt() throws Exception{ | |||
SSLContext sc = SSLContext.getInstance("SSL"); | |||
sc.init(null, trustAllCerts, new java.security.SecureRandom()); | |||
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); | |||
URL url = new URL("https://goc.egi.eu/gocdbpi/public/?method=get_site_list"); | |||
File df = File.createTempFile("sampleSiteList", ".xml"); | |||
org.apache.commons.io.FileUtils.copyURLToFile(url, df); | |||
System.out.println("downloadFile: "+df.getAbsolutePath()); | |||
String version = System.getProperty("java.version"); | |||
System.out.println("Using Java version: "+version); | |||
} | |||
} | |||
</Source> | |||
=== New requests and improvements === | === New requests and improvements === |
Revision as of 19:31, 12 February 2016
Main | EGI.eu operations services | Support | Documentation | Tools | Activities | Performance | Technology | Catch-all Services | Resource Allocation | Security |
GOC DB menu: | Home • | Documentation Index • |
GOCDB Programmatic Interface
Important information about using the GOCDBPI
Releases, changes and announcements
All client tool using GOCDB should subscribe to the GOCDB discussion mailing list: gocdb-discuss_at_mailman.egi.eu. Releases, new features, changes in methods and general announcements related to this interface are made on this list.
If your application is using the GOCDB Programmatic interface please subscribe to gocdb-discuss or send your subscription request by mail to gocdb-admins_at_mailman.egi.eu.
Interface description
Interface type
GOCDB Programmatic Interface is a REST (Representational State Transfer) based interface over https. The use of https guarantees URLs are properly secured when transiting. Some of the methods are nonetheless public and don't require client side authentication (see #Data protection and access).
Entry point
Enter GOCDB here https://goc.egi.eu/ or directly at https://goc.egi.eu/portal/ to browse and edit the global EGI information. Entry point for every single method is given on method definition pages listed below.
Outputs
Output format is XML. Calling any implemented method will return you a valid DOM document that can be parsed by any tool on client side.
Data protection and access
Protection levels
There are currently 3 protection levels for all methods of the interface
- Level 1: public methods (no critical information, no personal mail/details)
- Level 2: protected methods (contain mails or personal details)
- Level 3: private methods (contains security or critical information)
Methods at level 1 are available without restrictions. Level 2 requires client to present a valid recognized X509 certificate (reference CA list is the officially agreed EGI/LCG EUGridPMA list). Level 3 requires client to present a known, registered certificate (certificate DN needs to be stored on GOCDB side for authentication)
How to gain access?
If you need to access protected methods, a valid certificate will be enough. If you need to access private methods, please send a mail to gocdb-admins_at_mailman.egi.eu, indicating:
- Your name and affiliation
- The name and purpose of the tool that need to access the data
- certificate DN of the machine(s) that will retrieve the information
- An e-mail contact for your application, that will be subscribed to the GOCDB-discussion and announcement mailing list if it is not already there.
Data Visiblity
Sites and service endpoints can be shown or hidden from EGI and this affects what data is shown through the PI. By default the PI methods will return only objects that are marked as visible to EGI. For example get_site will, by default, return only sites that are visible to EGI. Similarly, get_downtime will only show downtimes that are linked to EGI service endpoints.
The "scope=Local" parameter can be used to access non-EGI data through the PI. For example: https://goc.egi.eu/gocdbpi/private/?method=get_site&scope=Local
Our user facing documentation for this feature is available at GOCDB/Input_System_User_Documentation#Data_Visibility_.2F_Scopes.
Available methods
Method name | type | description | Protection level |
---|---|---|---|
get_site | read - generic | Returns site information, grouped by site | 2 |
get_site_list | read - generic | Returns a list of sites with minimal associated information | 1 |
get_site_contacts | read - generic | Returns a list of persons (and associated info) having a role at site level, grouped per site | 2 |
get_site_security_info | read - generic | Returns security contact information for sites | 2 |
get_roc_list | read - generic | Returns a list of NGIs with minimal associated information | 1 |
read - generic | 1 | ||
get_roc_contacts | read - generic | Returns NGI contact details, including NGI contact mail address and list of NGI staff | 2 |
read - generic | 2 | ||
get_downtime | read - generic | Returns a list of service downtimes (downtimes are repeated for each affected service) | 1 |
get_downtime_nested_services | read - generic | Returns a list of downtimes with affected services nested as child elements of the downtime | 1 |
get_service_endpoint | read - generic | Returns a list of services (single node x single service) and associated information (same as get_service) | 1 |
get_service | read - generic | Returns a list of services (single node x single service) and associated information (same as get_service_endpoint) | 1 |
get_service_types | read - generic | Returns a list of valid service types and associated description | 1 |
get_user | read - generic | Returns a user or a list of users with associated details and roles | 2 |
get_downtime_to_broadcast | read - dedicated | Returns the list of downtimes recently declared with notification settings for CIC portal downtime notification service | 1 |
get_cert_status_changes | read - generic | Returns a list of changes to certification statuses | 2 |
get_cert_status_date | read - generic | Returns a list of current certification statuses for Production sites and the date they entered that status | 2 |
get_service_group | read - generic | Returns a list of service groups and the service endpoints under those groups. | 2 |
get_service_group_role | read - generic | Returns a list of service groups and the roles held by users over these groups. | 2 |
get_ngi | read - generic | Returns a list of NGIs with contact information | 2 |
get_project_contacts | read - generic | Returns a list of persons (and associated info) having a role over a project. | 2 |
get_site_count_per_country | read - generic | Returns a count of sites per country (not per NGI) | 1 |
Note about parameters:
- Unless otherwise stated, all parameters are optional.
- Unless otherwise state, all parameters are case sensitive.
- It is possible to combine parameters, for example: ?method=get_site&country=Italy&certification_status=Certified&production_status=Production
Client code examples
Simple data retrieval using wget/curl
If getting the data locally is what you're after, a simple cron job retrieving data via wget might do the trick.
- Public methods: You do not need any client certificate, so a simple wget instruction should work:
$ wget --no-check-certificate -O result.xml 'https://goc.egi.eu/gocdbpi/public/?method=get_site_list&sitename=RAL-LCG2'
--no-check-certificate option (like the -k given for curl below) bypasses server certificate validation; this is handy if you don't have access to a directory containing the appropriate CA certificates, but is not generally recommended; providing the CA directory as in the example below provides the information it needs for such validation. Note also that the URL must be encapsulated in single quotes to protect the ampersand from the shell.
- Protected and private methods: Here you will need authentication, which can be done through wget as shown in next example:
$ wget --certificate=cert.pem --private-key=cert.key.pem --private-key-type=PEM --ca-directory=/etc/grid-security/certificates/ -O result.xml 'https://goc.egi.eu/gocdbpi/private/?method=get_site&sitename=RAL-LCG2'
Note that if you don't specify the --ca-directory option and you haven't asked for --no-check-certificate then the SSL handshake will fail as the server cert check can't be performed.
Note: examples above are shown using wget-10.2. Some discrepencies are observed between version so check wget help for correct options. Bugs and weird behaviour have also been observed for wget with a version older than 10 and using wget on cygwin.
$ curl -k --cert ./cert.pem --key ./key.pem 'https://goc.egi.eu/gocdbpi/public/?method=get_site&sitename=RAL-LCG2'
Note: curl on SL6 uses NSS rather than openssl and this can cause some issues. One problem is that NSS would think "cert.pem" was an alias so you have to turn it into a relative path hence these examples use "./" in front of each filename. Another problem is that in the example above a passphrase isn't always requested for key.pem so one has to be passed in using the --pass option. One method of doing this without leaving your passphrase in your command line history file is as follows:
$ curl -k --cert ./cert.pem --key ./key.pem --pass $(echo -n "enter passphrase: ">&2 ; read -s a ; echo $a) 'https://goc.egi.eu/gocdbpi/public/?method=get_site&sitename=RAL-LCG2'
Perl example
Provided by Emir Imamagic, SRCE. This code is used by NCG
The 3 following examples are GOCDB modules in perl which use LWP::UserAgent for accessing PI and XML::DOM for parsing the data. Data is used to fill internal NCG structure.
Nagios plugin nagios-gocdb-downtime is used for retrieving downtimes and pushing them to Nagios via GridMon::Nagios module:
note Full code is available from cern svn
PHP example
Provided by Guillaume Cessieux, CC-IN2P3. This code is used by ENOC
The following example shows a simple PHP class with methods used to retrieve and parse XML data from GOCDB-PI using PHP curl and simplexml
Python example
Provided by Giuseppe Misurelli, CNAF-INFN
The following example is a Python client that queries the GOCDB programmatic interface, retrieves info about downtimes for a top entity region using curl, parses xml output with the xml.dom.minidom module and stores data into a MySQl db.
Java example
Provided by David Meredith, STFC
import java.io.File;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.junit.Test;
public class DownloadFileSample {
// trust all certificates (don't do this is production)
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
}
};
@Test
public void testIt() throws Exception{
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
URL url = new URL("https://goc.egi.eu/gocdbpi/public/?method=get_site_list");
File df = File.createTempFile("sampleSiteList", ".xml");
org.apache.commons.io.FileUtils.copyURLToFile(url, df);
System.out.println("downloadFile: "+df.getAbsolutePath());
String version = System.getProperty("java.version");
System.out.println("Using Java version: "+version);
}
}
New requests and improvements
Requests on GOCDB-PI can take the form of:
- Change request to an existing method
- Request for new generic methods
- Request for tool specific methods
Such requests are logged in The GOCDB RT Requirements Queue Tickets: View and submit RT Requirements tickets specifically for GOCDB