GOCDB/v4/Regional Module Technical Documentation

From EGIWiki
Jump to: navigation, search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


This page is the main technical documentation for GOCDB-4 regional module

Download

LATEST VERSION: GOCDB-4.4

svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.4 gocdb-4.4
svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb gocdbTrunk

Deployment

System prerequisites

GOCDB-4 regional module has two components: a database and a web front end. Both components can work on the same machine but it is very likely that in most environments they will be separated. This is what we would advise anyway. Machine requirements for these 2 components are described below.

Database

  • Required database: Oracle 10g or higher (note: Oracle 10g and 11g XE Express Editions, which come with free licenses are perfectly suitable)
  • Required space: 150 MB
  • Dowload free Oracle XE from the following link and follow the installation instructions: http://www.oracle.com/technetwork/database/express-edition/overview/index.html
  • For gocdb admin tasks, we also recommend the SQL Developer tool.

Web frontend

The machine to use as the web frontend will need the following:

  • Apache http server version 2.2 or higher.
  • PHP version 5.3 or higher (version <=5.2 has some OOP related bugs).
    • PHP oci8 extension
    • libxml2 and DOM support for PHP (Note: On RHEL, PHP requires the PHP XML RPM to be installed for this component to function).
    • OpenSSL Extension for PHP
    • Query2XML from PEAR
    • MDB2 from PEAR
    • OCI8 MDB2 Driver from PEAR
  • Oracle Instant client version 10 or higher (downloadable from Oracle website)
  • a X509 host certificate for the machine

Note, you can check you have installed pear and the required pear packages using:

$pear version
PEAR Version: 1.4.9
PHP Version: 5.3.3

$pear list
Installed packages, channel pear.php.net:
=========================================
Package          Version State
Archive_Tar      1.3.1   stable
Console_Getopt   1.2     stable
MDB2             2.4.1   stable
MDB2_Driver_oci8 1.4.1   stable
PEAR             1.4.9   stable
XML_Query2XML    1.7.2   stable
XML_RPC          1.5.0   stable

Preparing your installation

Database preparation

Important note: This section needs to be executed by Database System Administrators.

  • If you don't have admin rights on the DB you are using, please contact your DBAs with the instructions contained in this section.
1- Create Custom tablespaces for GOCDB (Optional)

GOCDB-4 database contains 3 distinct groups of tables: Metadata or core data, GOCDB data, and local data. Although this is not mandatory, we recommend that 3 different tablespaces are created to store these different tables.

  • If you prefer, you can skip this step and rely on the default 'USERS' tablespace.
Sugested Tablespace Name Min Recommended Size Purpose
TS_GRIDCORE 50m Will store the core tables containing schema metadata
TS_GOCDBDATA 50m Will store GOCDB data tables (common structure to all GOCDB modules)
TS_LOCALDATA 50m Will store your local tables if you have some local information to store

Note: You must configure gocdb to use these custom tablespace names in the 'local_info.xml' config file as described below.

To create these optional tablespaces, run the following script (as SYSTEM user):

 CREATE TABLESPACE TS_GRIDCORE; 
 CREATE TABLESPACE TS_GOCDBDATA; 
 CREATE TABLESPACE TS_LOCALDATA; 

The created tablespaces will then be:

  • Permanent, locally managed and with system allocated extent size.
  • The corresponding datafiles will be created in the location provided in the DB_CREATE_FILE_DEST parameter and with size 100 MB.
  • The datafiles will be autoextensible with no maximum size.
  • Names of datafiles will be similar to "ora_applicat_zxyykpt000.dbf"

You can customise tablespace creation, there are many possible options (see Oracle docs for further info). For example:

 CREATE TABLESPACE <tblspc_name> DATAFILE 'C:\ORA\<fileNameForTableSpace>.dbf' SIZE 50M AUTOEXTEND ON MAXSIZE 100M;
2- Create a DB user for GOCDB

We advise that you create a dedicated GOCDB Oracle user. You can create the gocdb user with the following script (substitute <PASSWORD> with a sensible password and run as system user):

-- USER SQL
CREATE USER GOCDB IDENTIFIED BY <PASSWORD> 
DEFAULT TABLESPACE "USERS"
TEMPORARY TABLESPACE "TEMP";
  • If you created the three custom tablespaces as described above, you will need to grant access and unlimited quotas to your GOCDB user on those previously defined tablespaces (you do not need to do this if you are relying on the default 'USERS' tablespace).
ALTER USER GOCDB QUOTA UNLIMITED ON TS_GRIDCORE;
ALTER USER GOCDB QUOTA UNLIMITED ON TS_GOCDBDATA;
ALTER USER GOCDB QUOTA UNLIMITED ON TS_LOCALDATA;
3- Manage user privileges

Grant privileges to your GOCDB user:

This can be done by executing the following script (run as SYSTEM user):

-- ROLES
GRANT "RESOURCE" TO GOCDB ;

-- SYSTEM PRIVILEGES
GRANT CREATE TRIGGER TO GOCDB ;
GRANT CREATE SEQUENCE TO GOCDB ;
GRANT CREATE TABLE TO GOCDB ;
GRANT CREATE JOB TO GOCDB ;
GRANT CREATE PROCEDURE TO GOCDB ;
GRANT CREATE TYPE TO GOCDB ;
GRANT CREATE SESSION TO GOCDB ;
GRANT CREATE MATERIALIZED VIEW TO GOCDB ; 

(note, if you are using the free Oracle XE, and depending on the version, the 'CREATE JOB' system privilege is not listed as an option in the 'Manage Database Users' part of the Oracle web-application interface).

4- Manage the GOCDB password expiry

By default, Oracle 11g will expire a password in 180 days. In previous versions of Oracle, the default policy was UNLIMITED, so please be aware of this change! As a system user, you can see your password expiry settings by looking at the PASSWORD_LIFE_TIME and PASSWORD_GRACE_TIME parameters in the DBA_PROFILES table:

-- select the profile for the GOCDB user (e.g. will return DEFAULT)
SELECT profile FROM dba_users WHERE username = 'GOCDB'; 

-- select the password expiry settings for the profile assigned to the GOCDB user
select resource_name,resource_type, limit from dba_profiles where profile=DEFAULT; 


If you prefer, you can update the default expiry from 180days to UNLIMITED using the following (assumng GOCDB user profile is DEFAULT):

-- requires system privilege
ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;

Web machine preparation

1- install host certificate for the machine
  • GOCDB web portal component needs the machine to run a https server which requires a server X509 certificate. Please check that Apache is properly configured to use the installed server certificate. Help about how to set up SSL configuration on Apache can be obtained from Apache mod_ssl online documentation.
  • For development and testing, you can use a self signed certificate (this must not be used in production!). These pages show you how to create a self signed cert for testing purposes:
  • Note, GOCDB uses trusted CA certificates to verify user certificates. You can dowload the EGI/IGTF CA certificate bundle from the following link: EGI_IGTF_Release
  • Note, CA certificates must be named differently according to the version of OpenSSL you are using (i.e. v1.0.X or v0.9.X). This is explained in more detail at the previous link.
2- Get and deploy GOCDB-4 code package
  • See #Download above
  • Move the code to a suitable location; this will be the root of your gocdb installation.
  • Verify the user running httpd has read access to the directories/files. The files are often not accessible by the Apache user by default. (Default installation location: /usr/share/GOCDB)
  • Create the following Apache Aliases (or set up virtual hosts)
    • Alias "/portal" "/usr/share/GOCDB/htdocs/web_portal"
    • Alias "/admin" "/usr/share/GOCDB/htdocs/admin"
    • Alias "/xml_input" "/usr/share/GOCDB/htdocs/xml_input"
    • Alias "/gocdbpi/public" "/usr/share/GOCDB/htdocs/PI/public"
    • Alias "/gocdbpi/private" "/usr/share/GOCDB/htdocs/PI/private"
  • Configure Apache to use SSL and client certificate verification for the web portal section of the site. The user certificate DN's are used for authentication to the web front end.
  • Ensure the web server port is open on your machine. As a first step, you probably want the port to be open internally only (giving the whole wide world access to your GOCDB admin page is probably not a good idea)
  • Restart Apache so that the new aliases are loaded from the configuration file.
  • Check that the pages can be correctly accessed by browsing to the newly created "admin" alias. Note that the system is not yet fully set up and will not work!

Note: Creating and managing http aliases is part of Apache http server handling. Please check Apache documentation if you need details. An example virtual host configuration file for GOCDB GOCDB/Sample GOCDB Virtual Host Config file

3- Create and edit prom.conf configuration file
  • Rename or copy /usr/share/GOCDB/config/prom.conf_TEMPLATE to /usr/share/GOCDB/config/prom.conf
  • Edit prom.conf. Under the [Oracle_server] section, specify the connection details to your DB. "host" can be a simple hostname or a full connection string (see examples below).

All settings must be enclosed in quotation marks.

example 1: Specifying a simple hostname (you will need to use this style if you are using the free Oracle XE)

host="dbhostname.domain.org"
port="1521"
database="XE" 
user="GOCDB"
password="gocDBpassWord"

example 2: Specifying a connection string for an Oracle cluster (warning: the whole string has to be on one single line)

host="(DESCRIPTION=(LOAD_BALANCE=yes)(FAILOVER=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=dbhostname1.domain.org)\\
 (PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = dbhostname2.domain.org)(PORT = 1521))\\
 (CONNECT_DATA=(SERVICE_NAME=gocdb.domain.com)(FAILOVER_MODE=(TYPE=SESSION)(METHOD=BASIC))))"
port=
database= 
user="GOCDB"
password="gocDBpassWord"
4- Create and edit local_info.xml configuration file
  • Rename or copy /usr/share/GOCDB/config/local_info.xml_TEMPLATE to /usr/share/GOCDB/config/local_info.xml
  • Edit local_info.xml whith the following information:
    • tablespace names, as configured in the previous section (if you are relying on the default 'USERS' tablespace, then specify 'USERS' for each of the three elements)
    • Collection ID, which is a unique ID that will eventually be used to distinguish different GOCDB regional instances. Currently this should be left as 0.
    • instance type, which can be either "central" or "regional". This should be set to 'regional' for a read/write GOCDB or 'central' for a read only instance.
    • web_portal_url, which is the full url of your GOCDB web portal as defined during the setup process above
5- Create and edit local_schema.xml configuration file
  • Rename or copy /usr/share/GOCDB/config/local_schema.xml_TEMPLATE to /usr/share/GOCDB/config/local_schema.xml
  • You don't need to edit/modify this file now. How to use this file (and why) will be described in the Customisation section

Final check

If you have carefully followed all the previous steps, you should now have:

  • An Oracle database where GOCDB will run, with connection details and proper privileges
  • A web machine that will host GOCDB front end
  • On the web machine, the following configuration files appropriately edited:
    • /usr/share/GOCDB/config/prom.conf, with proper database connection details
    • /usr/share/GOCDB/config/local_info.xml, with proper tablespace information and your assigned collection id
    • /usr/share/GOCDB/config/local_schema.xml, left as it is for now (content doesn't matter for the setup process as long as the file is here)
  • An alias to GOCDB admin web page, available through your browser.
  • An alias to GOCDB web front end, available through your browser.
  • An alias to GOCDB xml input, available through your browser.

And now the moment of truth...

  • Open GOCDB admin web page in your browser as shown below (the alias you have defined above: "/admin")
  • On the welcome page, click on the configuration check link
    • If it says "your settings look fine", congratulations, you are ready to go to the next step and deploy GOCDB!
    • If not, displayed error message should tell you what has gone wrong. Go through the preparation steps again and perform a configuration check until you have solved all problems.


Browse to your admin alias...

GocdbAdminHome.jpg


then click Configuration Check...


ConfigCheckOK.jpg

Note: SELinux

If your web portal configuration check shows problems connecting to the database then please check that SELinux is not enabled. We have found that SELinux can block access to the database. Currently the GOCDB team recommend disabling SELinux on a GOCDB server.

Running GOCDB-4 setup

If you have followed all the steps of the previous section, have reached the "final check" step and have managed to display a nice "your settings look fine" on a web page, then deploying GOCDB-4 is one of the simplest things on earth:

  • Open GOCDB admin web page in your browser ("/admin")
  • Browse to the Main Admin Menu
  • Click on the Deploy and setup GOCDB for the first time link
  • Click on the start deployment button
  • The system will then go through a configuration check again, and should display a success message (if not, then you've not followed previous steps properly!)
  • Click on the deploy now button
    • If you've got a success message then congratulations! You have successfully deployed GOCDB-4.
    • As part of the deployment process, a number of Oracle jobs are deployed to your database. These are known as 'Materialized Views'. These jobs are ran by the Oracle scheduler in order to extract and refresh data used by the PI. You can ensure that the Oracle jobs have been deployed correctly using an Oracle command line tool, see GOCDB/Regional Module Technical Documentation/Oracle setup#GOCDB_Materialized_View_Jobs.
      • Now browse to the "Administration" section for next steps
    • If you receive any error messages, please browse to the errors and troubleshooting section.

Importing the Seed Data and Sample Data

Seed Data (Required)

Executing the XML Input module without changing default configuration will import the Seed Data into the GOCDB-4 database. To do that, follow next steps:

Setting up an admin user within the seed data

The user management system currently prevents unprivileged users from editing data in the web portal. If you wish to become a privileged user you will need to edit the same data, adding your certificate DN as a member of the Admins group.

The following steps explain this process:

  • For GOCDB v4.2+ Open the config/xml_input/input/SeedData/User_Admins.xml file.
  • For GOCDB <= v4.1 Open the config/xml_input/input/SampleData/User_Admins.xml file.
  • identify the following tags:
<Certificate_DN key="primary" grid_id="0">/C=UK/O=eScience/OU=CLRC/L=RAL/CN=gilles mathieu</Certificate_DN>
<title>Mr</title>
<forename>Gilles</forename>
<surname>Mathieu</surname>
  • Change the certificate DN to the certificate you wish to grant Admin access to, and correspondingly update Name/surname information.

Note: if you want to set up more than one admin, just duplicate the whole of the <user> tag and all its children tag within the <results> tag, and put a different DN/forename/Surname each time.

Executing XML Input

By browsing to the previously configured "/xml_input" Apache alias, the data import will commence. The process completes by outputting "Done" to the web browser.

  • Note, GOCDB v4.2+ inserts required seed data and lookup types only (in v4.2, all required data was separated from sample sites/service/user and other imaginary data).


ImportSeedData.jpg


Sample Data (Optional)

To populate the database with the Sample Data (imaginary sites, services and users), follow these steps:

  1. Open file config/xml_input/xml_input.ini
  2. Change line xml_input = ../../config/xml_input/input/SeedData to xml_input = ../../config/xml_input/input/SampleData. This value denotes the data import directory (to speed up the data import, you can also try setting the temporisation parameter to '0' so that no delay occurs between each insert - we used to set this to '500000' millisecs as OracleXE would periodically fail if the data import was too fast).
  3. In your Web browser, go to the "/xml_input" Apache alias once more (or refresh this page).
  4. The data files located in your configured 'input_xml' directory will be imported into your database.
  5. Browse to your gocdb home page, the sample data will now be shown in the web portal.
  6. Repeat this procedure whenever you need to import other/custom XML data files into your gocdb database.

XML Input Data Format

  1. The structure of the XML input data files used for the Seed and Sample is described at GOCDB/Release4/XML Input

Testing

If all of the above has been successful then your web portal instance should be ready for testing. Browse to the web_portal Apache alias created in an earlier step and see the data that have been imported.

If everything has been successful up to this point then you have successfully completed the installation instructions for GOCDB-4!

Possible errors and troubleshooting

This documentation is still in a draft state and you might therefore not find here the error you are experiencing. Should that be the case, please contact us so that we can work together on it and feed documentation with this error and its solution.

TNS:listener errors (ORA-12516)

I get the following errors reported in the portal interface:

Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
...more

If you are running on Oracle XE (the free version), you may see these errors. If so you should try to increase the number of sessions available in your database for use by GOCDB. In an Oracle sql client, try the following (set the session count to a number that is reasonable for you load):

alter system set sessions = 400 scope=spfile;
alter system set processes = 400 scope=spfile;

Deployment is interrupted because the webserver timed out

The set of operations carried on during initial deployment can sometimes take a long time. Your webserver might be configured to time out after a certain time without answer, and if this time is too short this can result in a flawed deployment if the script stops half way through. The best thing to do in such a case is:

  • From your miserably looking, half blank page where your setup script has stopped, click on the main admin menu item in the top menu bar
  • You're back on the welcome page. Now click on the Object and Schema Management link
  • Hopefully you'll see nice tabs and a list of possible actions (refer to the "administration" section for further details)
  • click on the wipe clean link
  • go again through the setup section as described above

If this doesn't help, go to the Object and Schema Management page again and deploy each module one after the other (see the "administration" section for details).

If you still get a server timeout, you need to reconfigure Apache so that the timeout value is higher. Check the Apache documentation for that

Portal doesn't work

Even if everything goes OK during the setup it might happen that you can't browse the web portal (and get some PHP fatal errors along the way). The most probable cause for that is that https is not fully enabled on the server. Check your Apache SSL configuration and Apache documentation. If this is not the cause of the problem pleas contact us (see "support" section below) with the error message you are getting

Support

If you need to contact us, please do so with as many details as possible so that we can track down with you what's gone wrong. Support requests should be sent to gocdb-admins_AT_mailman.egi.eu

Administration

Object management

Customisation

GOCDB/Release4/Customisation

Adding a local schema

Custom Features

Custom features are those that aren't enabled on the central GOCDB but can be enabled on a standalone GOCDB instance.

Siteless Endpoints

Introduction

Siteless endpoints allows service groups to host service endpoints that do not have a hosting physical site. In the standard GOCDB configuration this isn't allowed - a service endpoint must belong to a physical site.

Configuration

In config/local_info.xml siteless endpoints can be enabled as follows:

       <optional_features>
           <siteless_endpoints>true</siteless_endpoints>
       </optional_features>

Behavior Changes

With siteless endpoints enabled the following behavior changes can be observed:

  • When selecting service endpoints to add to a service group, a new button is available: "Add a new service endpoint to this service group"
    • This leads to a page to add a new service endpoint to the existing service group
  • When searching for service endpoints to add to a service group, siteless endpoints will now be shown
  • Siteless endpoints will be shown in the search results
  • A service endpoint that doesn't have a hosting site can be edited and deleted by an administrator of the hosting service group (the service group that created the service endpoint)
  • When removing a siteless endpoint from the creating service group, the siteless endpoint will be deleted from the database entirely. (Previously it wouldn't be deleted as it would have a hosting physical site). An extra notification is shown confirming deletion.
  • Downtimes can be created that affect siteless endpoints

To facilitate the above we've added support for showing a list of downtimes on the service group page. This update has been put into the production GOCDB system.

Potential Improvements

The programmatic interface queries that show downtimes require that a service endpoint has a hosting physical site. If this requirement is made optional, downtimes over siteless endpoints would be shown through the PI.