Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

GOCDB/Regional Module Technical Documentation

From EGIWiki
Jump to navigation Jump to search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Tools menu: Main page Instructions for developers AAI Proxy Accounting Portal Accounting Repository AppDB ARGO GGUS GOCDB
Message brokers Licenses OTAGs Operations Portal Perun EGI Collaboration tools LToS EGI Workload Manager


<< Back to GOCDB/Documentation_Index

This page is the main technical documentation for GOCDB-4 regional module

Download

LATEST VERSION: GOCDB-4.2

svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.2/gocdb gocdb-4.2
svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb gocdbTrunk

Deployment

System prerequisites

GOCDB-4 regional module has two components: a database and a web front end. Both components can work on the same machine but it is very likely that in most environments they will be separated. This is what we would advise anyway. Machine requirements for these 2 components are described below.

Database

  • Required database: Oracle 10g or higher (note: Oracle 10g and 11g XE Express Editions, which come with free licenses are perfectly suitable)
  • Required space: 150 MB
  • Dowload free Oracle XE from the following link and follow the installation instructions: http://www.oracle.com/technetwork/database/express-edition/overview/index.html
  • For gocdb admin tasks, we also recommend the SQL Developer tool.

Web frontend

The machine to use as the web frontend will need the following:

  • Apache http server version 2.2 or higher.
  • PHP version 5.3 or higher (version <=5.2 has some OOP related bugs).
    • PHP oci8 extension
    • libxml2 and DOM support for PHP (Note: On RHEL, PHP requires the PHP XML RPM to be installed for this component to function).
    • OpenSSL Extension for PHP
  • Oracle Instant client version 10 or higher (downloadable from Oracle website)
  • a X509 host certificate for the machine

Preparing your installation

Database preparation

Important note: This section needs to be executed by Database System Administrators.

  • If you don't have admin rights on the DB you are using, please contact your DBAs with the instructions contained in this section.
1- Create Custom tablespaces for GOCDB (Optional)

GOCDB-4 database contains 3 distinct groups of tables: Metadata or core data, GOCDB data, and local data. Although this is not mandatory, we recommend that 3 different tablespaces are created to store these different tables.

  • If you prefer, you can skip this step and rely on the default 'USERS' tablespace.
Sugested Tablespace Name Min Recommended Size Purpose
TS_GRIDCORE 50m Will store the core tables containing schema metadata
TS_GOCDBDATA 50m Will store GOCDB data tables (common structure to all GOCDB modules)
TS_LOCALDATA 50m Will store your local tables if you have some local information to store

Note: You must configure gocdb to use these custom tablespace names in the 'local_info.xml' config file as described below.

To create these optional tablespaces, run the following script (as SYSTEM user):

 CREATE TABLESPACE TS_GRIDCORE; 
 CREATE TABLESPACE TS_GOCDBDATA; 
 CREATE TABLESPACE TS_LOCALDATA; 

The created tablespaces will then be:

  • Permanent, locally managed and with system allocated extent size.
  • The corresponding datafiles will be created in the location provided in the DB_CREATE_FILE_DEST parameter and with size 100 MB.
  • The datafiles will be autoextensible with no maximum size.
  • Names of datafiles will be similar to "ora_applicat_zxyykpt000.dbf"

You can customise tablespace creation, there are many possible options (see Oracle docs for further info). For example:

 CREATE TABLESPACE <tblspc_name> DATAFILE 'C:\ORA\<fileNameForTableSpace>.dbf' SIZE 50M AUTOEXTEND ON MAXSIZE 100M;
2- Create a DB user for GOCDB

We advise that you create a dedicated GOCDB Oracle user. You can create the gocdb user with the following script (substitute <PASSWORD> with a sensible password and run as system user):

-- USER SQL
CREATE USER GOCDBTest IDENTIFIED BY <PASSWORD> 
DEFAULT TABLESPACE "USERS"
TEMPORARY TABLESPACE "TEMP";
  • If you created the three custom tablespaces as described above, you will need to grant access and unlimited quotas to your GOCDB user on those previously defined tablespaces (you do not need to do this if you are relying on the default 'USERS' tablespace).
ALTER USER GOCDB QUOTA UNLIMITED ON TS_GRIDCORE;
ALTER USER GOCDB QUOTA UNLIMITED ON TS_GOCDBDATA;
ALTER USER GOCDB QUOTA UNLIMITED ON TS_LOCALDATA;
3- Manage user privileges

Grant privileges to your GOCDB user:

This can be done by executing the following script (run as SYSTEM user):

-- ROLES
GRANT "RESOURCE" TO GOCDB ;

-- SYSTEM PRIVILEGES
GRANT CREATE TRIGGER TO GOCDB ;
GRANT CREATE SEQUENCE TO GOCDB ;
GRANT CREATE TABLE TO GOCDB ;
GRANT CREATE JOB TO GOCDB ;
GRANT CREATE PROCEDURE TO GOCDB ;
GRANT CREATE TYPE TO GOCDB ;
GRANT CREATE SESSION TO GOCDB ;
GRANT CREATE MATERIALIZED VIEW TO GOCDB ; 

(note, if you are using the free Oracle XE, and depending on the version, the 'CREATE JOB' system privilege is not listed as an option in the 'Manage Database Users' part of the Oracle web-application interface).

Web machine preparation

1- install host certificate for the machine
2- Get and deploy GOCDB-4 code package
  • See #Download above
  • Move the code to a suitable location; this will be the root of your gocdb installation.
  • Verify the user running httpd has read access to the directories/files. The files are often not accessible by the Apache user by default. (Default installation location: /usr/share/GOCDB)
  • Create the following Apache Aliases (or set up virtual hosts)
    • Alias "/portal" "/usr/share/GOCDB/htdocs/web_portal"
    • Alias "/admin" "/usr/share/GOCDB/htdocs/admin"
    • Alias "/xml_input" "/usr/share/GOCDB/htdocs/xml_input"
  • Configure Apache to use SSL and client certificate verification for the web portal section of the site. The user certificate DN's are used for authentication to the web front end.
  • Ensure the web server port is open on your machine. As a first step, you probably want the port to be open internally only (giving the whole wide world access to your GOCDB admin page is probably not a good idea)
  • Restart Apache so that the new aliases are loaded from the configuration file.
  • Check that the pages can be correctly accessed by browsing to the newly created "admin" alias. Note that the system is not yet fully set up and will not work!

Note: Creating and managing http aliases is part of Apache http server handling. Please check Apache documentation if you need details. An example virtual host configuration file for GOCDB GOCDB/Sample GOCDB Virtual Host Config file

3- Create and edit prom.conf configuration file
  • Rename or copy /usr/share/GOCDB/config/prom.conf_TEMPLATE to /usr/share/GOCDB/config/prom.conf
  • Edit prom.conf. Under the [Oracle_server] section, specify the connection details to your DB. "host" can be a simple hostname or a full connection string (see examples below).

All settings must be enclosed in quotation marks.

example 1: Specifying a simple hostname

host="dbhostname.domain.org"
port="1521"
database="Database_Name" 
user="GOCDB_USER"
password="gocDBpassWord"

example 2: Specifying a connection string (warning: the whole string has to be on one single line) )

host="(DESCRIPTION=(LOAD_BALANCE=yes)(FAILOVER=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=dbhostname1.domain.org)\\
   (PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = dbhostname2.domain.org)(PORT = 1521))\\
   (CONNECT_DATA=(SERVICE_NAME=gocdb.domain.com)(FAILOVER_MODE=(TYPE=SESSION)(METHOD=BASIC))))"
port=
database= 
user="GOCDB_USER"
password="gocDBpassWord"
4- Create and edit local_info.xml configuration file
  • Rename or copy /usr/share/GOCDB/config/local_info.xml_TEMPLATE to /usr/share/GOCDB/config/local_info.xml
  • Edit local_info.xml whith the following information:
    • tablespace names, as configured in the previous section (if you are relying on the default 'USERS' tablespace, then specify 'USERS' for each of the three elements)
    • Collection ID, which is a unique ID that will eventually be used to distinguish different GOCDB regional instances. Currently this should be left as 0.
    • instance type, which can be either "central" or "regional". This should be set to 'regional' for a read/write GOCDB or 'central' for a read only instance.
    • web_portal_url, which is the full url of your GOCDB web portal as defined during the setup process above
5- Create and edit local_schema.xml configuration file
  • Rename or copy /usr/share/GOCDB/config/local_schema.xml_TEMPLATE to /usr/share/GOCDB/config/local_schema.xml
  • You don't need to edit/modify this file now. How to use this file (and why) will be described in the Customisation section

Final check

If you have carefully followed all the previous steps, you should now have:

  • An Oracle database where GOCDB will run, with connection details and proper privileges
  • A web machine that will host GOCDB front end
  • On the web machine, the following configuration files appropriately edited:
    • /usr/share/GOCDB/config/prom.conf, with proper database connection details
    • /usr/share/GOCDB/config/local_info.xml, with proper tablespace information and your assigned collection id
    • /usr/share/GOCDB/config/local_schema.xml, left as it is for now (content doesn't matter for the setup process as long as the file is here)
  • An alias to GOCDB admin web page, available through your browser.
  • An alias to GOCDB web front end, available through your browser.
  • An alias to GOCDB xml input, available through your browser.

And now the moment of truth...

  • Open GOCDB admin web page in your browser as shown below (the alias you have defined above: "/admin")
  • On the welcome page, click on the configuration check link
    • If it says "your settings look fine", congratulations, you are ready to go to the next step and deploy GOCDB!
    • If not, displayed error message should tell you what has gone wrong. Go through the preparation steps again and perform a configuration check until you have solved all problems.


Browse to your admin alias...

GocdbAdminHome.jpg


then click Configuration Check...


ConfigCheckOK.jpg

Running GOCDB-4 setup

If you have followed all the steps of the previous section, have reached the "final check" step and have managed to display a nice "your settings look fine" on a web page, then deploying GOCDB-4 is one of the simplest things on earth:

  • Open GOCDB admin web page in your browser ("/admin")
  • Browse to the Main Admin Menu
  • Click on the Deploy and setup GOCDB for the first time link
  • Click on the start deployment button
  • The system will then go through a configuration check again, and should display a success message (if not, then you've not followed previous steps properly!)
  • Click on the deploy now button
    • If you've got a success message then congratulations! You have successfully deployed GOCDB-4.
    • As part of the deployment process, a number of Oracle jobs are deployed to your database. These are known as 'Materialized Views'. These jobs are ran by the Oracle scheduler in order to extract and refresh data used by the PI. You can ensure that the Oracle jobs have been deployed correctly using an Oracle command line tool, see GOCDB/Regional_Module_Technical_Documentation/Oracle_setup#GOCDB_Materialized_View_Jobs.
      • Now browse to the "Administration" section for next steps
    • If you receive any error messages, please browse to the errors and troubleshooting section.

Importing the Seed Data

Executing the XML Input module without changing default configuration will import the seed data into the GOCDB-4 database. To do that, follow next steps:

Setting up an admin user within the sample data

The user management system currently prevents unprivileged users from editing data in the web portal. If you wish to become a privileged user you will need to edit the same data, adding your certificate DN as a member of the Admins group.

The following steps explain this process:

  • For GOCDB v4.2+ Open the config/xml_input/input/SeedData/User_Admins.xml file.
  • For GOCDB <= v4.1 Open the config/xml_input/input/SampleData/User_Admins.xml file.
  • identify the following tags:
<Certificate_DN key="primary" grid_id="0">/C=UK/O=eScience/OU=CLRC/L=RAL/CN=gilles mathieu</Certificate_DN>
<title>Mr</title>
<forename>Gilles</forename>
<surname>Mathieu</surname>
  • Change the certificate DN to the certificate you wish to grant Admin access to, and correspondingly update Name/surname information.

Note: if you want to set up more than one admin, just duplicate the whole of the <user> tag and all its children tag within the <results> tag, and put a different DN/forename/Surname each time.

Executing XML Input

By browsing to the previously configured "/xml_input" Apache alias, the data import will commence. The process completes by outputting "Done" to the web browser.

  • Note, GOCDB v4.2+ inserts required seed data and lookup types only (in v4.2, all required data was separated from sample sites/service/user and other imaginary data). If you also want to populate the database with the sample sites and imaginary data, open file config/xml_input/xml_input.ini and change line xml_input = ../../config/xml_input/input/SeedData to xml_input = ../../config/xml_input/input/SampleData and then browse to the "/xml_input" Apache alias once more.


This sample data can be shown through the web portal, presenting how the system displays data.

Testing

If all of the above has been successful then your web portal instance should be ready for testing. Browse to the web_portal Apache alias created in an earlier step and see the data that have been imported.

If everything has been successful up to this point then you have successfully completed the installation instructions for GOCDB-4!

Possible errors and troubleshooting

This documentation is still in a draft state and you might therefore not find here the error you are experiencing. Should that be the case, please contact us so that we can work together on it and feed documentation with this error and its solution.

TNS:listener errors (ORA-12516)

I get the following errors reported in the portal interface:

Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
Array ( [code] => 12516 [message] => ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] => 0 [sqltext] => ) aError is set: Array
...more

If you are running on Oracle XE (the free version), you may see these errors. If so you should try to increase the number of sessions available in your database for use by GOCDB. In an Oracle sql client, try the following (set the session count to a number that is reasonable for you load):

alter system set sessions = 400 scope=spfile;
alter system set processes = 400 scope=spfile;

Deployment is interrupted because the webserver timed out

The set of operations carried on during initial deployment can sometimes take a long time. Your webserver might be configured to time out after a certain time without answer, and if this time is too short this can result in a flawed deployment if the script stops half way through. The best thing to do in such a case is:

  • From your miserably looking, half blank page where your setup script has stopped, click on the main admin menu item in the top menu bar
  • You're back on the welcome page. Now click on the Object and Schema Management link
  • Hopefully you'll see nice tabs and a list of possible actions (refer to the "administration" section for further details)
  • click on the wipe clean link
  • go again through the setup section as described above

If this doesn't help, go to the Object and Schema Management page again and deploy each module one after the other (see the "administration" section for details).

If you still get a server timeout, you need to reconfigure Apache so that the timeout value is higher. Check the Apache documentation for that

Portal doesn't work

Even if everything goes OK during the setup it might happen that you can't browse the web portal (and get some PHP fatal errors along the way). The most probable cause for that is that https is not fully enabled on the server. Check your Apache SSL configuration and Apache documentation. If this is not the cause of the problem pleas contact us (see "support" section below) with the error message you are getting

Support

If you need to contact us, please do so with as many details as possible so that we can track down with you what's gone wrong. Support requests should be sent to gocdb-admins_AT_mailtalk.ac.uk

Administration

Object management

Customisation

GOCDB/Release4/Customisation

Adding a local schema