Difference between revisions of "GOCDB/Regional Module Technical Documentation"

From EGIWiki
Jump to: navigation, search
(Siteless Endpoints)
m (Download)
 
(134 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Template:Op menubar}} {{Template:Tools menubar}} &lt;&lt; Back to [[GOCDB/Documentation Index]] <br> __TOC__
+
{{Template:Op menubar}} {{Template:GOCDB_menubar}} {{TOC_top}} This page is the main install documentation for GOCDB Version 5
  
This page is the main technical documentation for GOCDB-4 regional module
+
== Download  ==
 +
 
 +
'''LATEST VERSION: GOCDBv5.6''' <br>
 +
https://github.com/GOCDB/gocdb
 +
 
 +
For install see:
 +
https://github.com/GOCDB/gocdb/blob/master/INSTALL.md
  
== Download  ==
+
<!--
 +
*Packaged V5.2: https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-5.2/GocDB-5.2.zip
 +
*svn v5.2:
 +
<pre>svn co https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-5.2
 +
</pre>
 +
*Latest dev/trunk svn:
 +
<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/branches/gocdb/MultipleEndpointsGocDB/</pre>
 +
-->
  
'''LATEST VERSION: GOCDB-4.3'''  
+
<!--<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/branches/gocdb/Doctrine%20Web%20Portal</pre> -->
 +
<!--Note, for svn co, if you are asked to authenticate with a client certificate as below, you can dismiss by pressing Enter
 +
<pre>Authentication realm: https://www.sysadmin.hep.ac.uk:443
 +
Client certificate filename: &lt;ENTER to dismiss&gt;
 +
</pre>
 +
<br> '''OLD v4.4 Install doc has been relocated''' <br>
  
*Download the GOCDB 4.3 regional package from: https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.3/GOCDB-4.3.zip '''or'''
+
*https://wiki.egi.eu/wiki/GOCDB/v4/Regional_Module_Technical_Documentation
*Checkout the src code using a subversion client
 
**v4.3 tagged release: https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.3/gocdb
 
<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-4.3/gocdb gocdb-4.3</pre>
 
*Trunk (latest development version): https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb
 
<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/trunk/gocdb gocdbTrunk</pre> <!--* Either install the GOCDB-4 RPM (rpm -i GOCDB-4.0-4.d7.noarch.rpm) or move the check-out code to a suitable location and use as the root of your gocdb installation. -->
 
  
== Deployment  ==
+
<br>
 +
-->
 +
<br>
  
 +
<!--
 +
== Prerequisites==
 
=== System prerequisites  ===
 
=== System prerequisites  ===
  
GOCDB-4 regional module has two components: a database and a web front end. Both components can work on the same machine but it is very likely that in most environments they will be separated. This is what we would advise anyway. Machine requirements for these 2 components are described below.  
+
GOCDB-5 has two components: a database and a web front end. Both components can work on the same machine but it is very likely that in most environments they will be separated. This is what we would advise. Machine requirements for these 2 components are described below.  
  
 
==== Database  ====
 
==== Database  ====
  
*'''Required database:''' Oracle 10g or higher (note: Oracle 10g and 11g XE Express Editions, which come with free licenses are perfectly suitable)  
+
Version 5 of GocDB uses Doctrine ORM (Object Relational Mapping) as it's interface to the datastore. This allows GocDB to be deployed to either a Oracle or MySQL database (Doctrine also supports other DBs but these are untested).
*'''Required space:''' 150 MB  
+
 
*Dowload free Oracle XE from the following link and follow the installation instructions: http://www.oracle.com/technetwork/database/express-edition/overview/index.html  
+
Oracle:
 +
 
 +
*Required database: Oracle 11g or higher (note: the free Oracle 11g XE Express Editions which comes with free license is perfectly suitable. 10g may also work but is untested)  
 +
*Required space: 150 MB Dowload free Oracle XE from the following link and follow the installation instructions: http://www.oracle.com/technetwork/database/express-edition/overview/index.html  
 
*For gocdb admin tasks, we also recommend the SQL Developer tool.
 
*For gocdb admin tasks, we also recommend the SQL Developer tool.
 +
 +
<br>
 +
 +
MySQL:
 +
 +
*Required database: Any version of MySQL will work including the free MySQL Community Server edition
 +
*Required space: See MySQL install guide:http://dev.mysql.com/doc/refman/5.7/en/installing.html
 +
 +
<br>
  
 
==== Web frontend  ====
 
==== Web frontend  ====
Line 31: Line 60:
 
The machine to use as the web frontend will need the following:  
 
The machine to use as the web frontend will need the following:  
  
*'''Apache http server''' version 2.2 or higher.  
+
*'''Apache http server'''&nbsp;version 2.2 or higher.  
 
*'''PHP''' version 5.3 or higher (version &lt;=5.2 has some OOP related bugs).  
 
*'''PHP''' version 5.3 or higher (version &lt;=5.2 has some OOP related bugs).  
**'''PHP''' oci8 extension
+
**If using Oracle: '''PHP oci8 extension''' and Oracle Instant client v10 or higher (downloadable from Oracle website)
**libxml2 and DOM support for '''PHP''' (Note: On RHEL, PHP requires the PHP XML RPM to be installed for this component to function).  
+
**'''libxml2 and DOM support''' for PHP (Note: On RHEL, PHP requires the PHP XML RPM to be installed for this component to function).  
**OpenSSL Extension for '''PHP'''
+
**'''OpenSSL Extension''' for PHP  
*'''Oracle Instant client''' version 10 or higher (downloadable from Oracle website)
+
*'''An X509 host certificate''' for the machine
*a X509 host certificate for the machine
+
 
 +
<br>
 +
 
 +
==== Optional  ====
  
=== Preparing your installation  ===
+
*'''PHP Unit''' for testing developments
  
==== Database preparation  ====
+
<br>
  
'''Important note: This section needs to be executed by Database System Administrators.'''
+
== Preparing your installation  ==
  
*If you don't have admin rights on the DB you are using, please contact your DBAs with the instructions contained in this section.
+
=== Database preparation  ===
  
===== 1- Create Custom tablespaces for GOCDB (Optional) =====
+
GocDB v5 comes with deploy script which will create the DB schema for GocDB (i.e. export the DDL to create the tables and sequences in the DB). The only preperation is that you have setup a database user/account in your chosen RDBMS and have the user access details which are required later in the installation when configuring GocDB for deployment.&nbsp;
  
GOCDB-4 database contains 3 distinct groups of tables: Metadata or core data, GOCDB data, and local data. Although this is not mandatory, we recommend that 3 different tablespaces are created to store these different tables.
+
Details for creating GOCDB user on specific RDBMS:  
 +
* [[GOCDB/Regional_Module_Technical_Documentation/DeployOnOracle|Deploy on Oracle Details]]
 +
* [[GOCDB/Regional_Module_Technical_Documentation/DeployOnMySQL|Deploy on MySQL Details]]
  
*If you prefer, you '''can <u>skip this step</u> and rely on the default 'USERS' tablespace'''.
+
=== GOCDB file system  ===
  
{|
+
Place the GocDB source folder on your chosen web server.
|-
 
! Sugested Tablespace Name
 
! Min Recommended Size
 
! Purpose
 
|-
 
| TS_GRIDCORE
 
| 50m
 
| Will store the core tables containing schema metadata
 
|-
 
| TS_GOCDBDATA
 
| 50m
 
| Will store GOCDB data tables (common structure to all GOCDB modules)
 
|-
 
| TS_LOCALDATA
 
| 50m
 
| Will store your local tables if you have some local information to store
 
|}
 
  
'''Note:''' You must configure gocdb to use these custom tablespace names in the 'local_info.xml' config file as described below.
+
=== PEAR  ===
  
To create these '''optional''' tablespaces, run the following script (as SYSTEM user):
+
Pear is recommended to install Doctrine and PHPUnit, many builds of PHP come pre-bundled with PEAR.&nbsp;  
<pre> CREATE TABLESPACE TS_GRIDCORE;  
 
CREATE TABLESPACE TS_GOCDBDATA;
 
CREATE TABLESPACE TS_LOCALDATA;
 
</pre>
 
The created tablespaces will then be:
 
  
*Permanent, locally managed and with system allocated extent size.  
+
[http://pear.php.net/manual/en/installation.getting.php http://pear.php.net/manual/en/installation.getting.php]
*The corresponding datafiles will be created in the location provided in the DB_CREATE_FILE_DEST parameter and with size 100 MB.  
 
*The datafiles will be autoextensible with no maximum size.  
 
*Names of datafiles will be similar to "ora_applicat_zxyykpt000.dbf"
 
  
You can customise tablespace creation, there are many possible options (see Oracle docs for further info). For example:
+
Verify the PEAR installation by running pear version on the command line.  
<pre> CREATE TABLESPACE &lt;tblspc_name&gt; DATAFILE 'C:\ORA\&lt;fileNameForTableSpace&gt;.dbf' SIZE 50M AUTOEXTEND ON MAXSIZE 100M;
+
<pre>$ pear version
 +
PEAR Version: 1.9.4
 +
PHP Version: 5.3.8
 +
Zend Engine Version: 2.3.0
 +
Running on: Windows NT ESCPC0095 6.1 build 7601 (Windows 7 Business Edition Service Pack 1) i586
 
</pre>  
 
</pre>  
===== 2- Create a DB user for GOCDB =====
+
=== Doctrine ===
  
We advise that you create a dedicated GOCDB Oracle user. You can create the gocdb user with the following script (substitute &lt;PASSWORD&gt; with a sensible password and run as system user):  
+
Doctrine can be installed using PEAR. First add the Doctrine and Symfony channels to PEAR:  
<pre>-- USER SQL
+
<pre>$pear channel-discover pear.doctrine-project.org
CREATE USER GOCDB IDENTIFIED BY &lt;PASSWORD&gt;
+
$pear channel-discover pear.symfony.com
DEFAULT TABLESPACE "USERS"
+
</pre>
TEMPORARY TABLESPACE "TEMP";
+
Then install Doctrine and it's dependencies:
 +
<pre>$pear install --alldeps doctrine/DoctrineORM
 +
</pre>
 +
Verify the Doctrine installation by running "doctrine --version" on the command line (or doctrine.bat --version if on Win):
 +
<pre>$ doctrine --version
 +
Doctrine Command Line Interface version 2.3.3
 
</pre>  
 
</pre>  
*If you created the three custom tablespaces as described above, you will need to grant access and unlimited quotas to your GOCDB user on those previously defined tablespaces (you do not need to do this if you are relying on the default 'USERS' tablespace).
+
[http://docs.doctrine-project.org/en/latest/ More information on Doctrine can be found here at the Doctrine2 site.]
<pre>ALTER USER GOCDB QUOTA UNLIMITED ON TS_GRIDCORE;
+
 
ALTER USER GOCDB QUOTA UNLIMITED ON TS_GOCDBDATA;
+
=== PHPUnit  ===
ALTER USER GOCDB QUOTA UNLIMITED ON TS_LOCALDATA;
+
 
 +
PHPUnit is an optional choice for GOCDB users. GOCDB5 includes a test suite of unit tests that can be used to verify buisness logic and database connections for a deployed instance of GOCDB. In addition developers extending and modifying GOCDB can write unit tests for their own developements. To install PHPUnit use PEAR:
 +
<pre>$pear clear-cache
 +
$pear config-set auto_discover 1
 +
$pear install pear.phpunit.de/PHPUnit
 
</pre>  
 
</pre>  
===== 3- Manage user privileges =====
+
This install can then be verified using "phpunit --version" on the command line:
 +
<pre>$ phpunit --version
 +
PHPUnit 3.7.27 by Sebastian Bergmann.
 +
</pre>
 +
[http://phpunit.de/manual/current/en/automating-tests.html More information on PHPUnit can be found via the PHPUnit documenation here.]
 +
 
 +
== Deploying GOCDB5  ==
 +
 
 +
=== Database Connection ===
 +
You should now have all the required blocks in place to deploy GOCDB V5. To Deploy the database schema to your desired database using Doctrine:
 +
* Navigate to to '<gocDBSrcHome>/lib/Doctrine' folder.
 +
* Locate the provided template file: ''bootstrap_doctrine_TEMPLATE.php''. In this file you will find three blocks of code commented out, once for each of the supported databased, SQLite, Oracle and MySQL as shown below:
 +
* '''Copy this file to''' ''bootstrap_doctrine.php'' in the same dir as the template file and modify to specify your chosen DB connection details.
 +
 
 +
<pre>
 +
Un-comment the area for your chosen database and fill in the details for your chosen database.
 +
 
 +
        ///////////////////////SQLITE CONNECTION DETAILS/////////////////////////////////////////////
 +
//      $conn = array(
 +
//         'driver' => 'pdo_sqlite',
 +
//         'path' => __DIR__ . '/db.sqlite',
 +
//      );
 +
        /////////////////////////////////////////////////////////////////////////////////////////////
 +
 
 +
 +
///////////////////////ORACLE CONNECTION DETAILS////////////////////////////////////////////
 +
// $conn = array(
 +
// 'driver' => 'oci8',
 +
// 'user' => 'docsvn',
 +
// 'password' => 'doc',
 +
// 'host' => 'localhost',
 +
// 'port' => 1521,
 +
        //              /*'service' = true,*/ //This may be needed depending on your Oracle server
 +
// 'dbname' => 'XE'
 +
// );
 +
//  // Need to explicitly set the Oracle session date format [1]
 +
//  $evm = new EventManager();
 +
//  $evm->addEventSubscriber(new OracleSessionInit(array('NLS_TIME_FORMAT' => 'HH24:MI:SS')));
 +
        /////////////////////////////////////////////////////////////////////////////////////////////
 +
 
 +
 +
///////////////////////MYSQL CONNECTION DETAILS////////////////////////////////////////////
 +
//  $conn = array(
 +
// 'driver' => 'pdo_mysql',
 +
// 'user' => 'doctrine',
 +
// 'password' => 'doc',
 +
// 'host' => 'localhost',
 +
// 'dbname' => 'doctrine'
 +
//  );
 +
        /////////////////////////////////////////////////////////////////////////////////////////////
 +
</pre>
 +
 
  
Grant privileges to your GOCDB user:  
+
Note: Doctrine can use APC caching however this is not suitable for all deployments. To disable comment out the APC configuration lines within the bootstrap_doctrine.php file:
 +
<pre>$config-&gt;setMetadataCacheImpl(new \Doctrine\Common\Cache\ApcCache());
 +
$config-&gt;setQueryCacheImpl(new \Doctrine\Common\Cache\ApcCache());
 +
</pre>
  
This can be done by executing the following script (run as SYSTEM user):  
+
=== Deploying GOCDB  ===
<pre>-- ROLES
+
 
GRANT "RESOURCE" TO GOCDB&nbsp;;
+
GocDB can be deployed as a blank instance ready for use or as a sample instance with a small amount of example data to demonstrate GocDB. The deploy script can be found at ''/lib/Doctrine/deploy/deploy.sh'':<br>
 +
 
 +
'''Windows Users:''' These scripts are designed for deployment on a Linux machine (Cygwin on Windows is ok). 
 +
 
 +
 
 +
By supplying a flag when executing the script you can deploy either a new empty database or a sample database eg:
 +
 
 +
<pre>
 +
$./deploy.sh -s
 +
 
 +
Deploying GocDB with sample data
 +
</pre>
 +
 
 +
or
 +
<pre>
 +
$./deploy.sh -n
 +
 
 +
Deploying new empty GocDB instance
 +
</pre>
 +
 
 +
=== Apache Configuration  ===
 +
 
 +
Now that your database is deployed the final step is to configure Apache.
 +
 
 +
==== /etc/sysconfig/httpd - Oracle configuration ====
 +
 
 +
The Oracle environment must be set correctly before starting Apache so that PHP OCI8 works correctly. In general you should set the same variables that are set by the ''$ORACLE_HOME/bin/oracle_env.sh'' script. The necessary environment variables can be set in Apache's environment configuration file.
 +
On RedHat Linux and its derivatives with the default httpd package, this is ''/etc/sysconfig/httpd''.
 +
<pre>
 +
export ORACLE_HOME=/u01/app/oracle/product/12.1    # Adjust pathname to match your installation
 +
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
 +
</pre>
 +
 
 +
==== httpd.conf  ====
 +
 
 +
A few modules need to be enabled for GOCDB to work, check these are enabled on your Apache and if not enable them.&nbsp;  
 +
 
 +
In ''httpd.conf'' enable ssl_module by un-commenting these lines:
 +
<pre>LoadModule ssl_module modules/mod_ssl.so</pre>
  
-- SYSTEM PRIVILEGES
 
GRANT CREATE TRIGGER TO GOCDB&nbsp;;
 
GRANT CREATE SEQUENCE TO GOCDB&nbsp;;
 
GRANT CREATE TABLE TO GOCDB&nbsp;;
 
GRANT CREATE JOB TO GOCDB&nbsp;;
 
GRANT CREATE PROCEDURE TO GOCDB&nbsp;;
 
GRANT CREATE TYPE TO GOCDB&nbsp;;
 
GRANT CREATE SESSION TO GOCDB&nbsp;;
 
GRANT CREATE MATERIALIZED VIEW TO GOCDB&nbsp;;
 
</pre>
 
(note, if you are using the free '''Oracle XE, and depending on the version, the 'CREATE JOB'''' system privilege is not listed as an option in the 'Manage Database Users' part of the Oracle web-application interface).
 
  
==== Web machine preparation  ====
 
  
===== 1- install host certificate for the machine =====
+
==== gocdbssl.conf ====
 +
A sample ''gocdbssl.conf'' is provided in GocDBSrc/config. This file will define the alias and SSL settings used by Apache to connect to GocDB. 
  
*GOCDB web portal component needs the machine to run a https server which requires a server X509 certificate. Please check that Apache is properly configured to use the installed server certificate. Help about how to set up SSL configuration on Apache can be obtained from Apache mod_ssl online documentation.
+
===== Paths =====
*For development and testing, you can use a self signed certificate (this must not be used in production!). These pages show you how to create a self signed cert for testing purposes:
 
**http://www.sslshopper.com/article-how-to-create-and-install-an-apache-self-signed-certificate.html
 
**https://help.ubuntu.com/8.04/serverguide/C/certificates-and-security.html
 
*Note, GOCDB uses trusted CA certificates to verify user certificates. You can dowload the EGI/IGTF CA certificate bundle from the following link: https://wiki.egi.eu/wiki/EGI_IGTF_Release
 
*Note, CA certificates must be named differently according to the version of OpenSSL you are using (i.e. v1.0.X or v0.9.X). This is explained in more detail at the previous link.  
 
  
===== 2- Get and deploy GOCDB-4 code package  =====
+
The first step is to setup the paths for the ''DocumentRoot, ErrorLog and TransferLog'' and define the ''ServerName'' and ''ServerAdmin'' details for your particular web service where you are hosting GocDB.
  
*See [[#Download]] above
+
===== SSL Keys  =====
*Move the code to a suitable location; this will be the root of your gocdb installation.
 
*Verify the user running httpd has read access to the directories/files. The files are often not accessible by the Apache user by default. (Default installation location: '''/usr/share/GOCDB''')
 
*Create the following Apache Aliases (or set up virtual hosts)
 
**Alias "/portal" "/usr/share/GOCDB/htdocs/web_portal"
 
**Alias "/admin" "/usr/share/GOCDB/htdocs/admin"
 
**Alias "/xml_input" "/usr/share/GOCDB/htdocs/xml_input"
 
**Alias "/gocdbpi/public" "/usr/share/GOCDB/htdocs/PI/public"
 
**Alias "/gocdbpi/private" "/usr/share/GOCDB/htdocs/PI/private"
 
*Configure Apache to use SSL and client certificate verification for the web portal section of the site. The user certificate DN's are used for authentication to the web front end.
 
*Ensure the web server port is open on your machine. As a first step, you probably want the port to be open internally only (giving the whole wide world access to your GOCDB admin page is probably not a good idea)
 
*Restart Apache so that the new aliases are loaded from the configuration file.
 
*Check that the pages can be correctly accessed by browsing to the newly created "admin" alias. Note that the system is not yet fully set up and will not work!
 
  
''Note: Creating and managing http aliases is part of Apache http server handling. Please check Apache documentation if you need details.'' ''An example virtual host configuration file for GOCDB [[GOCDB/Sample GOCDB Virtual Host Config file]]''  
+
The ''SSLCertificateFile'' should point to the location of your server SSL key file. The ''SSLCertificateKeyFile'' -- The ''SSLCACertificationPath'' should point to the location of your CA certificates.
  
===== 3- Create and edit prom.conf configuration file  =====
+
For development the ''SSLCertificatedFile'' and ''SSLCertificateKeyFile'' can be self signed certificates. However these must not be used in production.
  
*Rename or copy ''/usr/share/GOCDB/config/prom.conf_TEMPLATE'' to ''/usr/share/GOCDB/config/prom.conf''
+
===== Aliases  =====
*Edit ''prom.conf''. Under the [Oracle_server] section, specify the connection details to your DB. "host" can be a simple hostname or a full connection string (see examples below).
 
  
'''All settings must be enclosed in quotation marks.'''
+
The final step is to set the Alias to map your desired directory for GocDB webportal and Programmatic Interface (PI) to the correct folder and ensure the correct rules are defined for your server.  
  
example 1: Specifying a simple hostname (you will need to use this style if you are using the free Oracle XE)
+
For the webportal the following settings are an example:
 +
<pre>    Alias "/portal" "/gocDBsrc/htdocs/web_portal"
 +
    &lt;Directory "/gocDBsrc/htdocs/web_portal"&gt;
 +
            SSLVerifyClient  require
 +
            Options FollowSymLinks Indexes
 +
            Order deny,allow
 +
            Allow from 127.0.0.1
 +
            deny from all 
 +
        &lt;/Directory&gt;
 +
</pre>
 +
The alias should and file location should also be set for the public PI and private PI. Public PI:
 +
<pre>Alias "/gocdbpi/public" "/gocDBsrc/htdocs/PI/public"
 +
&lt;Directory "/gocDBsrc/htdocs/PI/public"&gt;
 +
</pre>
 +
Private PI:  
 +
<pre>Alias "/gocdbpi/private" "/gocDBsrc/htdocs/PI/private"
 +
&lt;Directory "/gocDBsrc/htdocs/PI/private"&gt;
 +
</pre>
  
host="dbhostname.domain.org"
+
=== Compiled Entities ===
port="1521"
+
This step is optional, but for production servers this step is strongly recommended. When Doctrine uses an entity it creates a compiled version of the entity which by default is stored in the machines temporary folder. This is un-suitable for production machines as this may well be emptied. To avoid this uncomment the following line in bootstrap_doctrine.php:
database="XE"
+
<pre>
user="GOCDB"
+
$config->setProxyDir(__DIR__."/compiledEntities");
password="gocDBpassWord"
+
</pre>
  
example 2: Specifying a connection string for an Oracle cluster (warning: the whole string has to be on one single line)
+
You now need to create this folder in the same directory as the ''bootstrap_doctrine.php'' file:
 +
<pre>
 +
$mkdir compiledEntities
 +
</pre>
  
host="(DESCRIPTION=(LOAD_BALANCE=yes)(FAILOVER=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=dbhostname1.domain.org)\\
+
Once this folder is created it, your apache server needs write permissions (the exact command may change depending on your apache server):
  (PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = dbhostname2.domain.org)(PORT = 1521))\\
 
  (CONNECT_DATA=(SERVICE_NAME=gocdb.domain.com)(FAILOVER_MODE=(TYPE=SESSION)(METHOD=BASIC))))"
 
port=
 
database=
 
user="GOCDB"
 
password="gocDBpassWord"
 
  
===== 4- Create and edit local_info.xml configuration file  =====
+
<pre>
 +
$chown apache compiledEntities/
 +
</pre>
  
*Rename or copy ''/usr/share/GOCDB/config/local_info.xml_TEMPLATE'' to ''/usr/share/GOCDB/config/local_info.xml''
+
<pre>
*Edit ''local_info.xml'' whith the following information:  
+
$ls -ltrh
**tablespace names, as configured in the previous section (if you are relying on the default 'USERS' tablespace, then specify 'USERS' for each of the three elements)
+
total 28
**Collection ID, which is a unique ID that will eventually be used to distinguish different GOCDB regional instances. Currently this should be left as 0.  
+
drwxr--r-- 2 apache root 4.0K Nov 20 16:11 entities
**instance type, which can be either "central" or "regional". This should be set to 'regional' for a read/write GOCDB or 'central' for a read only instance.
+
-rw-r--r-- 1 apache root  131 Nov 20 16:11 README.txt
**web_portal_url, which is the full url of your GOCDB web portal as defined during the setup process above
+
-rw-r--r-- 1 apache root  219 Nov 20 16:11 cli-config.php
 +
-rw-r--r-- 1 apache root 1.6K Nov 20 16:11 bootstrap.php
 +
-rw-r--r-- 1 apache root 3.0K Nov 20 16:11 bootstrap_doctrine_TEMPLATE.php
 +
-rw-r--r-- 1 apache root 2.8K Nov 27 09:38 bootstrap_doctrine.php
 +
drwxr--r-- 2 apache root 4.0K Nov 27 09:39 compiledEntities
 +
</pre>
  
===== 5- Create and edit local_schema.xml configuration file  =====
+
Finally use the doctrine command line client to generate the compiled entities and store them in the folder:
 +
<pre>
 +
doctrine orm:generate-proxies compiledEntities/
 +
</pre>
  
*Rename or copy ''/usr/share/GOCDB/config/local_schema.xml_TEMPLATE'' to ''/usr/share/GOCDB/config/local_schema.xml''
+
This folder is now configured with the compiledEntites. It can be copied and used for new instances of GocDB as long as there have been no changes made to any of the entities, or it can be recreated as demonstrated each time.
*You don't need to edit/modify this file now. How to use this file (and why) will be described in the Customisation section
 
  
==== Final check ====
+
=== First Use ===
  
If you have carefully followed all the previous steps, you should now have:  
+
You should now be able to navigate to the GocDB webportal on your host using the URL defined in your alias. You will need to install a browser certificate that is suitable for the SSL keys you defined for your host to be able to view GocDB.
 +
<pre>
 +
https://localhost/portal
 +
https://localhost/portal/GOCDB_monitor/index.php
 +
</pre>
  
*An Oracle database where GOCDB will run, with connection details and proper privileges
+
====Local_Info.xml====
*A web machine that will host GOCDB front end
+
GocDB picks up a number of its settings and variables from the local_info.xml file located in the ''config'' folder. The ''web_portal_url'' will be output in the PI to create links from PI objects back to Portal views. The ''pi_url'' and ''server_base_url'' will both be used by the monitor. The monitor is a quick look status check for GocDB. The mointor can be found at your web_portal_url/GOCDB_monitor/. If these URL's are not set in the local config this feature will not work correctly.  
*On the web machine, the following configuration files appropriately edited:
 
**/usr/share/GOCDB/config/prom.conf, with proper database connection details
 
**/usr/share/GOCDB/config/local_info.xml, with proper tablespace information and your assigned collection id
 
**/usr/share/GOCDB/config/local_schema.xml, left as it is for now (content doesn't matter for the setup process as long as the file is here)
 
*An alias to GOCDB admin web page, available through your browser.  
 
*An alias to GOCDB web front end, available through your browser.  
 
*An alias to GOCDB xml input, available through your browser.
 
  
And now the moment of truth...  
+
''default_scope'' defines which scope if any will be used if no scope is specified when running PI queries. This can be left blank or set to a scope of your choosing. ''default_scope_match'' is again used if no scope_match parameter is supplied with PI queries. This can be either all or any. The ''minimum_scopes'' section declares how many scopes an NGI, site, service or service group must have at creation. This can be left as 0 or set as 1 if you want all your users entities to belong to at least one scope or more as dictated by your use of GocDB.
  
*Open GOCDB admin web page in your browser as shown below (the alias you have defined above: "/admin")
+
It's important at this point to understand how the scopes work with GocDB v5 especially in relation to the output of the PI. If you specify a default scope within your local_info but none of your entities have this scope then nothing will show in the PI. For an in-depth look at the scopes mechanism the section please read the section 'Multiple Scopes and Projects' in the [https://wiki.egi.eu/w/images/d/d3/GOCDB5_Grid_Topology_Information_System.pdf GOCDB5 Grid Topology Information System] document on page 5.
*On the welcome page, click on the '''configuration check''' link
 
**If it says "your settings look fine", congratulations, you are ready to go to the next step and deploy GOCDB!
 
**If not, displayed error message should tell you what has gone wrong. Go through the preparation steps again and perform a configuration check until you have solved all problems.
 
  
<br> Browse to your '''admin alias'''... <br>
+
====Setup an Admin User====
<center>
 
[[Image:GocdbAdminHome.jpg|700px|GocdbAdminHome.jpg]]
 
</center>
 
<br> then click '''Configuration Check'''...
 
<center>
 
<br> [[Image:ConfigCheckOK.jpg|700px|ConfigCheckOK.jpg]]
 
</center>
 
=== Note: SELinux ===
 
  
If your web portal configuration check shows problems connecting to the database then please check that SELinux is not enabled. We have found that SELinux can block access to the database. Currently the GOCDB&nbsp;team recommend disabling SELinux on a GOCDB&nbsp;server.
+
To get started with GocDB you will need an admin user. This is done by first registering as a user on GocDB by clicking the 'Register' link on the left menu bar at the bottom. Once you have registered yourself you will then need to set yourself as an admin. To do this you need to change the user record directly in your database. The users table has a field called 'isAdmin' which by default is set to 0. To change a user to admin set this to 1. Below is a sample of the SQL query used when using an Oracle database to set a user as admin.  
  
=== Running GOCDB-4 setup  ===
+
<pre>
 +
UPDATE users SET isadmin=1 WHERE forename='John' AND surname='Doe'
 +
</pre>
  
If you have followed all the steps of the previous section, have reached the "final check" step and have managed to display a nice "your settings look fine" on a web page, then deploying GOCDB-4 is one of the simplest things on earth:
+
Once you have an admin user you can create Projects and NGI's and grant roles to other users over these entities to allow them to add to GocDB.
  
*Open GOCDB admin web page in your browser ("/admin")
+
== Testing ==
*Browse to the ''Main Admin Menu''
 
*Click on the '''Deploy and setup GOCDB for the first time''' link
 
*Click on the '''start deployment''' button
 
*The system will then go through a configuration check again, and should display a success message (if not, then you've not followed previous steps properly!)
 
*Click on the '''deploy now''' button
 
**If you've got a success message then congratulations! You have successfully deployed GOCDB-4.
 
**As part of the deployment process, a number of Oracle jobs are deployed to your database. These are known as 'Materialized Views'. These jobs are ran by the Oracle scheduler in order to extract and refresh data used by the PI. You can ensure that the Oracle jobs have been deployed correctly using an Oracle command line tool, see [[GOCDB/Regional Module Technical Documentation/Oracle setup#GOCDB_Materialized_View_Jobs]].
 
***Now browse to the "Administration" section for next steps
 
**If you receive any error messages, please browse to the errors and troubleshooting section.
 
  
=== Importing the Seed Data and Sample Data  ===
+
GocDB v5 comes with a suite of tests that can be run to validate the install and check that Doctrine and your chosen database are operating as expected. These tests require PHPUnit, the install instructions for this can be found in section 3.6. The tests can be found in the GocDB/tests/ folder.
  
==== Seed Data (Required) ====
+
===Install the Oracle OCI drivers for PDO===
 +
To run tests you will also need to install the Oracle OCI8 PDO (PHP Data Object) drivers for PHP since the DBUnit tests
 +
use PDO for their execution to validate that the results that Doctrine is returning are true. To install the OCI8 drivers for pdo. See: http://www.php.net/manual/en/pdo.drivers.php
  
Executing the XML Input module without changing default configuration will import the Seed Data into the GOCDB-4 database. To do that, follow next steps:  
+
You can test that you have the OCI8 pdo drivers installed via php info. This can be called on the command line with $php -i or via a script with phpinfo(). There will be a section that lists your enabled pdo drivers, e.g. as below:
 +
<pre>
 +
            |-----------------------------|
 +
            |            PDO            |
 +
            |-----------------------------|
 +
            | PDO support enabled  |
 +
            | PDO drivers mysql,oci |
 +
            |-----------------------------|
 +
</pre>
  
===== Setting up an admin user within the seed data  =====
+
=== Deploying a Test Database ===
  
The user management system currently prevents unprivileged users from editing data in the web portal. If you wish to become a privileged user you will need to edit the same data, adding your certificate DN as a member of the Admins group.
+
We STRONGLY recommend that you deploy a second database with different connection details that will be used for testing.
 +
* Create a second DB account/user for your test database (e.g. GOCDB5TEST).
 +
* Copy ''bootstrap_doctrine_TEMPLATE.php'' file found in tests/doctrine to ''bootstrap_doctrine.php'' (in the same dir).
 +
* Modify ''bootstrap_doctrine.php'' to specify your GOCDB5TEST account/user. Complete the section for your database and ensure that the other connection details are deleted or commented out. 
 +
* Copy ''bootstrap_pdo_TEMPLATE.php'' to ''bootstrap_pdo.php''. Complete the details for the connection details for your database and ensure that the other 2 connections are either deleted or commented out, see below:
  
The following steps explain this process:
+
<pre>
 +
Un-comment the area for your chosen database and fill in the details for your chosen database.
  
*For GOCDB v4.2+ Open the '''config/xml_input/input/SeedData/User_Admins.xml''' file.
+
    ///////////////////////SQLITE CONNECTION DETAILS/////////////////////////////////////////////
*For GOCDB &lt;= v4.1 Open the '''config/xml_input/input/SampleData/User_Admins.xml''' file.
+
    // $sqliteFile = __DIR__ . '/../db.sqlite';
*identify the following tags:
+
    // $pdo = new PDO("sqlite:" . $sqliteFile);
 +
    // return new PHPUnit_Extensions_Database_DB_DefaultDatabaseConnection($pdo, 'sqlite');
 +
    /////////////////////////////////////////////////////////////////////////////////////////////
 +
   
 +
    ///////////////////////ORACLE CONNECTION DETAILS/////////////////////////////////////////////
 +
    // $pdo = new PDO('oci:dbname=//localhost:1521/xe', 'DoctrineUnitTests', 'doc');     
 +
    // return new PHPUnit_Extensions_Database_DB_DefaultDatabaseConnection($pdo, 'USERS');
 +
    /////////////////////////////////////////////////////////////////////////////////////////////
 +
 +
    ///////////////////////MYSQL CONNECTION DETAILS//////////////////////////////////////////////
 +
    //  $pdo = new PDO('mysql:host=localhost;dbname=doctrine;charset=UTF8', 'doctrine', 'doc');
 +
    //  return new PHPUnit_Extensions_Database_DB_DefaultDatabaseConnection($pdo);
 +
    /////////////////////////////////////////////////////////////////////////////////////////////
 +
</pre>
  
&lt;Certificate_DN key="primary" grid_id="0"&gt;/C=UK/O=eScience/OU=CLRC/L=RAL/CN=gilles mathieu&lt;/Certificate_DN&gt;
+
=== Running the Test Suite ===
&lt;title&gt;Mr&lt;/title&gt;
 
&lt;forename&gt;Gilles&lt;/forename&gt;
 
&lt;surname&gt;Mathieu&lt;/surname&gt;
 
  
*Change the certificate DN to the certificate you wish to grant Admin access to, and correspondingly update Name/surname information.
+
A initial test suite is provided that will excercise core functions of GocDB and Doctrine. This suite can be ran using the command:
 +
<pre>
 +
phpunit DoctrineTestSuite1.php
 +
</pre>
  
Note: if you want to set up more than one admin, just duplicate the whole of the &lt;user&gt; tag and all its children tag within the &lt;results&gt; tag, and put a different DN/forename/Surname each time.  
+
You will be prompted to continue as this operation will drop any tables in your test database and then recreate the schema ready to run the tests. These tests should return with no errors to demonstrate that GocDB and Doctrine are installed correctly and are working as expected.
  
===== Executing XML Input  =====
+
=== Running Individual Tests ===
  
By browsing to the previously configured "/xml_input" Apache alias, the data import will commence. The process completes by outputting "Done" to the web browser.  
+
Individual tests can be run by calling:
 +
<pre>phpunit <testNameFile>.php</pre>
 +
It is advised that at the beginning of a testing session you drop and recreate the database by running the recreateTestDB.sh script.  
  
*Note, GOCDB v4.2+ inserts required seed data and lookup types only (in v4.2, all required data was separated from sample sites/service/user and other imaginary data).
+
===Recreating the Test DB===
 +
You can run the following script found in the ''tests/doctrine'' folder to Drop and Recreate the test database whenever required:
 +
<pre>
 +
./recreateTestDB.sh
 +
</pre>
  
<br>
+
=== Writing Tests ===
  
[[Image:ImportSeedData.jpg|center|700px|ImportSeedData.jpg]]
+
If you intend to develop new functionality for your instance of GocDB you can support this development by writing unit tests for the new features. A simple example of a test can be found in GocDB/tests/exampleTests. 'FirstTest.php' demonstrates  how to create data and assert that it is as expected.  
  
<br>
+
A good tutorial for getting started with PHPUnit can be found [http://pear.php.net/manual/en/package.php.phpunit.intro.php here.]
  
==== Sample Data (Optional)  ====
+
==== Writing a Doctrine Test ====
  
To populate the database with the Sample Data (imaginary sites, services and users), follow these steps:  
+
When working with Doctrine there are a number of setup functions that needed to be included in your test file. A template file has been provided in GocDBSrc/tests/exampleTests called 'DoctrineTestTemplate.php'. When executing any tests based on this template two files need to be present in the same folder as the test being executed:
  
#Open file '''config/xml_input/xml_input.ini'''
+
<pre>
#Change line '''xml_input = ../../config/xml_input/input/SeedData''' to '''xml_input = ../../config/xml_input/input/SampleData'''. This value denotes the data import directory (to speed up the data import, you can also try setting the temporisation parameter to '0' so that no delay occurs between each insert - we used to set this to '500000' millisecs as OracleXE would periodically fail if the data import was too fast).
+
bootstrap_doctrine.php
#In your Web browser, go to the "/xml_input" Apache alias once more (or refresh this page).  
+
truncateDataTables.xml
#The data files located in your configured 'input_xml' directory will be imported into your database.  
+
</pre>
#Browse to your gocdb home page, the sample data will now be shown in the web portal.
 
#Repeat this procedure whenever you need to import other/custom XML data files into your gocdb database.
 
  
==== XML Input Data Format  ====
+
The first holds your connection details and the second will initialize the database with the correct tables ready for testing.
  
#The structure of the XML input data files used for the Seed and Sample is described at [[GOCDB/Release4/XML Input]]
+
You can write multiple test functions at the bottom of this skeleton file.
  
=== Testing  ===
+
==== Doctrine Test Example ====
  
If all of the above has been successful then your web portal instance should be ready for testing. Browse to the web_portal Apache alias created in an earlier step and see the data that have been imported.  
+
'DoctrineTestExample.php' shows the following example test. In this test we will create a site, populate the site with some data and then assert that the data is correct.  
  
If everything has been successful up to this point then you have successfully completed the installation instructions for GOCDB-4!
+
'''This test will make use of the 'TestUtil.php' file which contains a few helper methods to quickly create objects. This file is located in GocDBSrc/test/doctrine. When following this tutorial execute 'DoctrineTestExample.php' from GocDBSrc/test/doctrine.'''
  
=== Possible errors and troubleshooting  ===
+
Creating the site:
 +
<pre>
 +
    public function testDoctrineExample() {
 +
        print __METHOD__ . "\n";
  
This documentation is still in a draft state and you might therefore not find here the error you are experiencing. Should that be the case, please contact us so that we can work together on it and feed documentation with this error and its solution.
+
        //Create a site
 +
    $ourSite = TestUtil::createSampleSite("Our Example Site");
 +
 +
    }
 +
</pre>
  
==== TNS:listener errors (ORA-12516)  ====
+
We now have a site and will add some extra information to the site utilizing setters methods defined in the site entity which can be found in GocDBSrc/lib/Doctrine/entites. We will add an email, telephone and location to the site:
  
I get the following errors reported in the portal interface:
+
<pre>
<pre>Array ( [code] =&gt; 12516 [message] =&gt; ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] =&gt; 0 [sqltext] =&gt; ) aError is set: Array
+
//Create a site
Array ( [code] =&gt; 12516 [message] =&gt; ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] =&gt; 0 [sqltext] =&gt; ) aError is set: Array
+
    $ourSite = TestUtil::createSampleSite("Our Example Site");
Array ( [code] =&gt; 12516 [message] =&gt; ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] =&gt; 0 [sqltext] =&gt; ) aError is set: Array
+
Array ( [code] =&gt; 12516 [message] =&gt; ORA-12516: TNS:listener could not find available handler with matching protocol stack [offset] =&gt; 0 [sqltext] =&gt; ) aError is set: Array
+
//Set some details of the site
...more
+
$ourSite->setEmail("myTest@email.com");
</pre>  
+
$ourSite->setTelephone("012345678910");
If you are running on Oracle XE (the free version), you may see these errors. If so you should try to increase the number of sessions available in your database for use by GOCDB. In an Oracle sql client, try the following (set the session count to a number that is reasonable for you load):
+
$ourSite->setLocation("United Kingdom");
<pre>alter system set sessions = 400 scope=spfile;
+
</pre>
alter system set processes = 400 scope=spfile;
 
</pre>  
 
==== Deployment is interrupted because the webserver timed out  ====
 
  
The set of operations carried on during initial deployment can sometimes take a long time. Your webserver might be configured to time out after a certain time without answer, and if this time is too short this can result in a flawed deployment if the script stops half way through. The best thing to do in such a case is:
 
  
*From your miserably looking, half blank page where your setup script has stopped, click on the ''main admin menu'' item in the top menu bar
+
We now must persist this object in memory by calling '$this->em->persist($outSite);'
*You're back on the welcome page. Now click on the ''Object and Schema Management'' link
 
*Hopefully you'll see nice tabs and a list of possible actions (refer to the "administration" section for further details)  
 
*click on the ''wipe clean'' link
 
*go again through the setup section as described above
 
  
If this doesn't help, go to the ''Object and Schema Management'' page again and deploy each module one after the other (see the "administration" section for details).
+
However this now only exist in memory - it is not yet in the database. To check this we can query the database directly using the PDO connection we setup:
  
If you still get a server timeout, you need to reconfigure Apache so that the timeout value is higher. Check the Apache documentation for that
+
<pre>
 +
//Get the site ID from the object
 +
$siteId = $ourSite->getId(); 
 +
 +
//Get a database connection    
 +
    $con = $this->getConnection();
 +
 +
//Search the database for this site
 +
$sql = "SELECT 1 FROM sites WHERE ID = '$siteId'";
 +
$result = $con->createQueryTable('', $sql);
  
==== Portal doesn't work  ====
+
        //We expect this query to return no rows as the site does not exist in the database yet
 +
$this->assertEquals(0, $result->getRowCount());
 +
</pre>
  
Even if everything goes OK during the setup it might happen that you can't browse the web portal (and get some PHP fatal errors along the way). The most probable cause for that is that https is not fully enabled on the server. Check your Apache SSL configuration and Apache documentation. If this is not the cause of the problem pleas contact us (see "support" section below) with the error message you are getting
+
When running the test at this point the outcome should be:
 +
<pre>
 +
OK (1 test, 1 assertion)
 +
</pre>
  
==== Support  ====
+
Our assertion was correct, there were no sites with that ID in the database yet as it has not been committed, the site only exists in memory at the moment. To commit the site to the database we must call:
  
If you need to contact us, please do so with as many details as possible so that we can track down with you what's gone wrong. Support requests should be sent to [mailto:gocdb-admins_AT_mailtalk.ac.uk gocdb-admins_AT_mailtalk.ac.uk]
+
<pre>
 +
      $this->em->flush();
 +
</pre>
  
== Administration  ==
+
Now if we run the query again we expect one result:
 +
<pre>
 +
//Search the database for this site again
 +
$sql = "SELECT 1 FROM sites WHERE ID = '$siteId'";
 +
$result = $con->createQueryTable('', $sql);
 +
 +
//We expect this query to return 1 rows as the site now exists in the database
 +
$this->assertEquals(1, $result->getRowCount());
  
=== Object management  ===
+
</pre>
  
== Customisation  ==
+
This shows the process of creating an entity, using its get and set methods to enter and retrieve data and then committing it to the database. It also shows the process of using the PDO connection to check that the data exists in the database as we expect.
  
[[GOCDB/Release4/Customisation]]
+
-->
  
=== Adding a local schema ===
 
= Custom Features =
 
Custom features are those that aren't enabled on the central GOCDB but can be enabled on a standalone GOCDB instance.
 
  
== Siteless Endpoints ==
 
=== Introduction ===
 
Siteless endpoints allows service groups to host service endpoints that do not have a hosting physical site. In the standard GOCDB configuration this isn't allowed - a service endpoint must belong to a physical site.
 
  
=== Configuration ===
+
In config/local_info.xml siteless endpoints can be enabled as follows:
 
  
{{{
+
[[Category:GOCDB]]
<optional_features>
 
            <!-- If false then a service endpoint must belong to a physical
 
                site.
 
                If true an option will be available to add a brand new 
 
                SE to a service group and the new SE won't belong to a
 
                physical site. -->
 
            <siteless_endpoints>false</siteless_endpoints>
 
        </optional_features>
 
}}}
 

Latest revision as of 16:02, 9 November 2016