Difference between revisions of "GOCDB/Regional Module Technical Documentation"

From EGIWiki
Jump to: navigation, search
(PHPUnit)
m (Download)
 
(62 intermediate revisions by 3 users not shown)
Line 3: Line 3:
 
== Download  ==
 
== Download  ==
  
'''LATEST VERSION: GOCDBv5''' <br>  
+
'''LATEST VERSION: GOCDBv5.6''' <br>  
 +
https://github.com/GOCDB/gocdb
  
*A packaged V5 release will also be made available for download soon.  
+
For install see:
*In the meantime, early adopters can check out the v5 src from svn:
+
https://github.com/GOCDB/gocdb/blob/master/INSTALL.md
<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/branches/gocdb/Doctrine%20Web%20Portal
+
 
 +
<!--
 +
*Packaged V5.2: https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-5.2/GocDB-5.2.zip
 +
*svn v5.2:
 +
<pre>svn co https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/tags/gocdb/GOCDB-5.2
 
</pre>  
 
</pre>  
Note, if you are asked to authenticate with a client certificate as below, you can dismiss by pressing Enter  
+
*Latest dev/trunk svn:
 +
<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/branches/gocdb/MultipleEndpointsGocDB/</pre>
 +
-->
 +
 
 +
<!--<pre>svn checkout https://www.sysadmin.hep.ac.uk/svn/grid-monitoring/branches/gocdb/Doctrine%20Web%20Portal</pre> -->
 +
<!--Note, for svn co, if you are asked to authenticate with a client certificate as below, you can dismiss by pressing Enter  
 
<pre>Authentication realm: https://www.sysadmin.hep.ac.uk:443
 
<pre>Authentication realm: https://www.sysadmin.hep.ac.uk:443
 
Client certificate filename: &lt;ENTER to dismiss&gt;
 
Client certificate filename: &lt;ENTER to dismiss&gt;
Line 18: Line 28:
  
 
<br>  
 
<br>  
 +
-->
 +
<br>
  
<br>
+
<!--
 
+
== Prerequisites==
== Deployment  ==
 
'''<font color="red">(Under Construction - more details comming soon)</font>'''
 
 
=== System prerequisites  ===
 
=== System prerequisites  ===
  
Line 29: Line 39:
 
==== Database  ====
 
==== Database  ====
  
Version 5 of GocDB uses Doctrine ORM (Object Relational Mapping) as it's interface to the datastore. This allows GocDB to be deployed to either a Oracle, MySQL or SQLite database (Postgres comming soon).  
+
Version 5 of GocDB uses Doctrine ORM (Object Relational Mapping) as it's interface to the datastore. This allows GocDB to be deployed to either a Oracle or MySQL database (Doctrine also supports other DBs but these are untested).  
  
 
Oracle:  
 
Oracle:  
Line 44: Line 54:
 
*Required space: See MySQL install guide:http://dev.mysql.com/doc/refman/5.7/en/installing.html
 
*Required space: See MySQL install guide:http://dev.mysql.com/doc/refman/5.7/en/installing.html
  
<br>  
+
<br>
  
SQLite:
 
 
*Required database: SQLite 3.x.x or higher. SQLite is produced as an open source database and all versions are free to use.
 
*Required: See the SQLite documentation: www.sqlite.org/docs.html
 
<br>
 
 
==== Web frontend  ====
 
==== Web frontend  ====
  
Line 56: Line 61:
  
 
*'''Apache http server'''&nbsp;version 2.2 or higher.  
 
*'''Apache http server'''&nbsp;version 2.2 or higher.  
*'''PHP'''&nbsp;version 5.3 or higher (version &lt;=5.2 has some OOP related bugs).  
+
*'''PHP''' version 5.3 or higher (version &lt;=5.2 has some OOP related bugs).  
**'''PHP'''&nbsp;oci8 extension<br>
+
**If using Oracle: '''PHP oci8 extension''' and Oracle Instant client v10 or higher (downloadable from Oracle website)
*<span style="line-height: 1.5em;">a X509 host certificate for the machine</span>
+
**'''libxml2 and DOM support''' for PHP (Note: On RHEL, PHP requires the PHP XML RPM to be installed for this component to function).  
 +
**'''OpenSSL Extension''' for PHP
 +
*'''An X509 host certificate''' for the machine
  
<br>  
+
<br>
  
 
==== Optional  ====
 
==== Optional  ====
Line 72: Line 79:
 
=== Database preparation  ===
 
=== Database preparation  ===
  
GocDB v5 comes with deploy script which will setup the DB schema for GocDB (i.e. export the DDL to create tables). The only preperation is that you have setup a database user/account in your chosen RDBMS and have the user access details which are required later in the installation when configuring GocDB for deployment.&nbsp;
+
GocDB v5 comes with deploy script which will create the DB schema for GocDB (i.e. export the DDL to create the tables and sequences in the DB). The only preperation is that you have setup a database user/account in your chosen RDBMS and have the user access details which are required later in the installation when configuring GocDB for deployment.&nbsp;
  
<br>
+
Details for creating GOCDB user on specific RDBMS:
 +
* [[GOCDB/Regional_Module_Technical_Documentation/DeployOnOracle|Deploy on Oracle Details]]
 +
* [[GOCDB/Regional_Module_Technical_Documentation/DeployOnMySQL|Deploy on MySQL Details]]
  
 
=== GOCDB file system  ===
 
=== GOCDB file system  ===
Line 102: Line 111:
 
<pre>$pear install --alldeps doctrine/DoctrineORM
 
<pre>$pear install --alldeps doctrine/DoctrineORM
 
</pre>  
 
</pre>  
Verify the Doctrine installation by running "doctrine --version" on the command line:  
+
Verify the Doctrine installation by running "doctrine --version" on the command line (or doctrine.bat --version if on Win):  
 
<pre>$ doctrine --version
 
<pre>$ doctrine --version
 
Doctrine Command Line Interface version 2.3.3
 
Doctrine Command Line Interface version 2.3.3
 
</pre>  
 
</pre>  
 
[http://docs.doctrine-project.org/en/latest/ More information on Doctrine can be found here at the Doctrine2 site.]
 
[http://docs.doctrine-project.org/en/latest/ More information on Doctrine can be found here at the Doctrine2 site.]
 
=== Apache  ===
 
 
Apache http server version 2.2 or higher.
 
  
 
=== PHPUnit  ===
 
=== PHPUnit  ===
Line 128: Line 133:
  
 
=== Database Connection ===
 
=== Database Connection ===
You should now have all the required blocks in place to deploy GOCDB V5. To Deploy the database schema to your desired database using Doctrine navigate to to gocDBSrc/lib/Doctrine folder. In this folder you will configure the database connection details for your database. A template file called ''bootstrap_doctrine_TEMPLATE.php'' is provided. In this file you will find three blocks of code commented out, once for each of the supported databased, SQLite, Oracle and MySQL:
+
You should now have all the required blocks in place to deploy GOCDB V5. To Deploy the database schema to your desired database using Doctrine:
 +
* Navigate to to '<gocDBSrcHome>/lib/Doctrine' folder.  
 +
* Locate the provided template file: ''bootstrap_doctrine_TEMPLATE.php''. In this file you will find three blocks of code commented out, once for each of the supported databased, SQLite, Oracle and MySQL as shown below:
 +
* '''Copy this file to''' ''bootstrap_doctrine.php'' in the same dir as the template file and modify to specify your chosen DB connection details.
  
 
<pre>
 
<pre>
Line 134: Line 142:
  
 
         ///////////////////////SQLITE CONNECTION DETAILS/////////////////////////////////////////////
 
         ///////////////////////SQLITE CONNECTION DETAILS/////////////////////////////////////////////
// $conn = array(
+
//     $conn = array(
// 'driver' => 'pdo_sqlite',
+
//         'driver' => 'pdo_sqlite',
// 'path' => __DIR__ . '/db.sqlite',
+
//         'path' => __DIR__ . '/db.sqlite',
// );
+
//     );
 
         /////////////////////////////////////////////////////////////////////////////////////////////
 
         /////////////////////////////////////////////////////////////////////////////////////////////
  
Line 158: Line 166:
 
 
 
///////////////////////MYSQL CONNECTION DETAILS////////////////////////////////////////////
 
///////////////////////MYSQL CONNECTION DETAILS////////////////////////////////////////////
//$conn = array(
+
// $conn = array(
 
// 'driver' => 'pdo_mysql',
 
// 'driver' => 'pdo_mysql',
 
// 'user' => 'doctrine',
 
// 'user' => 'doctrine',
Line 164: Line 172:
 
// 'host' => 'localhost',
 
// 'host' => 'localhost',
 
// 'dbname' => 'doctrine'
 
// 'dbname' => 'doctrine'
//);
+
// );
 
         /////////////////////////////////////////////////////////////////////////////////////////////
 
         /////////////////////////////////////////////////////////////////////////////////////////////
 
</pre>
 
</pre>
 
Once you have input your connection details for your database save this file as ''bootstrap_doctrine.php'' in the same location as the template file.
 
  
  
Line 176: Line 182:
 
</pre>
 
</pre>
  
=== Compiled Entities ===
+
=== Deploying GOCDB  ===
For local test machines this step isn't mandatory. For production servers it is strongly recommended. When Doctrine uses and entity it creates a compiled version of the entity which by default is stored in the machines temporary folder. This is un-suitable for production machines as this may well be emptied and then cause GocDB to cease working. To avoid this uncomment the following line in bootstrap_doctrine.php:
 
<pre>
 
$config->setProxyDir(__DIR__."/compiledEntities");
 
</pre>
 
  
At this point no further action is required, but later we will need to create this folder and populate it with the compiled entities data.
+
GocDB can be deployed as a blank instance ready for use or as a sample instance with a small amount of example data to demonstrate GocDB. The deploy script can be found at ''/lib/Doctrine/deploy/deploy.sh'':<br>
 
 
=== Deploying GOCDB  ===
 
  
GocDB can be deployed as a blank instance ready for use or as a sample instance with a small amount of example data to demonstrate GocDB. The deploy script can be found in ''/lib/Doctrine/deploy'' folder:<br>
+
'''Windows Users:''' These scripts are designed for deployment on a Linux machine (Cygwin on Windows is ok). 
  
<pre>deploy.sh</pre>
 
  
 
By supplying a flag when executing the script you can deploy either a new empty database or a sample database eg:
 
By supplying a flag when executing the script you can deploy either a new empty database or a sample database eg:
Line 206: Line 205:
  
 
=== Apache Configuration  ===
 
=== Apache Configuration  ===
 +
 +
Now that your database is deployed the final step is to configure Apache.
 +
 +
==== /etc/sysconfig/httpd - Oracle configuration ====
 +
 +
The Oracle environment must be set correctly before starting Apache so that PHP OCI8 works correctly. In general you should set the same variables that are set by the ''$ORACLE_HOME/bin/oracle_env.sh'' script. The necessary environment variables can be set in Apache's environment configuration file.
 +
On RedHat Linux and its derivatives with the default httpd package, this is ''/etc/sysconfig/httpd''.
 +
<pre>
 +
export ORACLE_HOME=/u01/app/oracle/product/12.1    # Adjust pathname to match your installation
 +
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
 +
</pre>
  
 
==== httpd.conf  ====
 
==== httpd.conf  ====
 
Now that your database is deployed the final step is to configure Apache.
 
  
 
A few modules need to be enabled for GOCDB to work, check these are enabled on your Apache and if not enable them.&nbsp;  
 
A few modules need to be enabled for GOCDB to work, check these are enabled on your Apache and if not enable them.&nbsp;  
Line 215: Line 223:
 
In ''httpd.conf'' enable ssl_module by un-commenting these lines:  
 
In ''httpd.conf'' enable ssl_module by un-commenting these lines:  
 
<pre>LoadModule ssl_module modules/mod_ssl.so</pre>  
 
<pre>LoadModule ssl_module modules/mod_ssl.so</pre>  
and
+
 
<pre>extension=php_openssl.dll</pre>
+
 
Depending on the database you are using the supporting module may also need to be enabled. For example Oracle requires the oracle module to be uncommented:
+
 
<pre>extension=php_oci8_11g.dll</pre>
 
Take the file ''gocdbssl.conf'' and place this file in the apache/conf/extra folder. In ''httpd.conf'' include the ''gocdbssl.con'' by adding the following line under the heading ''Supplemental Configuration'':
 
<pre>Include conf/extra/gocdbssl.conf
 
</pre>
 
 
==== gocdbssl.conf  ====
 
==== gocdbssl.conf  ====
 
+
A sample ''gocdbssl.conf'' is provided in GocDBSrc/config. This file will define the alias and SSL settings used by Apache to connect to GocDB.   
This file can be found in GocDBSrc/config. This file will define the alias and SSL settings used by Apache to connect to GocDB.   
 
  
 
===== Paths  =====
 
===== Paths  =====
Line 260: Line 263:
  
 
=== Compiled Entities ===
 
=== Compiled Entities ===
As described in the previous compiled entities section this step is not required for local test deployments of GocDB but strongly recommend for production deployments. You should have uncommented the line in ''bootstrap_doctrine.php'' that sets the proxy directory for the entities:
+
This step is optional, but for production servers this step is strongly recommended. When Doctrine uses an entity it creates a compiled version of the entity which by default is stored in the machines temporary folder. This is un-suitable for production machines as this may well be emptied. To avoid this uncomment the following line in bootstrap_doctrine.php:
 
<pre>
 
<pre>
 
$config->setProxyDir(__DIR__."/compiledEntities");
 
$config->setProxyDir(__DIR__."/compiledEntities");
Line 267: Line 270:
 
You now need to create this folder in the same directory as the ''bootstrap_doctrine.php'' file:
 
You now need to create this folder in the same directory as the ''bootstrap_doctrine.php'' file:
 
<pre>
 
<pre>
$ls
 
bootstrap_doctrine.php  bootstrap.php  cli-config.php  deploy  entities  README.txt
 
 
$mkdir compiledEntities
 
$mkdir compiledEntities
$ls
 
 
</pre>
 
</pre>
  
Once this folder is created it needs it's ownership changed to be owned by the apache user, the exact command may change depending on your apache server:
+
Once this folder is created it, your apache server needs write permissions (the exact command may change depending on your apache server):
  
 
<pre>
 
<pre>
Line 279: Line 279:
 
</pre>
 
</pre>
  
The compiled Entities folder should now be owned by Apache:
 
 
<pre>
 
<pre>
$ls -l
+
$ls -ltrh
 
total 28
 
total 28
-rw-r--r-- 1 root   root 3043 Nov 13 09:46 bootstrap_doctrine.php
+
drwxr--r-- 2 apache root 4.0K Nov 20 16:11 entities
-rwxrwxrwx 1 root  root 1620 Nov  7 12:08 bootstrap.php
+
-rw-r--r-- 1 apache root 131 Nov 20 16:11 README.txt
-rwxrwxrwx 1 root  root  219 Nov 7 12:08 cli-config.php
+
-rw-r--r-- 1 apache root  219 Nov 20 16:11 cli-config.php
drwxr-xr-x 2 apache root 4096 Nov 13 10:21 compiledEntities
+
-rw-r--r-- 1 apache root 1.6K Nov 20 16:11 bootstrap.php
drwxrwxrwx 4 root   root 4096 Nov 12 17:46 deploy
+
-rw-r--r-- 1 apache root 3.0K Nov 20 16:11 bootstrap_doctrine_TEMPLATE.php
drwxrwxrwx 2 root  root 4096 Nov 12 17:46 entities
+
-rw-r--r-- 1 apache root 2.8K Nov 27 09:38 bootstrap_doctrine.php
-rwxrwxrwx 1 root   root  131 Nov 12 17:37 README.txt
+
drwxr--r-- 2 apache root 4.0K Nov 27 09:39 compiledEntities
 
</pre>
 
</pre>
  
Finally command doctrine to generate the compiled entities and store them in the folder:
+
Finally use the doctrine command line client to generate the compiled entities and store them in the folder:
 
<pre>
 
<pre>
 
doctrine orm:generate-proxies compiledEntities/
 
doctrine orm:generate-proxies compiledEntities/
Line 302: Line 301:
  
 
You should now be able to navigate to the GocDB webportal on your host using the URL defined in your alias. You will need to install a browser certificate that is suitable for the SSL keys you defined for your host to be able to view GocDB.
 
You should now be able to navigate to the GocDB webportal on your host using the URL defined in your alias. You will need to install a browser certificate that is suitable for the SSL keys you defined for your host to be able to view GocDB.
 +
<pre>
 +
https://localhost/portal
 +
https://localhost/portal/GOCDB_monitor/index.php
 +
</pre>
 +
 +
====Local_Info.xml====
 +
GocDB picks up a number of its settings and variables from the local_info.xml file located in the ''config'' folder. The ''web_portal_url'' will be output in the PI to create links from PI objects back to Portal views. The ''pi_url'' and ''server_base_url'' will both be used by the monitor. The monitor is a quick look status check for GocDB. The mointor can be found at your web_portal_url/GOCDB_monitor/. If these URL's are not set in the local config this feature will not work correctly.
 +
 +
''default_scope'' defines which scope if any will be used if no scope is specified when running PI queries. This can be left blank or set to a scope of your choosing. ''default_scope_match'' is again used if no scope_match parameter is supplied with PI queries. This can be either all or any. The ''minimum_scopes'' section declares how many scopes an NGI, site, service or service group must have at creation. This can be left as 0 or set as 1 if you want all your users entities to belong to at least one scope or more as dictated by your use of GocDB.
  
==== Setup an Admin User ====
+
It's important at this point to understand how the scopes work with GocDB v5 especially in relation to the output of the PI. If you specify a default scope within your local_info but none of your entities have this scope then nothing will show in the PI. For an in-depth look at the scopes mechanism the section please read the section 'Multiple Scopes and Projects' in the [https://wiki.egi.eu/w/images/d/d3/GOCDB5_Grid_Topology_Information_System.pdf GOCDB5 Grid Topology Information System] document on page 5.
 +
 
 +
====Setup an Admin User====
  
 
To get started with GocDB you will need an admin user. This is done by first registering as a user on GocDB by clicking the 'Register' link on the left menu bar at the bottom. Once you have registered yourself you will then need to set yourself as an admin. To do this you need to change the user record directly in your database. The users table has a field called 'isAdmin' which by default is set to 0. To change a user to admin set this to 1. Below is a sample of the SQL query used when using an Oracle database to set a user as admin.  
 
To get started with GocDB you will need an admin user. This is done by first registering as a user on GocDB by clicking the 'Register' link on the left menu bar at the bottom. Once you have registered yourself you will then need to set yourself as an admin. To do this you need to change the user record directly in your database. The users table has a field called 'isAdmin' which by default is set to 0. To change a user to admin set this to 1. Below is a sample of the SQL query used when using an Oracle database to set a user as admin.  
Line 316: Line 326:
  
 
GocDB v5 comes with a suite of tests that can be run to validate the install and check that Doctrine and your chosen database are operating as expected. These tests require PHPUnit, the install instructions for this can be found in section 3.6. The tests can be found in the GocDB/tests/ folder.  
 
GocDB v5 comes with a suite of tests that can be run to validate the install and check that Doctrine and your chosen database are operating as expected. These tests require PHPUnit, the install instructions for this can be found in section 3.6. The tests can be found in the GocDB/tests/ folder.  
 +
 +
===Install the Oracle OCI drivers for PDO===
 +
To run tests you will also need to install the Oracle OCI8 PDO (PHP Data Object) drivers for PHP since the DBUnit tests
 +
use PDO for their execution to validate that the results that Doctrine is returning are true. To install the OCI8 drivers for pdo. See: http://www.php.net/manual/en/pdo.drivers.php
 +
 +
You can test that you have the OCI8 pdo drivers installed via php info. This can be called on the command line with $php -i or via a script with phpinfo(). There will be a section that lists your enabled pdo drivers, e.g. as below:
 +
<pre>
 +
            |-----------------------------|
 +
            |            PDO            |
 +
            |-----------------------------|
 +
            | PDO support enabled  |
 +
            | PDO drivers mysql,oci |
 +
            |-----------------------------|
 +
</pre>
  
 
=== Deploying a Test Database ===
 
=== Deploying a Test Database ===
  
We recommend that you deploy a second database that will be used for testing. Once you have created your second database input the connection details into the ''bootstrap_doctrine_TEMPLATE.php'' file found in GocDB/tests/doctrine. Within this file there are 3 sets of connection configurations, once for MySQL, Oracle and SQLite. Once you have input your details save this file as ''bootstrap_doctrine.php''. Complete the section for your database and ensure that the other connection details are deleted or commented out. In addition to the standard Doctrine connection to the database the tests make use of a PDO connection to validate that the results that Doctrine is returning are true. To setup the pdo connection open ''bootstrap_pdo_TEMPLATE.php''. Within this file you will again find 3 connection blocks. Complete the details for the connection details for your database and ensure that the other 2 connections are either deleted or commented out and then save this file as ''bootstrap_pdo.php''.
+
We STRONGLY recommend that you deploy a second database with different connection details that will be used for testing.  
 +
* Create a second DB account/user for your test database (e.g. GOCDB5TEST).
 +
* Copy ''bootstrap_doctrine_TEMPLATE.php'' file found in tests/doctrine to ''bootstrap_doctrine.php'' (in the same dir).  
 +
* Modify ''bootstrap_doctrine.php'' to specify your GOCDB5TEST account/user. Complete the section for your database and ensure that the other connection details are deleted or commented out.
 +
* Copy ''bootstrap_pdo_TEMPLATE.php'' to ''bootstrap_pdo.php''. Complete the details for the connection details for your database and ensure that the other 2 connections are either deleted or commented out, see below:
  
 
<pre>
 
<pre>
Line 343: Line 371:
 
=== Running the Test Suite ===
 
=== Running the Test Suite ===
  
A initial test suite is provided that will excercise core functions of GocDB and Doctrine, this suite can be ran using the command:
+
A initial test suite is provided that will excercise core functions of GocDB and Doctrine. This suite can be ran using the command:
 
<pre>
 
<pre>
 
phpunit DoctrineTestSuite1.php
 
phpunit DoctrineTestSuite1.php
 
</pre>
 
</pre>
  
You will be prompted to continue as this operation will drop any tables in your test database and then recreate the schema ready to run the tests. These tests should return with no errors to demonstrate that GocDB and Doctrine are installed correctly and are working as expected.  
+
You will be prompted to continue as this operation will drop any tables in your test database and then recreate the schema ready to run the tests. These tests should return with no errors to demonstrate that GocDB and Doctrine are installed correctly and are working as expected.
  
 
=== Running Individual Tests ===
 
=== Running Individual Tests ===
  
Individual tests can be run by calling phpunit testName.php. It is advised that at the beginning of a testing session you drop and recreate the database by running:
+
Individual tests can be run by calling:
 +
<pre>phpunit <testNameFile>.php</pre>
 +
It is advised that at the beginning of a testing session you drop and recreate the database by running the recreateTestDB.sh script.
 +
 
 +
===Recreating the Test DB===
 +
You can run the following script found in the ''tests/doctrine'' folder to Drop and Recreate the test database whenever required:  
 
<pre>
 
<pre>
 
./recreateTestDB.sh
 
./recreateTestDB.sh
 
</pre>
 
</pre>
Found in the GocDB/tests/doctrine folder.
 
  
 
=== Writing Tests ===
 
=== Writing Tests ===
Line 370: Line 402:
 
<pre>
 
<pre>
 
bootstrap_doctrine.php
 
bootstrap_doctrine.php
truncateDataTables
+
truncateDataTables.xml
 
</pre>
 
</pre>
  
 
The first holds your connection details and the second will initialize the database with the correct tables ready for testing.  
 
The first holds your connection details and the second will initialize the database with the correct tables ready for testing.  
  
You can write multiple test functions at the bottom of this skeleton file.  
+
You can write multiple test functions at the bottom of this skeleton file.
  
 
==== Doctrine Test Example ====
 
==== Doctrine Test Example ====
Line 450: Line 482:
 
This shows the process of creating an entity, using its get and set methods to enter and retrieve data and then committing it to the database. It also shows the process of using the PDO connection to check that the data exists in the database as we expect.  
 
This shows the process of creating an entity, using its get and set methods to enter and retrieve data and then committing it to the database. It also shows the process of using the PDO connection to check that the data exists in the database as we expect.  
  
 
+
-->
  
  

Latest revision as of 16:02, 9 November 2016