The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.
UsingCreamUMD3WithApelUMD2
Jump to navigation
Jump to search
Installation of previous APEL
- Add UMD-2 repos
wget http://repository.egi.eu/sw/production/umd/2/repofiles/sl5/UMD-2-base.repo -O /etc/yum.repos.d/UMD-2-base.repo wget http://repository.egi.eu/sw/production/umd/2/repofiles/sl5/UMD-2-updates.repo -O /etc/yum.repos.d/UMD-2-updates.repo
- Install the apel parser package, there is one flavor for each LRMS (pbs, sge and lsf)
$ yum install glite-apel-pbs Loaded plugins: kernel-module, priorities, protectbase Repository EGI-trustanchors is listed more than once in the configuration UMD-2-base | 1.9 kB 00:00 UMD-2-updates | 1.9 kB 00:00 552 packages excluded due to repository priority protections 0 packages excluded due to repository protections Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package glite-apel-pbs.noarch 0:2.0.6-8.sl5 set to be updated --> Processing Dependency: glite-apel-core >= 2.0.14-4 for package: glite-apel-pbs --> Running transaction check ---> Package glite-apel-core.noarch 0:2.0.14-5.sl5 set to be updated --> Finished Dependency Resolution Beginning Kernel Module Plugin Finished Kernel Module Plugin Dependencies Resolved ====================================================================================================== Package Arch Version Repository Size ====================================================================================================== Installing: glite-apel-pbs noarch 2.0.6-8.sl5 UMD-2-updates 24 k Installing for dependencies: glite-apel-core noarch 2.0.14-5.sl5 UMD-2-base 1.0 M Transaction Summary ====================================================================================================== Install 2 Package(s) Upgrade 0 Package(s) Total download size: 1.1 M Is this ok [y/N]: y Downloading Packages: (1/2): glite-apel-pbs-2.0.6-8.sl5.noarch.rpm | 24 kB 00:00 (2/2): glite-apel-core-2.0.14-5.sl5.noarch.rpm | 1.0 MB 00:01 ------------------------------------------------------------------------------------------------------ Total 654 kB/s | 1.1 MB 00:01 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing : glite-apel-core 1/2 Installing : glite-apel-pbs 2/2 Installed: glite-apel-pbs.noarch 0:2.0.6-8.sl5 Dependency Installed: glite-apel-core.noarch 0:2.0.14-5.sl5 Complete!
Configuration
EMI-3 CREAM does not configure the apel parsers with YAIM, but the functions for doing so are still available so it can be executed directly:
$ /opt/glite/yaim/bin/yaim -r -s site-info.def -f config_apel_pbs
INFO: Configuring HOST: test06.egi.cesga.es
INFO: Using site configuration file: /opt/glite/yaim/etc/site-info.def
INFO:
###################################################################
. /'.-. ')
. yA,-"-,( ,m,:/ ) .oo. oo o ooo o. .oo
. / .-Y a a Y-. 8. .8' 8'8. 8 8b d'8
. / ~ ~ / 8' .8oo88. 8 8 8' 8
. (_/ '====' 8 .8' 8. 8 8 Y 8
. Y,-''-,Yy,-.,/ o8o o8o o88o o8o o8o o8o
. I_))_) I_))_)
current working directory: /opt/glite/yaim/node-info.d
site-info.def date: May 6 10:19 /opt/glite/yaim/etc/site-info.def
yaim command: -r -s /opt/glite/yaim/etc/site-info.def -f config_apel_pbs
log file: /opt/glite/yaim/bin/../log/yaimlog
Mon May 6 11:05:16 CEST 2013 : /opt/glite/yaim/bin/yaim
Installed YAIM versions:
glite-apel-yaim ame:-ame:
glite-yaim-bdii 4.3.13-1
glite-yaim-core 5.1.1-1
glite-yaim-cream-ce 4.4.1-4
glite-yaim-mpi 1.1.12-1
glite-yaim-torque-server 5.1.0 -1
glite-yaim-torque-utils 5.1.1-2
####################################################################
INFO: The default location of the grid-env.(c)sh files will be: /usr/libexec
INFO: Sourcing the utilities in /opt/glite/yaim/functions/utils
INFO: Detecting environment
INFO: Assuming the node types: CE_torque
INFO: Using hostname: test06.egi.cesga.es
WARNING: config_apel_pbs function is not supported by CE_torque!
WARNING: Anyway you can force this operation.
WARNING: Would you really run config_apel_pbs on CE_torque? [y/]:
INFO: Executing function: config_apel_pbs_check
INFO: Executing function: config_apel_pbs_setenv
INFO: Executing function: config_apel_pbs
INFO: YAIM terminated succesfully.
That should create an apel configuration file at /etc/glite-apel-pbs/parser-config-yaim.xml and a cron file for parsing the records (/etc/cron.d/glite-apel-pbs-parser)
The configuration file does not point to the correct prefix for blah logs. The <BlahdLogPrefix> in the configuration file should be changed to:
<BlahdLogPrefix>blahp.log-</BlahdLogPrefix>
Test
Running manually the contents of the cron file:
APEL_HOME=/ /usr/bin/apel-pbs-log-parser -f /etc/glite-apel-pbs/parser-config-yaim.xml >> /var/log/apel.log 2>&1
Log file contents:
Mon May 6 10:32:02 UTC 2013: apel-pbs-log-parser - Read-in configuration: [quiet, p, inspectTables, c, e, recursiveEventFiles, b, recursiveBlahdFiles] [DBPassword=****, site=CESGA-EGEE, pbslog=/var/torque/server_priv/accounting, GIIS=test06.egi.cesga.es, CEhost=test06.egi.cesga.es, blahdlogPrefix=blahp.log-, DBURL=jdbc:mysql://test07.egi.cesga.es:3306/accounting, DBUsername=accounting, timeZone=UTC, blahdlog=/var/log/cream/accounting] Mon May 6 10:32:02 UTC 2013: apel-pbs-log-parser - ------ Starting the apel application ------ Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - **** APEL is examining the schema **** Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the LcgRecords table Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The LcgRecords schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the BlahdRecords table Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The BlahdRecords schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the LcgProcessedFiles table Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The LcgProcessedFiles schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the SpecRecords table for patch 28593 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the SpecRecords table for patch 65723 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The SpecRecords schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the GkRecords table Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The GkRecords schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the MessageRecords table Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The MessageRecords schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - **** Schema checks complete **** Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Finding CPU spec values from GIIS server: test06.egi.cesga.es Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - LDAP URL: ldap://test06.egi.cesga.es:2170 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required test06.egi.cesga.es:8443/cream-pbs-GRID_dteam:2395 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required test06.egi.cesga.es:8443/cream-pbs-GRID_ops:2395 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required test06.egi.cesga.es:8443/cream-pbs-GRID_iberibeu:2395 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The ClusterId : SpecInt pair are latest, no update required test06.egi.cesga.es:8443/cream-pbs-GRID_opsibeu:2395 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Reprocess disabled, checking new event logs only Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - **** Updating PBS end event table (EventRecords) **** Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Ignoring already processed event log file: test06.egi.cesga.es /var/torque/server_priv/accounting/20130430 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Processing batch log file: test06.egi.cesga.es /var/torque/server_priv/accounting/20130506 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Event records inserted: 9 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Checking the BlahdRecords table Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - The BlahdRecords schema is up-to-date Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Reprocess disabled, checking new event logs only Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Processing /var/log/cream/accounting/blahp.log-20130506 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Processing /var/log/cream/accounting/blahp.log-20130430 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - Blahd records inserted: 12 Mon May 6 10:32:03 UTC 2013: apel-pbs-log-parser - ------ Processing finished ------