https://wiki.egi.eu/w/api.php?action=feedcontributions&user=Perl&feedformat=atomEGIWiki - User contributions [en]2024-03-28T22:57:44ZUser contributionsMediaWiki 1.37.1https://wiki.egi.eu/w/index.php?title=EGI-InSPIRE:Jra1_2013-06-25&diff=56996EGI-InSPIRE:Jra1 2013-06-252013-06-25T15:03:54Z<p>Perl: /* 3.3 TJRA1.4 (Accounting for new resource types) */</p>
<hr />
<div>[[Inspire_JRA1_weekly_reports| << JRA1 weekly reports main page]] <br />
<br />
{{TOC_right}} <br />
<br />
== 1. Progress on Milestones Deliverables ==<br />
<br />
== 2. Critical Issues ==<br />
<br />
== 3. Major work done during the past week by task ==<br />
<br />
=== 3.1 TJRA1.1 ===<br />
<br />
=== 3.2 TJRA1.2 TJRA1.3 TJRA1.5 ===<br />
<br />
==== 3.2.1 SAM ====<br />
* Validation of SAM Update 22<br />
* SAM Update 22 installed in ops-monitor-preprod.cern.ch<br />
* SAM Update 22 testing release has been announced (call for volunteers willing to test it before it enters SR)<br />
<br />
==== 3.2.2 GOCDB ====<br />
<br />
==== 3.2.3 GGUS ====<br />
* Ticket monitoring<br />
* Weekly shopping list phone meeting<br />
* Preparing next release and implementing feature requests<br />
* Prepared slides for the URT meeting on 2013-06-24<br />
<br />
==== 3.2.4 Ops Portal ====<br />
<br />
==== 3.2.5 Accounting Portal ====<br />
<br />
==== 3.2.6 Metrics Portal ====<br />
<br />
=== 3.3 TJRA1.4 (Accounting for new resource types) ===<br />
* A new version of the APEL client and server code has been produced, in response to a security request, https://rt.egi.eu/rt/Ticket/Display.html?id=5615.<br />
* The code for the dbunloader is currently being worked on to provide enhancements to the Regional APEL Server, these updates will be tested during the next 2 weeks. Enhancements are: <br />
1) Sites can include and exclude certain sites in publishing to the central server, as well as including/excluding VOs. <br />
2) Additional functionality so that a server can publish the same summaries to several receiving SSMs (this is in response to the requests from OSG/DGAS for summaries to be sent back to them as well as being published to the Accounting Portal. <br />
3) Improved error handling.<br />
* Creating test environment and testing code changes to implement the User Data Retention Policy for CPU Accounting from 1st July.<br />
* Cloud Accounting - the remaining sites publishing using the SSM 1.2 test server have been contacted again, with details of the migration procedure. The majority of Cloud Resource Providers now publish to the SSM 2.0 test server and their data is summarised twice per day and sent on to the test Cloud Accounting Portal.<br />
* Application Accounting: working on a test setup.<br />
<br />
== 4. Next Meetings ==<br />
<br />
[[Category:JRA1_weekly_report]]</div>Perlhttps://wiki.egi.eu/w/index.php?title=Application_Accounting&diff=46522Application Accounting2012-11-28T14:43:10Z<p>Perl: </p>
<hr />
<div>= Application Accounting =<br />
<br />
== Motivation ==<br />
<br />
In order to account software usage, we want to have a sensor that provides the following data: <br />
<br />
*Which software/binary was called? <br />
*How long did the software run? <br />
*What was the return value? <br />
*If applicable: What signal forced the termination (SIGINT/SIGKILL/...)<br />
<br />
== Repository ==<br />
<br />
The software can be found on&nbsp;[https://github.com/hperl/app-accounting github.com/hperl/app-accounting]. <br />
<br />
== Technique ==<br />
<br />
In order to account software usage, we need to intercept all calls that start a new process. The bash and most other shells use the libc "execve()" function to execute a program. Further, one can overwrite library functions by specifying another library in the LD_PRELOAD environment variable. The library built from "acc_preload.c" replaces "execve()" with the following instructions: <br />
<br />
#Fork the process, let the child execute the actual program <br />
#record the start time <br />
#wait on the child using "waitpid()", e.g. wait until the actual program ends <br />
#record the end time <br />
#construct an accounting record <br />
#send the accounting record to the server <br />
#special case: if the server does not respond, send the process to the background and wait until the server answers again<br />
<br />
The accounting bash (acc_bash) is a short and simple shell script that sets the environment variables and calls the bash: <br />
<pre>#!/bin/sh<br />
<br />
LD_PRELOAD=libacc_preload bash<br />
</pre> <br />
== Design Decisions ==<br />
<br />
=== Why client/server? ===<br />
<br />
I assumed that the actual accounting bash (acc_bash) would run directly on the grid nodes. The user mapping would probably require additional information from the head nodes or the queueing node. So the rudimentary information is assembled on the grid nodes (client) and then sent to the server, which can put additional information into the record and submit it to the accounting repository. <br />
<br />
=== Why&nbsp;overwrite a libc function? ===<br />
<br />
It is a good tradeoff between low-levelness and easy deployment. Directly intercepting the SYS_execve systemcall would have been even more elegant, but it would also require a kernel patch. Patching the kernel of every grid node is an overkill. With the current method, the deployment only requires the following: <br />
<br />
*Installation of a library (libacc_preload) <br />
*Installation of a binary (acc_bash)<br />
<br />
This has been packaged into a DEB and RPM in addition to a tarball. <br />
<br />
== Integration into APEL ==<br />
<br />
To be discussed.<br />
<br />
[[Category:Accounting]]</div>Perlhttps://wiki.egi.eu/w/index.php?title=Application_Accounting&diff=46521Application Accounting2012-11-28T14:40:32Z<p>Perl: </p>
<hr />
<div>= Application Accounting =<br />
<br />
== Motivation ==<br />
<br />
In order to account software usage, we want to have a sensor that provides the following data: <br />
<br />
*Which software/binary was called? <br />
*How long did the software run? <br />
*What was the return value? <br />
*If applicable: What signal forced the termination (SIGINT/SIGKILL/...)<br />
<br />
== Repository ==<br />
<br />
The software can be found on&nbsp;[https://github.com/hperl/app-accounting github.com/hperl/app-accounting]. <br />
<br />
== Technique ==<br />
<br />
In order to account software usage, we need to intercept all calls that start a new process. The bash and most other shells use the libc "execve()" function to execute a program. Further, one can overwrite library functions by specifying another library in the LD_PRELOAD environment variable. The library built from "acc_preload.c" replaces "execve()" with the following instructions: <br />
<br />
#Fork the process, let the child execute the actual program <br />
#record the start time <br />
#wait on the child using "waitpid()", e.g. wait until the actual program ends <br />
#record the end time <br />
#construct an accounting record <br />
#send the accounting record to the server <br />
#special case: if the server does not respond, send the process to the background and wait until the server answers again<br />
<br />
The accounting bash (acc_bash) is a short and simple shell script that sets the environment variables and calls the bash: <br />
<pre>#!/bin/sh<br />
<br />
LD_PRELOAD=libacc_preload bash<br />
</pre> <br />
== Design Decisions ==<br />
<br />
=== Why client/server? ===<br />
<br />
I assumed that the actual accounting bash (acc_bash) would run directly on the grid nodes. The user mapping would probably require additional information from the head nodes or the queueing node. So the rudimentary information is assembled on the grid nodes (client) and then sent to the server, which can put additional information into the record and submit it to the accounting repository. <br />
<br />
=== Why&nbsp;overwrite a libc function? ===<br />
<br />
It is a good tradeoff between low-levelness and easy deployment. Directly intercepting the SYS_execve systemcall would have been even more elegant, but it would also require a kernel patch. Patching the kernel of every grid node is an overkill. With the current method, the deployment only requires the following: <br />
<br />
*Installation of a library (libacc_preload) <br />
*Installation of a binary (acc_bash)<br />
<br />
This has been packaged into a DEB and RPM in addition to a tarball. <br />
<br />
== Integration into APEL ==<br />
<br />
To be discussed.</div>Perlhttps://wiki.egi.eu/w/index.php?title=Application_Accounting&diff=46520Application Accounting2012-11-28T14:40:15Z<p>Perl: Created page with "= &nbsp;Application Accounting = == Motivation == In order to account software usage, we want to have a sensor that provides the following data: *Which software/binary was ..."</p>
<hr />
<div>= &nbsp;Application Accounting =<br />
<br />
== Motivation ==<br />
<br />
In order to account software usage, we want to have a sensor that provides the following data: <br />
<br />
*Which software/binary was called? <br />
*How long did the software run? <br />
*What was the return value? <br />
*If applicable: What signal forced the termination (SIGINT/SIGKILL/...)<br />
<br />
== Repository ==<br />
<br />
The software can be found on&nbsp;[https://github.com/hperl/app-accounting github.com/hperl/app-accounting]. <br />
<br />
== Technique ==<br />
<br />
In order to account software usage, we need to intercept all calls that start a new process. The bash and most other shells use the libc "execve()" function to execute a program. Further, one can overwrite library functions by specifying another library in the LD_PRELOAD environment variable. The library built from "acc_preload.c" replaces "execve()" with the following instructions: <br />
<br />
#Fork the process, let the child execute the actual program <br />
#record the start time <br />
#wait on the child using "waitpid()", e.g. wait until the actual program ends <br />
#record the end time <br />
#construct an accounting record <br />
#send the accounting record to the server <br />
#special case: if the server does not respond, send the process to the background and wait until the server answers again<br />
<br />
The accounting bash (acc_bash) is a short and simple shell script that sets the environment variables and calls the bash: <br />
<pre>#!/bin/sh<br />
<br />
LD_PRELOAD=libacc_preload bash<br />
</pre> <br />
== Design Decisions ==<br />
<br />
=== Why client/server? ===<br />
<br />
I assumed that the actual accounting bash (acc_bash) would run directly on the grid nodes. The user mapping would probably require additional information from the head nodes or the queueing node. So the rudimentary information is assembled on the grid nodes (client) and then sent to the server, which can put additional information into the record and submit it to the accounting repository. <br />
<br />
=== Why&nbsp;overwrite a libc function? ===<br />
<br />
It is a good tradeoff between low-levelness and easy deployment. Directly intercepting the SYS_execve systemcall would have been even more elegant, but it would also require a kernel patch. Patching the kernel of every grid node is an overkill. With the current method, the deployment only requires the following: <br />
<br />
*Installation of a library (libacc_preload) <br />
*Installation of a binary (acc_bash)<br />
<br />
This has been packaged into a DEB and RPM in addition to a tarball. <br />
<br />
== Integration into APEL ==<br />
<br />
To be discussed.</div>Perlhttps://wiki.egi.eu/w/index.php?title=Accounting_Repository&diff=46515Accounting Repository2012-11-28T14:11:44Z<p>Perl: </p>
<hr />
<div>{{Template:Op menubar}} {{Template:Tools menubar}} <br />
{{TOC_right}}<br />
=Accounting portal =<br />
<br />
The accounting infrastructure is a complex system that involves various sensors in different regions, all publishing data to a central repository. The data is processed, summarized and displayed in the accounting portal, which acts as a common interface to the different accounting record providers and presents a homogeneous view of the data gathered and a user-friendly access to understanding resource utilization. <br />
<br />
* Go to the [http://accounting.egi.eu/ Accounting portal]<br />
<br />
== [[Accounting_Portal_Release_Notes|Release notes]] ==<br />
<br />
== Documentation ==<br />
<br />
*[http://www4.egee.cesga.es/accounting/links/acct_ibergrid06_final14.pdf Architecture and Implementation] <br />
*[http://www4.egee.cesga.es/accounting/links/accounting_portal_installation.pdf Installation Instruction] <br />
*[https://documents.egi.eu/document/517 Accounting Portal Roadmap]<br />
*[http://goc.grid.sinica.edu.tw/gocwiki/Outstanding_Accounting_Issues Outstanding Accounting Issues]<br />
<br />
= APEL repository =<br />
== [[APEL|APEL documentation]] ==<br />
<br />
= [[Application Accounting]] =<br />
<br />
[[Category:Accounting]]</div>Perlhttps://wiki.egi.eu/w/index.php?title=Accounting_Repository&diff=46514Accounting Repository2012-11-28T14:11:35Z<p>Perl: </p>
<hr />
<div>{{Template:Op menubar}} {{Template:Tools menubar}} <br />
{{TOC_right}}<br />
=Accounting portal =<br />
<br />
The accounting infrastructure is a complex system that involves various sensors in different regions, all publishing data to a central repository. The data is processed, summarized and displayed in the accounting portal, which acts as a common interface to the different accounting record providers and presents a homogeneous view of the data gathered and a user-friendly access to understanding resource utilization. <br />
<br />
* Go to the [http://accounting.egi.eu/ Accounting portal]<br />
<br />
== [[Accounting_Portal_Release_Notes|Release notes]] ==<br />
<br />
== Documentation ==<br />
<br />
*[http://www4.egee.cesga.es/accounting/links/acct_ibergrid06_final14.pdf Architecture and Implementation] <br />
*[http://www4.egee.cesga.es/accounting/links/accounting_portal_installation.pdf Installation Instruction] <br />
*[https://documents.egi.eu/document/517 Accounting Portal Roadmap]<br />
*[http://goc.grid.sinica.edu.tw/gocwiki/Outstanding_Accounting_Issues Outstanding Accounting Issues]<br />
<br />
= APEL repository =<br />
== [[APEL|APEL documentation]] ==<br />
<br />
= [[Application Accounting]]<br />
<br />
[[Category:Accounting]]</div>Perlhttps://wiki.egi.eu/w/index.php?title=Accounting_Repository&diff=46513Accounting Repository2012-11-28T14:11:23Z<p>Perl: </p>
<hr />
<div>{{Template:Op menubar}} {{Template:Tools menubar}} <br />
{{TOC_right}}<br />
=Accounting portal =<br />
<br />
The accounting infrastructure is a complex system that involves various sensors in different regions, all publishing data to a central repository. The data is processed, summarized and displayed in the accounting portal, which acts as a common interface to the different accounting record providers and presents a homogeneous view of the data gathered and a user-friendly access to understanding resource utilization. <br />
<br />
* Go to the [http://accounting.egi.eu/ Accounting portal]<br />
<br />
== [[Accounting_Portal_Release_Notes|Release notes]] ==<br />
<br />
== Documentation ==<br />
<br />
*[http://www4.egee.cesga.es/accounting/links/acct_ibergrid06_final14.pdf Architecture and Implementation] <br />
*[http://www4.egee.cesga.es/accounting/links/accounting_portal_installation.pdf Installation Instruction] <br />
*[https://documents.egi.eu/document/517 Accounting Portal Roadmap]<br />
*[http://goc.grid.sinica.edu.tw/gocwiki/Outstanding_Accounting_Issues Outstanding Accounting Issues]<br />
<br />
= APEL repository =<br />
== [[APEL|APEL documentation]] ==�<br />
<br />
= [[Application Accounting]]<br />
<br />
[[Category:Accounting]]</div>Perl