VT VAPOR:Progress Report

Progress Report

VT Reporting guidelines

Reporting period: Sept. 4th to 17th

Progress

• Continue the customization of the JobMonitor tool by partner IPHC to support a multi-vo environment. Several phone conferences with the developer.
• Global actions on the web application:
  • Continued to improve the ergonomy and look and feel of web pages: enrich helps, add customizable tooltips.
  • Tests have revealed weaknesses in the application development (parameters not tested, no error trace...): an important action has consisted in improving the safety of the application and adding much more log traces to be able to follow up on the application life.
• VO Operations, reports of availability of resources:
  • Added some reports in the web pages to show the "running ratio" R(R/W)
  • Almost completed the Lavoisier VO-views to consolidate data from the BDII and GOCDB.
• VO data management: continuation of the study on VO Data management, very rich discussion with developers of GFAL2 in GGUS ticket #97076

Plans for next period

• Face to face meeting with the developers of the EGI Operation Portal to discuss future steps.
• Add 2 new web pages to show (1) the current status of resources supporting the VO (CE, SE, WMS, VOMS, LFC), and (2) the list of resources which status is currently not in producton (consolidate BDII/GOCDB statuses)

Problems we encounter, but can solve

The lack of experience of the main developer is being handled and things are imporving, altough the code production and quality remain lower than expected. Action: heavy support is being provided to him.

Problems and issues we need external help with

None.

Reporting period: Aug. 21st - Sept. 3rd

Progress

• Follow up of the improvement of the ergonomy and look and feel of web pages.
• VO Operations, reports of availability of resources:
  • completed the web page and code dedicated to the production of white list of CEs based on job monitoring reports.
  • Writing of VO-views to consolidate the data from the BDII and GOCDB, using the Lavoisier data integration service. Numerous interactions with the developers of the tool.
• VO data management: continuation of the study on VO Data management (dark data in particular), discussions with sites admins (FR, UK) to refine the procedures

Plans for next period

• VO Operations, reports of availability of resources: still some reports to add in the web pages to show the "running ratio" R(R/W)

Problems we encounter, but can solve

• The lack of experience of the main developer is being handled and things are imporving, altough the code production remains slower than expected. Action: heavy support is being provided to him.
• Complexity of the Lavoisier data integration service, but the development team is very helpful and reactive.

Problems and issues we need external help with

None.

Reporting period: Aug. 7th to 20th

Progress

• Work with partner IPHC to upgrade tool JobMonitor to support a multi-vo environment. Several phone conferences.
• Improvements of the ergonomy of web pages.
• Improve the management of misc. errors.
• VO Operations, reports of availability of resources:
  • development of new web pages to report running and waiting jobs, and ratio R(R/W).
• VO data management:
  • testing of scripts to detect and clean up VO dark data
  • discussions held with sites admins (FR IPHC, UK QMUL) in order to refine the procedure to deal with dark data and lost files. Study of good practices from HEP VOs.

Plans for next period

• VO Operations management features (resources availability): add new reports on ratio R(R/W), complete the production of the white list of CEs.
• Continue study on VO Data management (dark data).

Problems we encounter, but can solve

Lack of experience of the main developer in terms of development good practices and application design. This has slown down the activity but this is handled.

Problems and issues we need external help with

None.

Reporting period: July 24th - Aug. 6th

Progress

• This period was used to make a strong focus on code quality including code review, cleaning up of code, and the first commit of the current version after code and environment cleaning up.
• Documentation: write a document to describe the application environment and installation procedure.

Development:

• VO Operations management features
  • resources availability: continue development of reports on running and waiting jobs, and ratio R(R/W).
  • VO data management: start development of scripts to detect and clean up VO dark data.
• Focus on styles and look and feel of web pages

Plans for next period

Complete VO Operations management features (resources availability): reports on running and waiting jobs, and ratio R(R/W); white list of CEs Continue work on VO Data management (dark data).

Problems we encounter, but can solve

Lack of experience of the main developer in terms of development good practices and application design. This has slown down the activity but this is handled.

Problems and issues we need external help with

None.

Reporting period: Jun. 26th - Jul. 23rd

Progress

Low activity due to summer vacation period: 2 weeks vacation for Flavien Forestier (developer) and 3.5 weeks for Franck Michel (project manager) => 10 days work for Flavien and 2 days for Franck.

• VO Operations management features (resources availability) :
  • continue development of web pages to view results of the Job Monitor tool of IPHC partner: chart for the history report, table view by computing element.
  • development of web page to view results of the CE monitor: view of number of running, waiting jobs and ration R(R/W) (chart)
  • discuss the parameters of the white list of CEs with IPHC.
• Deployment of the dev and test environment on a virtual machine.

Plans for next period

• Refine existing with better ergonomy and presentation
• Focus on styles and look and feel of web pages
• Display a white list of CEs

Problems we encounter, but can solve

None.

Problems and issues we need external help with

None.

Reporting period: June 19th to 25th

Exceptionally this is a one-week report as I'll be on vacation starting end of this week.

Progress

• Get skills about using the Twitter Bootstrap software, about Ajax technologies, and javascript graphical librairies (Dygraph)
• Continue the development of web pages for the VO Operations management features about resources availability: chart for the results of the job monitor of IPHC (number of jobs ok, ko, timed out).
• Further install and configuration of the virtual machine deployed last week to host developments and tests of VAPOR.

Plans for next period

Very few will be done in the next period due to summer vacations of Franck Michel (back 22 July), and Flavien Forestier (VAPOR developer, back 15 July).

Problems we encounter, but can solve

none.

Problems and issues we need external help with

none.

Reporting period: June 5th to 18th

Progress

• Work with partner IPHC to upgrade tool JobMonitor to support a multi-vo environment. Several phone conferences.
• 2 days face-to-face meeting in Lyon (France) between I3S and EGI Operations Portal and VO Operations Dashboard developers team: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1721. Goal: technical discussions on the way to integrate VAPOR developments into the existing VO Operations Dashboard.
• Also in Lyon, discussion with biomed LFC manager about Data Management procedures to set up.
• Continued technical phone conferences with partner CNRS IPHC: the existing job monitor tool is being customized to be more general (initially dedicated to biomed).
• Prototyping of first web pages for the VO Operations management features about resources availability, using both the job monitor of IPHC and the data integrator web service from EGI Operations Portal (Lavoisier).
• Deployment of a virtual machine at I3S to host developments and tests of VAPOR.

Plans for next period

• Keep on developing the VO Operations management features: continue integration of job monitor along with appropriate web pages in tabular and chart formats.
• Start development of web pages to report evolution of running and waiting jobs in the VO in grapical charts
• Start using technologies such as Twitter Bootstrap and Ajax to make a good-looking, user friendly and reactive web interface.

Problems we encounter, but can solve

The work on the VO data management procedures has been started in Lyon with discussions with LFC manager. Further work will be postponed later during the summer.

Reporting period: May 22nd - June 5th

Progress

• Decision made on the priorities of the developments, following the discussion with partner VOs about the Functional specification of VAPOR features :
  1. VO Operations management > Report GOCDB and BDII status and Monitor resources availability.
  2. VO Operations management > VO Data Management procedures.
  3. Users database implementation.
• Developments started on point 1: 2 conference calls held during the period with partner CNRS IPHC that develops a tool to monitor CEs.
• 6 days (2x3 days) of training courses for Franck Michel during this period.

Plans for next period

• 2 days face to face meeting with VO Operations Dashboard developers team in Lyon (France), to bootstrap developments within a common environment.
• Will try to organise calls with VO AUGER who shoed interest for VAPOR.
• Find volunteer among partners to start working on the possible procedures that can be envisaged about data management (priority 2).

Problems we encounter, but can solve

Difficulty to find someone among partners to start working on the VO data management procedures. Call to be organised with partners CNRS Creatis and CNRS IPHC.

Problems and issues we need external help with

None.

Reporting period: May 8th to 21st

Progress

• 4 national holidays in the last two weeks explain a rather light advance in this period.
• Work continued on the Functional specification of VAPOR features : restructuring, additionals, more in depth details, additional related material.
• Conference held with France Grille VO to get their opinion on the features of VAPOR and the possibility that they use it in the future. Meetings list updated: https://indico.egi.eu/indico/categoryDisplay.py?categId=100
• Conference with AMC can't be done for now due to constraints of AMC. Will be rescheduled in July.
• Face to face meeting scheduled on 5 and 6 of June with VO Operations Dashboard developers team in Lyon (France): the idea is to bootstrap joint developments.
• Fix and improve existing tools to be integrated into VAPOR about GOCDB and BDII status report.
• Self training of Flavien Forestier continued regarding grid technologies, Symphony2 framework and related development technologies.

Plans for next period

• Will try to organise calls with 2 other VOs that showed interest in VAPOR: AUGER
• Flavien Forestier to continue self training on dev technos, and acquire strong in-depth knowledge about functional features and technical solutions.
• Conatct members of the project to investigate possible technical solutions on data management.
• Contact biomed members about the tool used to perform CE monitoring tools.

Problems we encounter, but can solve

None.

Problems and issues we need external help with

None.

Reporting period: Apr. 22nd - May 7th

Progress

• Initialization of the functional specification of VAPOR features: https://wiki.egi.eu/wiki/VT_VAPOR:VAPOR_features_description
• Started discussions with the project partners to discuss/prioritize the VAPOR features: conferences held with COMPCHEM, eNMR, and EGI Operations Portal team. Meetings list: https://indico.egi.eu/indico/categoryDisplay.py?categId=100
• Decision made to integrate VAPOR developments into the existing VO Operations Dashboard, using the Symphony2 web application development framework. A 2 days face to face workshop to be schedule beginning of June.

Plans for next period

• Meetings with partners to continue: call scheduled with France Grille VO. Call with AMC to be scheduled.
• Flavien Forestier to start self training on Symphony2 framework.

Problems we encounter, but can solve

None.

Problems and issues we need external help with

None.