EGI-InSPIRE:UNICORE integration task force
|Main||EGI.eu operations services||Support||Documentation||Tools||Activities||Performance||Technology||Catch-all Services||Resource Allocation||Security|
Coordinator: E. Imamagic (SRCE)
The integration of UNICORE in operational tools encompasses different areas including how are UNICORE end nodes are registered in GOCDB, how UNICORE can be integrated in the Nagios-based monitoring system, how usage of UNICORE resources can be accounted for, how the VO concept can be adapted to UNICORE. A task force has been setup at the January OMB to focus on these topics. Partners interested in participating are: PL-Grid, NGI_DE, J. Gordon (TSA1.5), M. Lechner (TSA1.3)
Meetings page: https://www.egi.eu/indico/categoryDisplay.py?categId=49
Mailing list: firstname.lastname@example.org
Mailing list page: https://www.egi.eu/sso/groupView/unicore-integration-tf
Open action points
|ACTION 1||Open a JIRA ticket for adding available CIS probes into next SAM release.||Emir Imamagic||Monitoring|
|ACTION 2||Open a SAM JIRA ticket for integration of the new version of complex UNICORE job test.||Emir Imamagic||Monitoring|
||Check the status of UNICORE probes on NGI_DE SAM instance.||Emir Imamagic||Monitoring|
||Add NGI_BY UNICORE services to GOCDB.||Serge Salamanka||Management|
|ACTION 1||Clarify certification procedures from perspective of other middleware together with PL-Grid.||Malgorzata Krakowian||Documentation|
Closed action points
ACTION 2: Notify German and Poland via RT tickets once the SR of SAM Update-17 starts (Mario David)
- Poland participated in SR of SAM Update-17
ACTION 3: Integration of atomic job test in SAM Update-19: https://tomtools.cern.ch/jira/browse/SAM-2722 (Emir Imamagic)
- Integrated two new tests: emi.unicore.StorageFactory and emi.unicore.UNICORE-Job.
ACTION 4: Provide update once new probes based on UCC 6.5.0 are available. (Krzysztof Benedyczak)
- Probes will be provided in the next EMI release. It was agreed to use solution with local registry in Update-19. The next SAM release will use new version of UCC.
ACTION 5 (ADDITIONAL): Added new UNICORE service types to GOCDB (Emir Imamagic):
- org.ogf.bes.BESFactory: BES job submission entry point
- unicore6.Catalogue: service providing unified view of files in multiple unicore6.StorageManagement
- unicore6.CISInfoProvider: provides data for CIS
- unicore6.CISRegistryPortType: provides human friendly interface for browsing the UNICORE Grid status
- unicore6.GridBeanService: enables download of GridBeans (modules providing application-specific GUI).
ACTION 1: Test using local registry for probes. (Krzysztof Benedyczak)
- Testing with local registries was implemented in SAM Update-19. In the next release of UCC (6.5.0), probes do not require Registry at all.
ACTION 2: Ask NGIs on interest in UNICORE deployment. (Tiziana Ferrari & Emir Imamagic)
- Asked at the OMB in Munich, no-one expressed interest.
Old action points
- AP: Add more UNICORE services into GOCDB.
- Update: https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0
- 110704 Found problems with entering all RFC 3986 chars parsing of some characters in Service Point URL field, fixed by DM 110705.
- Update: Still some minor things: description of the ServiceEndPointURL input field should contain the now correct RFC 3986 characters and initial letters (R and G) of unicore6.Registry and unicore6.Gateway should be capital.
- AP: "Regular probe integration ticket updates" for everybody (especially EI, KB and FR): to keep https://rt.egi.eu/rt/Ticket/Display.html?id=306 updated.
- Update: KB to make an update that the probes will be maintained by EMI, EI to close the ticket after the corresponding release.
- Update: KB has updated the ticket.
- Update: UNICORE probes integration delay due to misjudging the amount of effort needed for actual integration. Deadline for SAM for the Update-12 release missed. Next deadline is SAM Update-13 which will be released around the end of July
- Update: SAM Update-13 pushed back to August 12 (from end of July), we think we manage until then, even if the integration requires more changes on the EGI side than initially expected.
- AP: "Keep an eye on accounting": See that EMI follows its roadmap and we get the possibility to send accounting different from gLite to APEL. Related to get updated on the work going on in the new EMI Workinggroup led by Andrea Guarese https://twiki.cern.ch/twiki/bin/view/EMI/ComputeAccounting
- Update: Important deadlines end of June and end of July. A wiki should come out of it and a discussion group. JG will take initiative to organize something when the time has come to discuss the details on how to best integrate the current UNICORE accounting solution.
- Update: The wiki and the mailinglist have been created. A new production system is online based on messaging, which can be used to test against by UNICORE. Existing direct database insert clients like SGAS, DGAS,.. will also have to be moved to this new system. JG planning a meeting with Hannover within the next two weeks to get an impression of the German D-Grid solution. Later on we should have a functional comparison between the Polish and the German solution and see what fits best to EGI in a taskforce meeting focused on accounting.
- Background information http://dl.dropbox.com/u/9548018/APEL%20SSM%20background.txt and links to wiki and mailinglist: http://dl.dropbox.com/u/9548018/APEL%20SSM%20testing.txt
- AP: "Best practices" KB and FR to send documentation links to email@example.com as soon as considered sufficiently completed, suggested procedure which could be of interest in this context: how to install more than one UNICORE service on one host.
- Update: PL-Grid restructuring whole UNICORE documentation system otherwise concentration on EMI 1 release. MB will now start by pointing them to standard unicore.eu documents which conclude good documentation and installation guides and see if they would like to already include a link to them. What is missing: real best practices, the very solutions to different real integration problems. Some material exists in Polish language.
- Update from Mathilde: UNICORE installation documentation (not yet covering the EMI UNICORE rpms) can be found at http://www.unicore.eu/documentation/manuals/unicore6/files/manual_installation.pdf The basic scenario (section 2) should answer the question on how to install multiple services on one server. It is not perfect for EGI purposes but maybe it is of interest.
- Update: Michaela sent those links to the best practices mailinglist, but hasn't got any feedback yet.
- Update: MB will try to contact them again.
- AP on TF: to bring up site-information system (BDII) discussion up at the TCB
- AP: "Reformulation of OLA" MB to formulate a more highlevel and middleware independent version of point 10.3 in the OLA https://documents.egi.eu/secure/ShowDocument?docid=650 and create a new ticket in the noc-managers queue for it and assign it to Tiziana, so it can be discussed at the OMB
- Update: https://rt.egi.eu/rt/Ticket/Display.html?id=2574
- Update: Planned roadmap according to COO: discussion for an update at the Nov OMB, with the purpose of having a new version in december of January.
after meeting on 110715
- AP: "EMI registry use case" Find out more about the EMI registry and the static info contained: How is the distinction between the static data GOCDB and EMI registry, how can we propagate downtime info for just one Service Endpoint (URLs are needed to distungish between different instances)? (contact Laurence Field)
- Progress: MB will report on her email discussion with Laurence Field and plans to push the GLUE 2.0 use case in propagating downtime information
- Update: EMI registry in very early planning stage, XML solution in the meantime. Second solution for GOCDB (the GLUE 2.0 based approach) again in discussion. MB to bring forward and discuss the usecase in the next OGF PGI wg meeting.
- Update: Too late to be part of V1 of http://www.ogf.org/documents/GFD.180.pdf waiting together with 2 other use cases for V2.
- Update: currently reviewing related document EMI Execution Service Specification https://twiki.cern.ch/twiki/pub/EMI/EmiExecutionService/EMI-ES-Specification_v1.0.odt
- Update: EMI ESS reviewed, currently no PGI wg meetings.
- Update: PGI working group start again with regular phoneconferences now after OGF 32, but EMI registry currently no longer directly relevant to us now with ServicePointURLs working in GOCDB.
- AP: "List of service types to be integrated." MB compiling a list of the next service types which should be integrated into GOCDB and sending it to our mailinglist for discussion
- Update: Some discussion of this list during the meeting, AP on KB to send his list to the mailinglist as well and to continue discussing there and to put the output of this discussion in ticket https://rt.egi.eu/rt/Ticket/Display.html?id=944
- Update: remaining service name description discussed during the meeting. MB to update the ticket. Decision that currently no more are needed.
- Update: MB updated the ticket.
- Closed: List successfully taken over and we are fine with the current set of services for the moment.
- AP: Start a more tight collaboration with Belarus. AL to keep us updated on efforts to go open source or other possibilities to collaborate, KB to send info on new release with all major features.
- Update: A new release of UNICORE RUS accounting system is available from sourceforge. It is major milestone as it completes our move to JMS based architecture with a full fail-over support. It is also updated to USE and 6.4.0 release of UNICORE. http://unicore-dev.zam.kfa-juelich.de/documentation/rus-accounting-1.3.1/
- Update: The NGI_BY UNICORE Accounting service AL presented at the EGI UF 2011 got permission to go Open Source. An English version and translated documentation is being prepared for initial export to Sourceforge. NGI-DE already became a non open source version to have some tests and do some comparison.
- Update: Allowance to go open source was drawn back after a week, AP on AL to send contact information do deciding director to closer investigate the reasons behind this decision.
- Update: AL sent contact details to MB. MB asked Tiziana Ferrari and Steven Newhouse for help. Reply from SN: "Clearly, if an NGI refuses to distribute a tool they have developed they are free to do so." New questions: Will it still be possible to deploy the NGI_BY accounting solution and use it to publish accounting records? Will it still be translated? How interested will some NGIs be to use a closed source solution?
- Update: Already translated to English, but it cannot easily distributed with closed source because it's written in PHP. TF contacted the responsible director: Not so easy to make it legally open source, because it was bound to the funding by Belarus and Russia within the SKIF-GRID program and if not putting in our own funding, we don't have chances to make it OpenSource. Money allocation in the near future within EGI-InSPIRE very unlikely, though. Suggestion to try to get money somewhere else. NGI_BY of course still welcome in our task force.
- AP: "Next Meeting": MB making a doodlepoll for the weeks 28 and 30 (Global Timezones and maybe function enabled). http://www.doodle.com/58w93q6xqi32pvqd
(* AP: KB to create an RT requirement ticket about integration of NGI specific service types into the regional dashboard. Update: Ticket has been created and will be handled next OTAG 110720)
- AP: "ServiceEndpointURLs in GOCDB" Reopen GOCDB requirement ticket for EndPointServiceURL https://rt.egi.eu/rt/Ticket/Display.html?id=975 and ask for second solution (Add URL field for ServiceEndpoint and repeat ServiceEndpoint to represent different URLs /Associate a ServiceEndpoint with a new „1-to-many‟ EndpointLocation entity), with a proposed timeline of 6 months.
- Progress: David provided us with some updated slides on the possible implementation with more detail: https://wiki.egi.eu/wiki/File:GocdbGlue2Unicore.pdf
- Update: Personal discussion at the EGI UF, Vilnius with John Casson: A new proposed solution with getting rid of the primary key constraint and a new timeline of one month has been discussed. It fulfills all our requirements. This requirement has now top-priority within GOCDB development.
- Update: Requirement has been fixed and seems to work: https://goc.gridops.org/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0
- Update: closed. New AP created to add more UNICORE services into GOCDB.
- AP: EI to contact Maciej Pawlik and Paweł Wolniewicz to discuss sensible Nagios configuration default definitions.
- Update: redundant because of fast GOCDB integration, will eventually be done anyway. Closed.
- AP: "Next Meeting": MB making again a doodlepoll for next meeting with Global Timezones and maybe function enabled.
- AP: ML making again a doodlepoll for next meeting with Global Timezones and maybe function enabled. http://www.doodle.com/2ysf5wa675bqifxd
- AP: official requirement towards APEL (accounting monitoring must be sufficient to decide whether all deployed middlewares of a site publish accounting data)
- AP: ML to test sending a GGUS ticket to UNICORE:
- See ticket https://gus.fzk.de/dmsu/dmsu_ticket.php?ticket=68192 Who got it?
- Update: https://gus.fzk.de/dmsu/dmsu_ticket.php?ticket=68177 was produced as another test ticket. Check why nobody got the tickets!
- Progress: Ticket towards GGUS created: https://gus.fzk.de/ws/ticket_info.php?ticket=68354
- Updated: GGUS ticket correctly sent to Jülich managed list firstname.lastname@example.org , this list has currently a small number of subscribed members. Awaiting internal EMI decision to add more people like e.g. KB
Final: Currently keep the small number who will get the GGUS ticket, who will forward them to the UNICORE tracking system in sourceforge. Follow up of this AP is out of the scope for this task force. Will be taken up gain within EMI. UNICORE related GGUS FAQ has been transported to EGI wiki: GGUS:UNICORE-Client FAQ and GGUS:UNICORE-Server FAQ.
- AP: DM to rename the first 3 UNICORE service types.
- Final: GOCDB now uses the new Unicore service type names (unicore6.gateway, unicore6.registry, unicore6.UNICOREX). See: https://wiki.egi.eu/wiki/GOCDB/Input_System_User_Documentation#Service_types
- AP: ML making a doodlepoll for next meeting, this time with Global Timezones and maybe function enabled. http://www.doodle.com/9zw6qmkcmzvy5sq9
- AP: KB sends a summary of CIS, why it is not supported by EMI and why the XML approach is advantageous monitoring wise.
- Why this is not supported by EMI is a political decision based on limited manpower and effort, KB wrote a detailed description on the usage of CIS in the monitoring system.
- AP: ML making a doodlepoll for next meeting in March
- AP: ML to send EI minutes of GOCDB dedicated phone conference
- AP: KB contacting Germany through ticket 306