EGI-InSPIRE:UNICORE integration task force

From EGIWiki
Jump to: navigation, search
EGI Activity groups Special Interest groups Policy groups Virtual teams Distributed Competence Centres


Alert.png This article is Deprecated and should no longer be used, but is still available for reasons of reference.

Contents

Interoperations menu: Home Unicore Globus Mapper/QCG ARC Desktop Grids EGI Federated Cloud XSEDE Contact



Coordinator: M. Krakowian/EGI.eu

 Meetings page: https://www.egi.eu/indico/categoryDisplay.py?categId=49

Mailing list: unicore-integration-tf@mailman.egi.eu

Mailing list page: https://www.egi.eu/sso/groupView/unicore-integration-tf

NGIs with UNICORE RC's: NGI_PL, NGI_DE, NGI_BY


Current status

Status of integration

Status Comments
Management

Completed

  • unicore6.Registry
  • unicore6.Gateway
  • unicore6.TargetSystemFactory
  • unicore6.StorageFactory
  • unicore6.StorageManagement
  • unicore6.ServiceOrchestrator
  • unicore6.WorkflowFactory
  • unicore6.UVOSAssertionQueryService
Monitoring

Completed

  • unicore6.Gateway - emi.unicore.Gateway
  • unicore6.Registry - emi.unicore.Registry
  • unicore6.ServiceOrchestrator - emi.unicore.ServiceOrchestrator
  • unicore6.StorageManagement - emi.unicore.GlobalStorage, emi.unicore.GlobalStorage-FreeSpace
  • unicore6.TargetSystemFactory - emi.unicore.TargetSystemFactory, emi.unicore.UNICORE-Job
  • unicore6.UVOSAssertionQueryService - emi.unicore.UVOS
  • unicore6.WorkflowFactory - emi.unicore.WorkflowService
  • unicore6.StorageFactory - emi.unicore.StorageFactory

EGI OPS Availability and Reliability Profile:

  • unicore6.TargetSystemFactory - emi.unicore.TargetSystemFactory, emi.unicore.UNICORE-Job
Accounting

In progress

Accounting services for UNICORE have been developed by NGI_PL and NGI_BY. UNICORE resource centres can publish accounting information through a UNICORE RUS server

Support Completed Support is provided by the UNICORE product team through the EGI Helpdesk with medium level of service support
Dashboard

Completed

  • GGUS-81278
    • emi.unicore.Gateway
    • emi.unicore.Registry
    • emi.unicore.ServiceOrchestrator
    • emi.unicore.UVOS
    • emi.unicore.TargetSystemFactory
    • emi.unicore.GlobalStorage
    • emi.unicore.GlobalStorage-FreeSpace
    • emi.unicore.WorkflowService
Documentation

Completed

Information System


In progress




Legend

Actions

Open action points

September 2012

Action

Descripton Assigned to Deadline Concerning
ACTION 1 Open a JIRA ticket for adding available CIS probes into next SAM release. Emir Imamagic
Monitoring
ACTION 2 Open a SAM JIRA ticket for integration of the new version of complex UNICORE job test. Emir Imamagic
Monitoring
ACTION 3
Check the status of UNICORE probes on NGI_DE SAM instance. Emir Imamagic
Monitoring
ACTION 4
Add NGI_BY UNICORE services to GOCDB. Serge Salamanka
Management


June 2012

Action

Descripton Assigned to Deadline Concerning
ACTION 1 Clarify certification procedures from perspective of other middleware together with PL-Grid. Malgorzata Krakowian
Documentation


Closed action points

June 2012

ACTION 2: Notify German and Poland via RT tickets once the SR of SAM Update-17 starts (Mario David)

ACTION 3: Integration of atomic job test in SAM Update-19: https://tomtools.cern.ch/jira/browse/SAM-2722 (Emir Imamagic)

ACTION 4: Provide update once new probes based on UCC 6.5.0 are available. (Krzysztof Benedyczak)

ACTION 5 (ADDITIONAL): Added new UNICORE service types to GOCDB (Emir Imamagic):

March 2012

ACTION 1: Test using local registry for probes. (Krzysztof Benedyczak)

ACTION 2: Ask NGIs on interest in UNICORE deployment. (Tiziana Ferrari & Emir Imamagic)

Old action points

2011

Update: https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0
https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=14727&grid_id=0
110704 Found problems with entering all RFC 3986 chars parsing of some characters in Service Point URL field, fixed by DM 110705.
Update: Still some minor things: description of the ServiceEndPointURL input field should contain the now correct RFC 3986 characters and initial letters (R and G) of unicore6.Registry and unicore6.Gateway should be capital.


Update: KB to make an update that the probes will be maintained by EMI, EI to close the ticket after the corresponding release.
Update: KB has updated the ticket.
Update: UNICORE probes integration delay due to misjudging the amount of effort needed for actual integration. Deadline for SAM for the Update-12 release missed. Next deadline is SAM Update-13 which will be released around the end of July
Update: SAM Update-13 pushed back to August 12 (from end of July), we think we manage until then, even if the integration requires more changes on the EGI side than initially expected.


Update: Important deadlines end of June and end of July. A wiki should come out of it and a discussion group. JG will take initiative to organize something when the time has come to discuss the details on how to best integrate the current UNICORE accounting solution.
Update: The wiki and the mailinglist have been created. A new production system is online based on messaging, which can be used to test against by UNICORE. Existing direct database insert clients like SGAS, DGAS,.. will also have to be moved to this new system. JG planning a meeting with Hannover within the next two weeks to get an impression of the German D-Grid solution. Later on we should have a functional comparison between the Polish and the German solution and see what fits best to EGI in a taskforce meeting focused on accounting.
Background information http://dl.dropbox.com/u/9548018/APEL%20SSM%20background.txt and links to wiki and mailinglist: http://dl.dropbox.com/u/9548018/APEL%20SSM%20testing.txt
Update: PL-Grid restructuring whole UNICORE documentation system otherwise concentration on EMI 1 release. MB will now start by pointing them to standard unicore.eu documents which conclude good documentation and installation guides and see if they would like to already include a link to them. What is missing: real best practices, the very solutions to different real integration problems. Some material exists in Polish language.
Update from Mathilde: UNICORE installation documentation (not yet covering the EMI UNICORE rpms) can be found at http://www.unicore.eu/documentation/manuals/unicore6/files/manual_installation.pdf The basic scenario (section 2) should answer the question on how to install multiple services on one server. It is not perfect for EGI purposes but maybe it is of interest.
Update: Michaela sent those links to the best practices mailinglist, but hasn't got any feedback yet.
Update: MB will try to contact them again.
Update: https://rt.egi.eu/rt/Ticket/Display.html?id=2574
Update: Planned roadmap according to COO: discussion for an update at the Nov OMB, with the purpose of having a new version in december of January.

after meeting on 110715

Progress: MB will report on her email discussion with Laurence Field and plans to push the GLUE 2.0 use case in propagating downtime information
Update: EMI registry in very early planning stage, XML solution in the meantime. Second solution for GOCDB (the GLUE 2.0 based approach) again in discussion. MB to bring forward and discuss the usecase in the next OGF PGI wg meeting.
Update: Too late to be part of V1 of http://www.ogf.org/documents/GFD.180.pdf waiting together with 2 other use cases for V2.
Update: currently reviewing related document EMI Execution Service Specification https://twiki.cern.ch/twiki/pub/EMI/EmiExecutionService/EMI-ES-Specification_v1.0.odt
Update: EMI ESS reviewed, currently no PGI wg meetings.
Update: PGI working group start again with regular phoneconferences now after OGF 32, but EMI registry currently no longer directly relevant to us now with ServicePointURLs working in GOCDB.


Update: Some discussion of this list during the meeting, AP on KB to send his list to the mailinglist as well and to continue discussing there and to put the output of this discussion in ticket https://rt.egi.eu/rt/Ticket/Display.html?id=944
Update: remaining service name description discussed during the meeting. MB to update the ticket. Decision that currently no more are needed.
Update: MB updated the ticket.
Closed: List successfully taken over and we are fine with the current set of services for the moment.
GOCDB/Input System User Documentation#Service_types


Update: A new release of UNICORE RUS accounting system is available from sourceforge. It is major milestone as it completes our move to JMS based architecture with a full fail-over support. It is also updated to USE and 6.4.0 release of UNICORE. http://unicore-dev.zam.kfa-juelich.de/documentation/rus-accounting-1.3.1/
Update: The NGI_BY UNICORE Accounting service AL presented at the EGI UF 2011 got permission to go Open Source. An English version and translated documentation is being prepared for initial export to Sourceforge. NGI-DE already became a non open source version to have some tests and do some comparison.
Update: Allowance to go open source was drawn back after a week, AP on AL to send contact information do deciding director to closer investigate the reasons behind this decision.
Update: AL sent contact details to MB. MB asked Tiziana Ferrari and Steven Newhouse for help. Reply from SN: "Clearly, if an NGI refuses to distribute a tool they have developed they are free to do so." New questions: Will it still be possible to deploy the NGI_BY accounting solution and use it to publish accounting records? Will it still be translated? How interested will some NGIs be to use a closed source solution?
Update: Already translated to English, but it cannot easily distributed with closed source because it's written in PHP. TF contacted the responsible director: Not so easy to make it legally open source, because it was bound to the funding by Belarus and Russia within the SKIF-GRID program and if not putting in our own funding, we don't have chances to make it OpenSource. Money allocation in the near future within EGI-InSPIRE very unlikely, though. Suggestion to try to get money somewhere else. NGI_BY of course still welcome in our task force.
Closed.



(* AP: KB to create an RT requirement ticket about integration of NGI specific service types into the regional dashboard. Update: Ticket has been created and will be handled next OTAG 110720)

older

Progress: David provided us with some updated slides on the possible implementation with more detail: https://wiki.egi.eu/wiki/File:GocdbGlue2Unicore.pdf
Update: Personal discussion at the EGI UF, Vilnius with John Casson: A new proposed solution with getting rid of the primary key constraint and a new timeline of one month has been discussed. It fulfills all our requirements. This requirement has now top-priority within GOCDB development.
Update: Requirement has been fixed and seems to work: https://goc.gridops.org/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0
Update: closed. New AP created to add more UNICORE services into GOCDB.


Update: redundant because of fast GOCDB integration, will eventually be done anyway. Closed.


Update: http://www.doodle.com/7xcgzsqxadp45v6q



https://rt.egi.eu/rt/Ticket/Display.html?id=1607
See ticket https://gus.fzk.de/dmsu/dmsu_ticket.php?ticket=68192 Who got it?
Update: https://gus.fzk.de/dmsu/dmsu_ticket.php?ticket=68177 was produced as another test ticket. Check why nobody got the tickets!
Progress: Ticket towards GGUS created: https://gus.fzk.de/ws/ticket_info.php?ticket=68354
Updated: GGUS ticket correctly sent to Jülich managed list emi-support@unicore.eu , this list has currently a small number of subscribed members. Awaiting internal EMI decision to add more people like e.g. KB

Final: Currently keep the small number who will get the GGUS ticket, who will forward them to the UNICORE tracking system in sourceforge. Follow up of this AP is out of the scope for this task force. Will be taken up gain within EMI. UNICORE related GGUS FAQ has been transported to EGI wiki: GGUS:UNICORE-Client FAQ and GGUS:UNICORE-Server FAQ.


unicore6.registry
unicore6.UNICOREX
unicore6.gateway
Final: GOCDB now uses the new Unicore service type names (unicore6.gateway, unicore6.registry, unicore6.UNICOREX). See: GOCDB/Input System User Documentation#Service_types



Why this is not supported by EMI is a political decision based on limited manpower and effort, KB wrote a detailed description on the usage of CIS in the monitoring system.




https://rt.egi.eu/rt/Ticket/Display.html?id=306
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Print/export