VT MPI within EGI

General Project Information

• Leader: Alvaro Simon (CESGA, Spain) and Zdenek Sustr (CESNET, Czech Republic)
• Mailing List: vt-mpi at mailman.egi.eu
• Status: Active
• Start Date: 10/Nov/2011
• End Date:
• Meetings: 1. 12/12/2011 - MPI VT Management meeting

Motivation

Despite a dedicated SA3 activity to support MPI there still seem to be significant issues in uptake and satisfaction amongst the user communities. This VT

• Works with user communities and projects that use MPI resources (e.g. ITER, MAPPER, A&A, etc) to demonstrate that MPI can work successfully in EGI.
• Sets up a VO on EGI with sites committed to support MPI jobs.
• Improve the communication between MPI users and developers of MPI support within EGI SA3.

Output

The VT is expected to produce the following outputs:

• Materials (tutorials, white papers, etc) about successful use cases of MPI on EGI that can be used by new communities to use MPI on EGI.
• An MPI VO that provides:
  • dedicated CPUs for MPI jobs
  • MPI specific test probes can run on all sites using the VO Monitoring services of Ibergrid (EGI-InSPIRE VO Services group)
  • accounting for MPI jobs
  • user support
• Improved communication channels with MPI users
• The above set of resources and feedback to resource centers, user communities and technology providers on how to improve MPI within EGI.

Tasks

• Task 1: MPI documentation: This documentation will be reviewded and we will decide what needs updating or extending.
  • Gergely comments:

For users:
https://wiki.egi.eu/wiki/MPI_User_Guide
https://wiki.egi.eu/wiki/MPI_User_manual
Should be merged in a single wiki page.

For site admins:
https://wiki.egi.eu/wiki/MAN03

• Task 2: Nagios probes: Current nagios probes should be reviewed to test EGI MPI infrastructure.
  • New nagios probes requirements: https://wiki.egi.eu/wiki/Nagios-requirements.html
  • John Walsh comments:

a) A non-critcal test that tests MPI scalability above two nodes.
Ideally, I would like to see this test set to  ceiling(average number of cores) x 2 +1.
This should increase the likelihood that the job runs on multiple nodes.
This test should only be run may once or twice a week and allow at least a day for scheduling
(so as to be non-intrusive on valuable site resources).

b) Improve the baseline MPI tests.
We should test basic MPI API functionality (scatter, gather, etc), rather than the simpler "hello world".
I will try to see whether I assemble a basic test-suite.

c) Following up on https://ggus.eu/ws/ticket_info.php?ticket=76755,  I have suggested that we may not be using GlueCEPolicyMaxCPUTime and GlueCEPolicyMaxObtainableCPUTime  properly, and that site queues may not be correctly set up for MPI jobs.
(i.e not setting torque queue resources_max.cput  and resources_max.pcput values). Perhaps we can develop a (BDII?) sanity "warning" check for this?

• Task 3: Information system: Problems detecting MPI resources.
  • Checking for MPI availability -- mostly decided by checking installed applications.
  • Not all sites reporting MPI capability correctly
  • Ivan comments:

For BDII, the metrics portal checks the GlueHostApplicationSoftwareRunTimeEnvironment property for the *MPI* regular expression.

• • John Walsh comments:

GGUS ticket: https://ggus.eu/ws/ticket_info.php?ticket=76755
the problem seems to be related to the torque settings for pcput an cput on each of the queue.

cput = Maximum amount of CPU time used by all processes in the job.
pcput = Maximum amount of CPU time used by any single process in the job.
walltime = Maximum amount of real time during which the job can be in the running state.

So, for example, on one of the "medium"  queue on
deimos.htc.biggrid.nl, the config is:
set queue medium resources_max.cput = 24:00:00
set queue medium resources_max.pcput = 24:00:00
set queue medium resources_max.walltime = 36:00:00

This would not be sufficient to allow an 6 core job to run for a full
24 hours, and the job is likely to be removed after it has run for 4 hours.
We need to check that these queue settings are sensible for MPI jobs.

• Task 4: Accounting system: Implement MPI accounting system. (JRA1.4)
  • Ivan Comments:

No special accounting support. Only way to recognize MPI jobs is to check jobs with >100 % efficiency
-Still development to be done.
-Apel needs to give data for each batch system.

• Task 5: Batch system support: All batch systems must support MPI jobs. Check the current batch system status and issues.
  • Roberto Rosende comments:

Starting work on MPI support for SGE, to be ready for EMI2
The main problem with the batch system is that it is not receiving reliable info from information system (not truly a batch system matter).

• • Alvaro Simon comments:

Two bugs were found during the first UMD verification of
WN/Torque + EMI-MPI.1.0. Is a torque/maui problem that affects all MPI jobs. Maui versions
prior to 3.3.4 do not allocate correctly all the nodes for the  job execution. GGUS tickets:
- https://ggus.eu/ws/ticket_info.php?ticket=57828
- https://ggus.eu/ws/ticket_info.php?ticket=67870

• Task 6: Gather information from MPI sites: After establishing the VO, and contacting sites for resources, more requests for information can be added.
  • Comments from Zedenek

- MPI VO -- bring together sites and users interested in MPI
- This VO is NOT intended for everyday use by all users wishing to use MPI
- This VO IS intended for users who wish to cooperate with the VT to make MPI support in EGI better
- The main reason for its establishment is to collect experience that will be later adopted by regular VOs

• • Comments from Ivan:

User Community under SA3 would also be a good idea. 

Members

• NGIs - confirmed:
  • CZ: Zdenek Sustr (leader)
  • ES/IBERGRID: Alvaro Simon (leader), Enol Fernandez, Iván Díaz, Alvaro Lopez, Pablo Orviz, Isabel Campos.
  • GR: Dimitris Dellis, Marios Chatziangelou, Paschalis Korosoglou
  • HR: Emir Imamagic, Luko Gjenero
  • IE: John Walsh
  • IT: Daniele Cesini, Alessandro Costantini, Marco Bencivenni
  • PT: Gonçalo Borges
  • SK: Viera Sipkova, Viet Tran, Jan Astalos
• EGI.eu: Gergely Sipos, Karolis Eigelis

Progress

• Task 1
• Task 2
• ...
• Task N