MARMOT-MPI Investigation and Checking Apparatus.


65 views
Uploaded on:
Description
MARMOT does not oblige source code changes, simply relinking ... MARMOT is said in the U.S. DoD High Performance Computing Modernization ...
Transcripts
Slide 1

MARMOT-MPI Analysis and Checking Tool Bettina Krammer, Matthias Müller krammer@hlrs.de , mueller@hlrs.de HLRS High Performance Computing Center Stuttgart Allmandring 30 D-70550 Stuttgart http://www.hlrs.de

Slide 2

Overview General Problems of MPI Programming Related Work New Approach: MARMOT Results inside CrossGrid Outlook

Slide 3

Problems of MPI Programming All issues of serial programming Additional issues: Increased trouble to confirm rightness of project Increased trouble to investigate N parallel procedures Portability issues between various MPI executions and stages, e.g. with mpich on the Crossgrid testbed: WARNING: MPI_Recv: tag= 36003 > 32767 ! MPI just ensures labels up to this. THIS execution permits labels up to 137654536 versions of LAM-MPI < v 7.0 just ensure labels up to 32767 New issues, for example, race conditions and gridlocks

Slide 4

Related Work Parallel Debuggers, e.g. totalview, DDT, p2d2 Debug form of MPI Library Post-mortem investigation of follow documents Special confirmation instruments for runtime examination MPI-CHECK Limited to Fortran Umpire First form constrained to shared memory stages Distributed memory rendition in arrangement Not openly accessible MARMOT

Slide 5

What is MARMOT? Instrument for the improvement of MPI applications Automatic runtime investigation of the application: Detect erroneous utilization of MPI Detect non-compact builds Detect conceivable race conditions and gridlocks MARMOT does not require source code changes, simply relinking C and Fortran authoritative of MPI - 1.2 is bolstered

Slide 6

Application Debug Server (extra process) Profiling Interface MARMOT center apparatus MPI library Design of MARMOT

Slide 7

Examples of Client Checks: confirmation on the nearby hubs Verification of legitimate development and use of MPI assets, for example, communicators, bunches, datatypes and so forth., for instance Verification of MPI_Request use invalid reusing of dynamic solicitation invalid utilization of unregistered solicitation cautioning if number of solicitations is zero cautioning on the off chance that all solicitations are MPI_REQUEST_NULL Check for pending messages and dynamic solicitations in MPI_Finalize Verification of every single other contention, for example, positions, labels, and so forth

Slide 8

Examples of Server Checks: confirmation between the hubs, control of project Everything that requires a worldwide perspective Control the execution stream, follow the MPI approaches every hub all through the entire application Signal conditions, e.g. halts (with traceback on every hub.) Check coordinating send/get sets for consistency Check aggregate calls for consistency Output of comprehensible log document

Slide 9

MARMOT inside CrossGrid

Slide 10

40 MHz (40 TB/sec) level 1 - exceptional equipment 75 KHz (75 GB/sec) 5 KHz (5 GB/sec) level 2 - installed processors 100 Hz (100 MB/sec) level 3 - PCs information recording & disconnected from the net investigation CrossGrid Application Tasks

Slide 11

CrossGrid Application: WP 1.4: Air contamination displaying (STEM-II) Air contamination demonstrating with STEM-II model Transport condition explained with Petrov-Crank-Nikolson-Galerkin strategy Chemistry and Mass exchange are incorporated utilizing semi-verifiable Euler and pseudo-systematic techniques 15500 lines of Fortran code 12 diverse MPI calls: MPI_Init, MPI_Comm_size, MPI_Comm_rank, MPI_Type_extent, MPI_Type_struct, MPI_Type_commit, MPI_Type_hvector, MPI_Bcast, MPI_Scatterv, MPI_Barrier, MPI_Gatherv, MPI_Finalize.

Slide 12

STEM application on an IA32 bunch with Myrinet

Slide 13

Conclusion MARMOT underpins MPI 1.2 for C and Fortran restricting Tested with a few applications and benchmarks Tested on various stages, utilizing diverse compilers and MPI usage, e.g. IA32/IA64 bunches (Intel, g++ compiler) with mpich (CrossGrid testbed, … ) IBM Regatta NEC SX5, SX6 Hitachi SR8000, Cray T3e

Slide 14

Impact and Exploitation MARMOT is said in the U.S. DoD High Performance Computing Modernization Program (HPCMP) Programming Environment and Training (PET) venture as one of two uninhibitedly accessible instruments (MPI-Check and MARMOT) for the advancement of convenient MPI programs. Contact with instrument designers of the other 2 known confirmation apparatuses like MARMOT (MPI-Check, umpire) Contact with clients outside CrossGrid MARMOT has been exhibited at different gatherings and productions http://www.hlrs.de/association/tsc/ventures/marmot/pubs.html

Slide 15

Future Work Scalability and general execution enhancements Better client interface to present issues and notices Extended usefulness: more tests to confirm aggregate calls MPI-2 Hybrid programming

Slide 16

Thanks to all CrossGrid accomplices http://www.eu-crossgrid.org/Thanks for your consideration A dditional data and download: http://www.hlrs.de/association/tsc/ventures/marmot

Slide 17

Back Up

Slide 18

Impact and Exploitation MARMOT is said in the U.S. DoD High Performance Computing Modernization Program (HPCMP) Programming Environment and Training (PET) venture as one of two unreservedly accessible apparatuses (MPI-Check and MARMOT) for the advancement of convenient MPI programs. Contact with device engineers of the other 2 known confirmation devices like MARMOT (MPI-Check, umpire) MARMOT has been exhibited at different gatherings and distributions http://www.hlrs.de/association/tsc/ventures/marmot/pubs.html

Slide 19

Impact and Exploitation cont\'d Tescico proposition ("Technical and Scientific Computing") French/German consortium: Bull, CAPS Entreprise, Université Versailles, T-Systems, Intel Germany, GNS, Universität Dresden, HLRS ITEA-marked French powers consented to give subsidizing German powers did not

Slide 20

Impact and Exploitation cont\'d MARMOT is incorporated in the HLRS instructional courses Deployment in Germany MARMOT is being used in two of the 3 National High Performance Computing Centers: HLRS at Stuttgart (IA32, IA64, NEC SX) NIC at Jülich (IBM Regatta): Just supplanted their Cray T3E with an IBM Regatta invested a great deal of energy finding an issue that would have been distinguished naturally by MARMOT

Slide 21

Feedback of Crossgrid Applications Task 1.1 (biomedical) C application Identified issues: Possible race conditions because of utilization of MPI_ANY_SOURCE halt Task 1.2 (surge): Fortran application Identified issues: Tags outside of substantial reach Possible race conditions because of utilization of MPI_ANY_SOURCE Task 1.3 (hep): ANN (C application) no issues found by MARMOT Task 1.4 (meteo): STEMII (Fortran) MARMOT recognized openings in self-characterized datatypes utilized as a part of MPI_Scatterv, MPI_Gatherv. These gaps were expelled, which enhanced the execution of the correspondence.

Slide 22

Examples of Log File 54 rank 1 performs MPI_Cart_shift 55 rank 2 performs MPI_Cart_shift 56 rank 0 performs MPI_Send 57 rank 1 performs MPI_Recv WARNING: MPI_Recv: Use of MPI_ANY_SOURCE may bring about race conditions! 58 rank 2 performs MPI_Recv WARNING: MPI_Recv: Use of MPI_ANY_SOURCE may bring about race conditions! 59 rank 0 performs MPI_Send 60 rank 1 performs MPI_Recv WARNING: MPI_Recv: Use of MPI_ANY_SOURCE may bring about race conditions! 61 rank 0 performs MPI_Send 62 rank 1 performs MPI_Bcast 63 rank 2 performs MPI_Recv WARNING: MPI_Recv: Use of MPI_ANY_SOURCE may bring about race conditions! 64 rank 0 performs MPI_Pack 65 rank 2 performs MPI_Bcast 66 rank 0 performs MPI_Pack

Slide 23

Examples of Log File (proceeded) 7883 rank 2 performs MPI_Barrier 7884 rank 0 performs MPI_Sendrecv 7885 rank 1 performs MPI_Sendrecv 7886 rank 2 performs MPI_Sendrecv 7887 rank 0 performs MPI_Sendrecv 7888 rank 1 performs MPI_Sendrecv 7889 rank 2 performs MPI_Sendrecv 7890 rank 0 performs MPI_Barrier 7891 rank 1 performs MPI_Barrier 7892 rank 2 performs MPI_Barrier 7893 rank 0 performs MPI_Sendrecv 7894 rank 1 performs MPI_Sendrecv

Slide 24

Examples of Log File (Deadlock) 9310 rank 1 performs MPI_Sendrecv 9311 rank 2 performs MPI_Sendrecv 9312 rank 0 performs MPI_Barrier 9313 rank 1 performs MPI_Barrier 9314 rank 2 performs MPI_Barrier 9315 rank 1 performs MPI_Sendrecv 9316 rank 2 performs MPI_Sendrecv 9317 rank 0 performs MPI_Sendrecv 9318 rank 1 performs MPI_Sendrecv 9319 rank 0 performs MPI_Sendrecv 9320 rank 2 performs MPI_Sendrecv 9321 rank 0 performs MPI_Barrier 9322 rank 1 performs MPI_Barrier 9323 rank 2 performs MPI_Barrier 9324 rank 1 performs MPI_Comm_rank 9325 rank 1 performs MPI_Bcast 9326 rank 2 performs MPI_Comm_rank 9327 rank 2 performs MPI_Bcast 9328 rank 0 performs MPI_Sendrecv WARNING: all customers are pending!

Slide 25

Examples of Log File (Deadlock: traceback on 0) timestamp= 9298: MPI_Sendrecv (*sendbuf, sendcount = 7220, sendtype = MPI_DOUBLE, dest = 2, sendtag = 1, *recvbuf, recvcount = 7220, recvtype = MPI_DOUBLE, source = 1, recvtag = 1, comm = self-characterized communicator, *status) timestamp= 9300: MPI_Sendrecv (*sendbuf, sendcount = 7220, sendtype = MPI_DOUBLE, dest = 1, sendtag = 1, *recvbuf, recvcount = 7220, recvtype = MPI_DOUBLE, source = 2, recvtag = 1, comm = self-characterized communicator, *status) timestamp= 9304: MPI_Barrier (comm = MPI_COMM_WORLD) timestamp= 9307: MPI_Sendrecv (*sendbuf, sendcount = 7220, sendtype = MPI_DOUBLE, dest = 2, sendtag = 1, *recvbuf, recvcount = 7220, recvtype = MPI_DOUBLE, source = 1, recvtag = 1, comm = self-characterized communicator, *status) timestamp= 9309: MPI_Sendrecv (*sendbuf, sendcount = 7220, sendtype = MPI_DOUBLE, dest = 1, sendtag = 1, *recvbuf, recvcount = 7220, recvtype = MPI_DOUBLE, source = 2, recvtag = 1, comm = self-characterized communicator, *status) timestamp= 9312: MPI_Barrier (comm = MPI_COMM_WORLD) timestamp= 9317: MPI_Sendrecv (*sendbuf, sendcount = 7220, sendtype = MPI_DOUBLE, dest = 2, sendtag = 1, *recvbuf, recvcount = 7220, recvtype = MPI_DOUBLE, source = 1, recvtag = 1, comm = self-characterized communicator, *status) timestamp= 9319: MPI_Sendrecv (*sendbuf, sendcount = 7220, sendtype = MPI_DOUBLE, dest = 1, sendtag = 1, *recvbuf, recvcount = 7220, recvtype = MPI_DOUBLE, source = 2, recvtag = 1, comm = s

Recommended
View more...