MUST detects usage errors of the Message Passing Interface (MPI) and reports them to the user. As MPI calls are complex and usage errors common, this functionality is extremely helpful for application developers that want to develop correct MPI applications. This includes errors that already manifest – segmentation faults or incorrect results – as well as many errors that are not visible to the application developer or do not manifest on a certain system or MPI implementation.
To detect errors, MUST intercepts the MPI calls that are issued by the target application and evaluates their arguments. The two main usage scenarios for MUST arise during application development and when porting an existing application to a new system. When a developer adds new MPI communication calls, MUST can detect newly introduced errors, especially also some that may not manifest in an application crash. Further, before porting an application to a new system, MUST can detect violations to the MPI standard that might manifest on the target system. MUST reports errors in a log file that can be investigated once the execution of the target executable finishes.
MUST provides correctness checks for the following classes of errors:
- Constants and integer values
- Communicator usage
- Datatype usage
- Group usage
- Operation usage
- Request usage
- Leak checks (MPI resources not freed before calling MPI Finalize)
- Type mis-matches
- Overlapping buffers passed to MPI
- Deadlocks resulting from MPI calls
- Basic checks for thread level usage (MPI_Init_thread)
Scalability of MUST depends strongly on the scalability of the attached application. For high scaling applications we successfully analyzed up to 16'000 parallel processes.
- MUST depends on the infrastructure libraries PnMPI and GTI.
- For installation you need a build environment including CMake 3.9 or higher and python 2.6 or 2.7
- GTI depends on libxml2. The headers can be found in the libxml2-dev(el) package of your distro.
- For more detailed error output you optionally may install dyninst from http://www.dyninst.org.
- Another optional dependency is graphviz for graphical representations of deadlock and datatype error situations.
For details on installation visit the Documentation (pdf).
MUST Known issues:
- Version 1.6-rc1: during execution pnmpif.so cannot be found. Workaround:
- export LD_LIBRARY_PATH=<must-install-dir>/lib:$LD_LIBRARY_PATH
this is fixed in 1.6-rc3
- Version 1.6-rc3: Some issues with Fortran file-IO and datatype functions are still under investigation
Release of MUST Version 1.6.0 (December 2019):
Includes fixes for building with certain MPI versions. Fixes some issues in Fortran codes.
MUST tool: MUST-v1.6.tar.gz
Release of MUST Version 1.6.0-rc3 (January 2019):
This version of MUST adds missing MPI-3 base datatypes; fixes some installation issues; speeds up tool startup with prebuilt configurations (see updated documentation); and adds support for multi-threaded applications.
MUST tool: MUST-v1.6-rc3.tar.gz
Release of MUST Version 1.6.0-rc1 (March 2018):
This version of MUST fixes some compatibility issues with newer versions of system libraries. Adds analysis for MPI-3 non-blocking collectives. And includes the build of GTI and PnMPI into a single build.
MUST tool: MUST-v1.6-rc1.tar.gz
see also Known issues
Release of MUST Version 1.5.0 (July 2016):
MUST tool: must-1.5.0.tgz
GTI infrastructure: gti-for-must-1.5.0.tgz
PnMPI infrastructure: pnmpi-for-must-1.5.0.tgz
Release of MUST Version 1.5.0-rc1 (February 2016):
MUST tool: must-1.5.0rc1.tgz
GTI infrastructure: gti-for-must-1.5.0rc1.tgz
PnMPI infrastructure: pnmpi-for-must-1.5.0rc1.tgz
Release of MUST Version 1.4.0 (March 2015):
This version of the tool is supposed to support MPI-3.0 compatible libraries
MUST tool: must-1.4.0.tgz
GTI infrastructure: gti-for-must-1.4.0.tgz
PnMPI infrastructure: pnmpi-for-must-1.4.0.tgz
Release Candidate of MUST Version 1.4.0 (November 2014):
MUST tool: must-1.4.0rc1.tgz
GTI infrastructure: gti-for-must-1.4.0rc1.tgz
PnMPI infrastructure: pnmpi-for-must-1.4.0rc1.tgz
Release of MUST Version 1.3.0 (March 2014):
MUST tool: must-1.3.0.tgz
GTI infrastructure: gti-for-must-1.3.0.tgz
PnMPI infrastructure: pnmpi-for-must-1.3.0.tgz