MPIGDB

A wrapper around mpiexec, gdbserver, and gdb that makes debugging MPI programs easier with a moderate number of processes.

Example Usage

GDB Extensions and Useful commands

This wrapper defines several GDB extension commands that should help make things easier too.

mpic continue, but all processes

mpict continue this thread, and switch to the next one that is stopped if there is one

mpip print on all or a subset of threads using -t $tid

mpib break on all or a subset of threads using -t $tid

mpiw preform a command when all processes have exited

You also should probably know about the following buildin commands

thread apply all applies a command to all threads

continue & in GDB any command ending in & runs a command "in the background" allowing the user to continue to interact with GDB. In this case continue this thread in the background

info threads get a list of threads and their status

interupt if the current thread is running stop it.

thread $tid switch to thread id

CLI Arguments

• --mpigdb_frontend debugging frontend to use defaults to gdb. Also accepts vscode to support writing launch.json files for VSCode
• --mpigdb_dbg_arg pass a argument to GDB
• --mpigdb_helper path to the MPI GDB helper on the worker nodes
• --mpigdb_gdbserver path to the gdbserver helper on the worker nodes. default "gdbserve"
• --mpigdb_gdb path to the gdb executable. default: "gdb"
• --mpigdb_port port to connect on the master process. default "8000"
• --mpigdb_mpi_flag flags to pass to mpigdb
• --mpigdb_dryrun print out the command, but do not launch mpidgb

The flags --interpreter=mi and --tty= are forwarded to GDB hopefully enable usage from VSCode in the future.

Examples

mpigdb -np 8 -- ./play/build/heatdist

launch heatdist with 8 processes.

mpigdb -np 8 --mpigdb_dbg_arg -x --mpigdb_dbg_arg /tmp/debug.dbg -- ./play/build/heatdist

launch heat dist with 8 processes using the script /tmp/debug.dbg

mpigdb --mpigdb_gdb cuda-gdb --mpigdb_gdbserver cuda-gdbserver -np 8 -- ./play/build/heatdist

same, but use cuda-gdb to debug GPU programs

Building and Installation

Dependencies:

• gdb 12.1 or later with gdbserver which is sometimes packaged separately and python support which may be disabled if compiled from source
• a MPI installation including mpiexec mpiexec
• python 3.8 or later
• Rust+Cargo 1.65 or later

Earlier versions may work, but are not tested

Limitations and Known Bugs

For HPC systems with meaningful scale, we should probably integrate with PMIx, but we currently do not.

Does not currently work with LLDB which is required for MacOS -- PRs are accepted.

Has limited support for VSCode. See issue for some of the issues with a fuller implementation.

Acknowledgments

Credit to @mystery-e204 and his mpidb tool for an example of how to produce a file that can be used with VSCode. This approach is limited in that the debugger instances in vscode are seperate and can't easily be controlled together.