DDT Debugging Guide
Last update: December 9, 2016

DDT is a symbolic, parallel debugger providing graphical debugging of MPI, OpenMP, and Pthreads applications.

Setting up code and environment for debugging

Before running any debugger, the application code must be compiled with the "-g" and "-O0" options as shown below:

login1$ mpif90 -g -O0 mycode.f90

or

login1$ mpiCC -g -O0 mycode.c

Follow these steps to set up your debugging environment on Stampede, Lonestar and other compute resoures.

  1. Enable X11 forwarding: To use the DDT GUI, ensure that X11 forwarding is enabled when you ssh to the TACC system. Use the "-X" option on the ssh command line if X11 forwarding is not enabled in your ssh client by default.

    localhost$ ssh -X username@stampede.tacc.utexas.edu
  2. Then load the DDT module on the remote system along with any other modules needed to run the application:

    $ module load ddt mymodule1 mymodule2
  3. Start the debugger

    $ ddt myprogram

    If this error message appears:

    ddt: cannot connect to X server

    then X11 forwarding was not enabled or the system may not have local X11 support. If logging in with the "-X" flag doesn't fix the problem, please contact the help desk for assistance.

  4. Click the "Run and Debug a Program" button in the "DDT - Welcome" window.

  5. This displays the "DDT-Run" window, where you specify the executable path, command-line arguments, and processor count. Once set, they remain from one session to the next.

  6. Select each of the "Change" buttons in this window, and adjust the job parameters, as follows:

    • For the "Options" change button, set the MPI Implementation to either "mvapich 1", "mvapich 2", or "openmpi", depending on which MPI software stack you used to compile your program. Click OK.

    • For the "Queue Submission Parameters" window, fill in the following fields:

      Queue default queue is "development" queue
      Time (hh:mm:ss)
      Project Allocation to charge the batch job to
      Cores per node Actually, this is MPI tasks/node

      You must set the Project field to a valid project id. When you login, a list of the projects associated with your account and the corresponding balance should appear. The "Cores per node" setting controls how many MPI tasks will run on each node. For example, if you wanted to run with 4 MPI tasks per node and 4 OpenMP threads per task, this would be set to "4way". Click OK, and you will return to the "DDT-Run" window.

  7. Back in the "DDT-Run" window, set the number of tasks you will need in the "Number of processes" box. If you are debugging an OpenMP program, set the number of OpenMP threads also.

  8. Finally, click "Submit". A "DDT-Job Submitted" status box will appear.

Running DDT

When your job is launched by the SLURM scheduler, the DDT GUI will fill up with a code listing, stack trace, local variables and a project file list. Double click on line numbers to set break points, and then click on the play symbol (>) in the upper left corner to start the run.

DDT with Reverse Connect

By starting DDT from a login node you let it use X11 graphics, which can be slow. Using a VNC connection or the visualization portal is faster, but has its own annoyances. Another way to use DDT is through a "remote client" using "reverse connect". The remote client is a program running entirely on your private machine, and the reverse connection means that the client is contacted by the DDT program on the cluster, rather than the other way around.

  1. Download and install a remote client: https://www.allinea.com/developer-tools/debug-allinea-ddt/download. The client version and the DDT version on the TACC cluster have to correspond. Since the latest client is 6.1.x whereas TACC has DDT 6.0.x installed, you need to follow the link for older versions.

  2. Under "Remote Launch" make a new configuration:

    Fill in your login name and the cluster to connect to, for instance stampede.tacc.utexas.edu. The remote installation directory is "/opt/apps/ddt/6.0"

  3. Make the connection; you will be asked for your password and two-factor code:

  4. From any login node, submit a batch job where the mpirun line is replaced by:

    $ ddt --connect -n 123 ./yourprogram
  5. When your batch job (and therefore your ddt execution) starts, the remote client will ask you to accept the connection:

Your DDT session will now use the remote client.

References

The DDT user guide is available under the "Help" menu item.