Debugging: Interactive techniques and print statements

Trying to fix a bug in a program can be a long and tedious process. Learning what techniques to use can save you from headaches and wasted time. Starting out, I figured that a single best tool for debugging must exist. But which was it? Were print statements the answer? Or was it interactive debugging (like that described in “Debugging in Python (using PyCharm) Parts 1, 2, and 3)?

Speaking to different people and reading forums, I could not find a consensus. Some people would refer to print statements as “a lost art”, while others criticize print statements as “poor substitutes [for interactive debugging] and, in fact, at times dangerous tools.” I’ve read many accounts of experienced programmers who swear by print statements but feel embarrassed to admit it. As if it were taboo to use such a simple technique even if it was effective at solving their problem.

There are strong opinions on either side, but based on my experiences, I believe the answer lies somewhere in the middle. Interactive debugging and print statements are two effective techniques which each have a time and a place. I think this post summed up my opinion on the matter well:

“Print statements and debugger are not mutually exclusive. They are just different tools available to you in order to locate/identify bugs. There are those who will claim how they never touch a debugger and there are those who do not have a single logging/print statement anywhere in the code that they write. My advice is that you do not want to be in either one of those groups.” 

Below, I’ve compiled some opinions which highlight the benefits and drawbacks of each technique. Hopefully, these can serve as a guide for your future debugging needs!

Interactive Debugging

  • Less recompiling: with interactive debugging you can change variable values while the program is running. This gives you the freedom to test new scenarios without the need to recompile.
  • Get custom notifications: set up watch variables which notify you when that variable changes
  • Control: step into functions of interest or skip over functions that are not important for debugging. You can also set conditional breakpoints which are only activated when a certain value is triggered.
  • Use an IDE or the command line: debugging can be performed in an IDE (interactive development environment) or from the command line. IDEs are generally preferred, but there are instances—such as when using a command line interface to access a supercomputer—when they cannot be used. In these circumstances, interactive debugging can still be performed from the command line with tools such as GDB. Furthermore, most of these tools can be run with a text user interface (e.g. $gdb –tui).
  • Travel back in time: view the call stack at any time and inspect values up the stack trace. This can be an especially helpful feature when determining why a program has crashed. For example, see “What to Do After a Crash” in this GDB post.
  • Scales well with large projects: Although I don’t have much experience in this area, this post claims that interactive debugging is better suited for large projects or those with parallel code.
  • No clean-up: unlike print statements which often must be cleaned up (deleted or commented out) after debugging is done, there is no trace left behind from interactive debugging.

Print Statements

  • Reproducibility: leaving print statements in your code can help you reproduce past debugging sessions or help collaborators that may need to debug the program in the future. And instead of commenting out or deleting these print statements when not in use, they can be placed within an if-statement with a debugging flag that can be turned on and off (e.g. if (debug == TRUE) print <value>).
  • Consistency between environments: interactive debugging can be done via the command line, but the experience does not generally compare to the same task in an IDE. Print statements, however, give the user a consistent experience across environments.
  • Permanent record-keeping: when an interactive debugging session is over, there is no record of the information that the user came across. On the other hand, print statements can be used to create an extensive diagnostic report of the program. The user can then analyze this report to gain more insight about the program at any time in the future.
  • Easy to use: print statement are simple to use and understand. Although interactive debugging has nice bells and whistles, there is a learning curve for new users.

Thanks for reading and please feel free to edit and add your own thoughts!




Debugging in Python (using PyCharm) – Part 2

Debugging in Python (using PyCharm) – Part 2

This post is part 2 of a multi-part series of posts intended to provide discussion of some basic debugging tools that I have found to be helpful in developing a pure Python simulation model using a Python Integrated Development Environment (IDE) called PyCharm. You can see part 1 of the debugging discussion here, and other discussion of useful PyCharm functionality here.

In this post I will focus on PyCharm’s “Profile” features, which are very useful for debugging by allowing you to see what parts of your program (e.g., modules, classes, methods) are/are not being called, how frequently, and how much time is being spent in them. This has been especially useful to me when my program runs without error messages, but isn’t exactly performing as I expect it to (e.g., the program is taking a very long time to produce a result).

Step 1. Open your python files in PyCharm. In this case, my file is called I suggest you do this with a file that already executes without errors.

Step 2. In menu at the top of the screen, click Run–>Profile SedSim_v2 (or your file name), as is shown in the image below.

Fig. 1

Step 3. This will open a new window in the “Run” section (PyCharm will not execute the file in a console).    Wait for the profiling run to be completed, then within the Run window click on the upper (“capture snapshot”) icon. This will open a new file that summarizes our profile run with a “Statistics” tab and a “Call Graph” tab.

The statistics tab contains an overwhelming amount of information. A screenshot is shown below for just some of the Statistics tab.

Fig 2

The call graph presents a visual summary of function/method calls. An example screen shot of this is shown below. Note that in this case, the diagram we are seeing has a “hierarchical group” layout. Other groups include circular, organic, orthogonal, and directed orthogonal. Right-clicking in the call graph space will reveal various options, including diagram layout.

FIg 3

In the photo below I have zoomed in on a particular area of the figure. We can learn a lot from the small amount of information presented here, as I will describe below.

Fig 4

From the information presented in this figure, we can gather that is responsible for 100% of execution time (this, of course, makes sense, as this was the top-level file I ran). I am still learning about the Profiling features in PyCharm, so my very loose interpretation of the remaining information is as follows. The “Total” run took 264,381 milliseconds (or about 264 seconds). Only 1.7% (“Own” time) of this run-time was taken up by actions within that did not involve calling the other branched methods/functions listed in this diagram.

One can then follow the other branches in the Call diagram to see how much of the time spent calling methods outside of SedSim_v2 was spent in those methods. For example, the “Master_Method_Caller” method (which is a method of the Reservoir class in my model), shown in the upper left of the figure, is responsible for 21.3% of the run time. This method is called 73,000 times.

Step 4. Make necessary adjustments to your code based on what you have discovered. If, for example, you are experiencing a very long program run time, the approach I outline above may allow you to pinpoint what methods are responsible. This diagram helped me figure out that the Reservoir_Volume_Reduction method (another method of the Reservoir class) is taking up far too much time for what it does. Something clearly needed to be fixed!

Debug in Real-time on SLURM

Debugging a code by submitting jobs to a supercomputer is an inefficient process. It goes something like this:

  1. Submit job and wait in queue
  2. Check for errors/change code
  3. (repeat endlessly until your code works)

Debugging in Real-Time:

There’s a better way to debug that doesn’t require waiting for the queue every time you want to check your code. On SLURM, you can debug in real-time like so:
  1. Request a debugging or interactive node and wait in queue
  2. Check for errors/change code continuously until code is fixed or node has timed out

Example (using Summit supercomputer at University of Colorado Boulder):

  1. Log into terminal (PuTTY, Cygwin, etc.)
  2. Navigate to directory where the file to be debugged is located using ‘cd’ command
  3. Load SLURM
    • $module load slurm
  4. Enter the ‘sinteractive’ command
    • $sinteractive
  5. Wait in line for permission to use the node (you will have a high priority with a debugging QOS so it shouldn’t take long)
  6. Once you are granted permission, the node is yours! Now you can debug to your hearts content (or until you run out of time).
I’m usually debugging shell scripts on Unix. If you want advice on that topic check out this link. I prefer the ‘-x’ command (shown below) but there are many options available.
Debugging shell scripts in Unix using ‘-x’ command: 
 $bash -x mybashscript.bash
Hopefully this was helpful! Please feel free to edit/comment/improve as you see fit.
Debugging in Python (using PyCharm) – Part 1

Debugging in Python (using PyCharm) – Part 1

This blog post is the first of a multi-part series (here are the second and third debugging blog posts) intended to provide discussion of some basic debugging tools that I have found to be helpful in developing a pure Python simulation model using a Python Integrated Development Environment (IDE) called PyCharm. (Some of this discussion surely applies to other languages and development environments as well. I just want folks to know that I’ve prepared this post with Python/PyCharm in mind).  The post was updated with some additional tips from Joe Kasprzyk, on 10/15/2015.

If you didn’t catch my previous post, I strongly recommend PyCharm as a Python IDE, and have described in that post how to get access to PyCharm. PyCharm can be a bit heavy just for scientific computing, but in developing a large, object-oriented simulation model, I have found PyCharm’s code completion and debugging features to be a major asset for coding quickly and efficiently.

When I am debugging a program, there are often two fundamental issues at hand. The first is to make sure the program runs without any syntax errors (or any other errors generated while interpreting your code). The second is to be confident that your code is doing what you think it is. That is, even if it runs, there may be problems within the implementation of the routines you have coded. Python’s highly intelligent code editor is very helpful for solving the former problem before you even run your program: it will highlight errors on the fly, help you complete code to avoid variable misspellings, etc. However, to handle the latter issue, you may want to glimpse into the code’s functioning (e.g., variable values) as the program executes. I will now describe some of PyCharm’s debugging features that I have found to be very helpful in this regard.

I prepared the instructions and figures that appear below with the Professional version of PyCharm (JetBrains PyCharm 4.5.2), and I’m using it in Windows 7.

I am going to show examples of how to use the Pycharm’s debug features on a very simple Python script, which appears below. However, clearly you can follow these same steps for your own files. This script should work, so it shouldn’t need to be debugged, but I’ll show how I would use debugging tools to check the script anyway.

# My program file
import matplotlib.pyplot as plt
import numpy as np

# Initialize empty arrays
array_len = 10000  # Desired length of arrays
var1 = np.zeros(array_len)
var2 = np.zeros(array_len)
var3 = np.zeros(array_len)
var4 = np.zeros(array_len)

# Define two functions:

def function_1(input):
    # Function 1
    for i in range(array_len):
        var1[i] = 2 + i*input
        var2[i] = 2*var1[i]
    return var1, var2

def function_2(input):
    # Function 2
    for j in range(array_len):
        var3[j] = 3 + j*input
        var4[j] = 2*var3[j]
    return var3, var4

# Add var1 and var3, and plot result:
Output = function_1(1)[0] + function_2(2)[0]

Step 1. Open your files.

In PyCharm, open a file (or files) that you have coded, or are in the process of coding, and want to debug. For example, a screenshot of my open file appears below.

Fig 1

Step 2. Turn on the debugger.

There are numerous ways to do this. I prefer to open up a Python console to work from, and turn it on from there (as is shown in the screenshot below). To do this, click on the “Python console” button at the bottom of the screen, then click on the green bug icon on the console menu on the left to turn on the debugger. Note that you can actually have open multiple consoles (by clicking the “plus” icon on the console menu), and you can have debugging turned on in whichever consoles you prefer.

If you can’t figure out how to open up a Python console, you can also just click on the green bug icon in the menu at the top of the screen next to the play button, while your script is open. This may actually run the debugger as well, which is a step that comes later on in my post.

Fig 2

Step 3. Create breakpoints.

Breakpoints are spots in your code where you want to temporarily interrupt execution so you can glimpse into the program’s operation (e.g., view values of variables as the program runs). If you left click in the area just to the left of where the code editor stops, a closed red circle will appear, denoting a breakpoint.

In my example, I will set breakpoints on the lines where variables var2 and var4 are defined. (Note that I like to have line numbers turned on, particularly when debugging. You can access this features in PyCharm’s settings menu).

Breakpoints are only used when running the program in debug mode. That is, when you turn debugging off (e.g., by clicking once again on the bug icon in the console menu), the program will execute without stopping on breakpoints. Also, note that breakpoints are useful not only for viewing the program’s variable values and so forth, but also to see if the program is entering particular sections of code. If the program doesn’t stop on a breakpoint you’ve set, then the program is not entering the section of code where that breakpoint is located, which can be very valuable information.

Fig 3

Step 4. Create a run configuration and run the debugger.

There are several ways to do this as well. You can simply click on the green bug icon at the top of the screen again to run the debugger. Or, highlight your code, right click, and select the option to “Debug” your file.

Joe adds: If you want to do a quick debug of an uncomplicated script, the above will work well.  However sometimes you may want to debug a Python file that has command line arguments, such as  In order to create a configuration, right click on the configuration pull down menu in the upper right hand corner of the program, and click Edit Configurations…  Then you can add the command line arguments by typing them into the “Script parameters” box in that dialog window.  Then, you can continue debugging as usual.  Back to your regularly scheduled programming…

I have noticed that my programs execute significantly more slowly in debug mode, so you may experience the same. A debugger pane should appear as below.

Fig 4

Step 5. Set “Watches”.

Note the program execution has been interrupted on the “var1” line. Suppose we want to know the value of this local variable in its current loop. In the debugger pane, go to the “Watches” window, click the “plus” symbol, and enter “var1”. Repeat this process for the rest of the variables.

Fig 5

The debugger still shows no value for var1. This is because when the Pycharm debugger stops on a breakpoint, it interrupts execution before the breakpoint line is run, rather than after.

If you click on the “play” icon to resume execution in the debugger, you will see values appear in your “Watches” pane. (If you hold your cursor over the different icons in the debugging pane, an icon description at the bottom of the window appears, so you can figure out which buttons I’m referring to this way).

PyCharm will also display values of watch variables (as well as the counter i) in the actual code editor as well (note the new text in gray in the figure below). Watches are helpful for more complex programs than this one, where you could have hundreds or thousands of variables, attributes, objects, etc., and want to track the values only of specific ones, so you can check them as you debug.

Fig 6

Joe adds:

Step 6. Add some exception handling to find information about an error.

Sometimes it is unclear why it is that you’re getting a particular error, such as “list index out of range.”  The Python run may give you information about what line number caused the problem, but that may not be enough.  Exception handling provides a way to create a ‘soft landing’ when the program runs into a problem.  Check out the Python documentation of this here.

For example, I had a problem with the function ‘withobjectives’ within  It was telling me list index out of range.  This seemed strange to me, and setting a breakpoint would be a lot of work because I didn’t know when the actual error was happening.  But exception handling saves the day.  Here’s the original offending line:

for oo in oindices:

I add ‘try’ and ‘except’ statements around the offending line:

for oo in oindices:
        print &quot;You messed up. Here is row:&quot;
        print &quot;%s&quot; % row

Now, there will be a line printed to the console that shows you the variable that caused the problem. In this case, there was an error in the text file, and I can go and use a find command in a text editor to realize that there was an extra column somewhere in the file that should not have been there.

We return to the original post:

I have described just a few basic features. Other neat ones include:

  • If you want to temporarily turn off all breakpoints during the debug run, use “mute breakpoints”.
  • If you want the breakpoints only to turn on when particular conditions are satisfied, click on the “view breakpoints” icon. In the image below I am setting a rule that means the breakpoint will only interrupt execution when the counter (i) is >= 9998.

Fig 7

  • If you’ve been scrolling around in a big file and can’t find where the breakpoint is, click on the “Show Execution Point” icon.
  • You can change your code as you are debugging without any problem. So, if you discover and fix an error, you don’t have to stop the debugger and start over again. (In many cases, though, it obviously might be a good idea to just start again, in which case you can click on the “Stop” and/or “Rerun” icons).
  • In addition to watches, and values displayed by PyCharm in the editor, you can see the values of a variable (or attribute, etc.) by holding your mouse over that variable.
  • If you want to see plots of variables as you are debugging, you can import matplotlib and insert code in your script to generate a plot of those variables. PyCharm will print such figures during the debug. More details on matplotlib integration with the PyCharm debugger can be found here.

In a future post I will demonstrate PyCharm’s Coverage and Profile features, which are very useful for debugging by allowing you to see what parts of your program (e.g., modules) are/are not being accessed, how frequently, and how much time is being spent in them.

Debugging MPI By Dave Hadka

Dave wrote the following instructions on how to debug MPI in an email recently, and I thought I’d post it here as a private post on the blog.

In case this isn’t already known, here’s instructions I came up with for running gdb and valgrind on MPI programs:

Debugging MPI with GDB

1) Run an interactive PBS job:

qsub -I -l walltime=16:00:00 -l nodes=1:ppn=4

The interactive job will start you in your home folder. CD to your working directory.

2) Load the OpenMPI module with GNU GCC support:

module load openmpi/gnu

3) Compile your code with the -ggdb flag to include GDB debugging info in the executable.

4) Create the GDB script, gdbscript.txt, to run when GDB is launched.
This is needed since the program will not start running until the
GDB ‘run’ command is called, and we need to automatically run all
jobs on remote nodes. This will also enable logging to gdb.txt.

set logging on

5) Run the MPI program with GDB:

mpirun gdb -x gdbscript.txt ./mpiprog.exe

6) When the program exits or an error is detected, you will be left in
GDB. You can now use any GDB commands, or quit by typing ‘quit’.

Memory Checking MPI Programs

First, follow steps 1-3 above.

4) When the interactive PBS job starts, run the MPI program with Valgrind:

mpirun valgrind –tool=memcheck –log-file=valgrind_%p.txt ./mpiprog.exe

5) Look at the valgrind_NNNN.txt files that were created, one for each process,
to determine if any memory leaks occurred. Valgrind often detects
uninitialized values in the Open MPI code, which should be ignored.

Using gdb, and notes from the book “Beginning Linux Programming”

I just started thumbing through Beginning Linux Programming by Matthew and Stones.  It covers in great detail a lot of the issues we talk about on this blog often — how to debug code, how to program using the BASH shell, file input/output, different development environments, and making makefiles.

Using gdb

One tool I haven’t talked about much is the debugger, gdb.  It is a text-based debugging tool, but it gives you some powerful ways to step through a program.  You can set a breakpoint, and then make rules for what variables are being displayed in each iteration.

Let’s say you have a file, myCode.c that you compile into an executable, myCode.  Compile using the -g flag, and then start gdb on your code by entering “gdb ./myCode”.  If your code has command line arguments, you need to specify an argument to gdb like this:

gdb –args ./myCode -a myArgument1 -b myArgument2

The important phrase here is “–args”, two dashes and the word args, that appears after gdb.  That lets gdb know that your ./myCode program itself has arguments.

You can also set a breakpoint inside gdb (you’d need to do that before you actually run the code).  To do this, say at line 10, simply type “break 10”.  This will be breakpoint 1.  To create rules to display data at each breakpoint type “display”.  It will ask what commands you’d like… for example, to display 5 values of an array, the command is “display array[0]@5”, then “cont” to continue, and “end” to end.

After setting up your breakpoints, simply type “run” to run the code.

If your program has a segmentation fault, it will let you know what line the segmentation fault occurred at, and using “backtrace” you can see what functions called that line.

If you have a segfault and the program is halted, the nice thing is that all the memory is still valid and you can see the value of certain variables.  To see the value of variables say “print myVariableName”.  It is quite informative.  For example, if a variable has a “NAN” next to it, you know there may be something wrong with that variable, that could cause an error somewhere else.

Here’s one example of a possible problem in pseudocode:

levelA = 0;

levelB = 0;

myLevel = 0.5;

myFrac = myLevel / (levelA + levelB);

The fourth line there looks innocuous enough, but this will cause a “divide by zero” error given the levelA and levelB value.  In gdb, you may get a segfault on the fourth line, but a simple “print levelA” and “print levelB” will help you solve the problem.

Here’s a short link that explains the basics of gdb with more detail.

Other notes

Also interesting are several C preprocessor macros that can tell you what line, file, date, and time the code was compiled at.   Predictably, these are __LINE__ __FILE__ __DATE__ and __TIME__ (that’s two underscores for each).

I also like the bash scripting examples that are contained in the book.  They taught me about some Linux utilities like “cut” that are very helpful, and covered elsewhere on this blog.

Any additional tips and tricks are welcome in the comments!