5 tips for GNU Debugger

Bug tracking magnifying glass on computer screen

Image by:

Pixabay, testbytes, CC0

The GNU Debugger (gdb) is an invaluable tool for inspecting running processes and fixing problems while you're developing programs.

You can set breakpoints at specific locations (by function name, line number, and so on), enable and disable those breakpoints, display and alter variable values, and do all the standard things you would expect any debugger to do. But it has many other features you might not have experimented with. Here are five for you to try.

Conditional breakpoints

Setting a breakpoint is one of the first things you'll learn to do with the GNU Debugger. The program stops when it reaches a breakpoint, and you can run gdb commands to inspect it or change variables before allowing the program to continue.

For example, you might know that an often-called function crashes sometimes, but only when it gets a certain parameter value. You could set a breakpoint at the start of that function and run the program. The function parameters are shown each time it hits the breakpoint, and if the parameter value that triggers the crash is not supplied, you can continue until the function is called again. When the troublesome parameter triggers a crash, you can step through the code to see what's wrong.

(gdb) break sometimes_crashes
Breakpoint 1 at 0x40110e: file prog.c, line 5.
(gdb) run
[...]
Breakpoint 1, sometimes_crashes (f=0x7fffffffd1bc) at prog.c:5
5      fprintf(stderr,
(gdb) continue
Breakpoint 1, sometimes_crashes (f=0x7fffffffd1bc) at prog.c:5
5      fprintf(stderr,
(gdb) continue

To make this more repeatable, you could count how many times the function is called before the specific call you are interested in, and set a counter on that breakpoint (for example, "continue 30" to make it ignore the next 29 times it reaches the breakpoint).

But where breakpoints get really powerful is in their ability to evaluate expressions at runtime, which allows you to automate this kind of testing. Enter: conditional breakpoints.

break [LOCATION] if CONDITION

(gdb) break sometimes_crashes if !f
Breakpoint 1 at 0x401132: file prog.c, line 5.
(gdb) run
[...]
Breakpoint 1, sometimes_crashes (f=0x0) at prog.c:5
5      fprintf(stderr,
(gdb)

Instead of having gdb ask what to do every time the function is called, a conditional breakpoint allows you to make gdb stop at that location only when a particular expression evaluates as true. If the execution reaches the conditional breakpoint location, but the expression evaluates as false, the

debugger automatically lets the program continue without asking the user what to do.

Breakpoint commands

An even more sophisticated feature of breakpoints in the GNU Debugger is the ability to script a response to reaching a breakpoint. Breakpoint commands allow you to write a list of GNU Debugger commands to run whenever it reaches a breakpoint.

We can use this to work around the bug we already know about in the sometimes_crashes function and make it return from that function harmlessly when it provides a null pointer.

We can use silent as the first line to get more control over the output. Without this, the stack frame will be displayed each time the breakpoint is hit, even before our breakpoint commands run.

(gdb) break sometimes_crashes
Breakpoint 1 at 0x401132: file prog.c, line 5.
(gdb) commands 1
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>if !f
 >frame
 >printf "Skipping call\n"
 >return 0
 >continue
 >end
>printf "Continuing\n"
>continue
>end
(gdb) run
Starting program: /home/twaugh/Documents/GDB/prog
warning: Loadable section ".note.gnu.property" outside of ELF segments
Continuing
Continuing
Continuing
#0  sometimes_crashes (f=0x0) at prog.c:5
5      fprintf(stderr,
Skipping call
[Inferior 1 (process 9373) exited normally]
(gdb)

Dump binary memory

GNU Debugger has built-in support for examining memory using the x command in various formats, including octal, hexadecimal, and so on. But I like to see two formats side by side: hexadecimal bytes on the left, and ASCII characters represented by those same bytes on the right.

When I want to view the contents of a file byte-by-byte, I often use hexdump -C (hexdump comes from the util-linux package). Here is gdb's x command displaying hexadecimal bytes:

(gdb) x/33xb mydata
0x404040 <mydata>:    0x02    0x01    0x00    0x02    0x00    0x00    0x00    0x01
0x404048 <mydata+8>:    0x01    0x47    0x00    0x12    0x61    0x74    0x74    0x72
0x404050 <mydata+16>:    0x69    0x62    0x75    0x74    0x65    0x73    0x2d    0x63
0x404058 <mydata+24>:    0x68    0x61    0x72    0x73    0x65    0x75    0x00    0x05
0x404060 <mydata+32>:    0x00

What if you could teach gdb to display memory just like hexdump does? You can, and in fact, you can use this method for any format you prefer.

By combining the dump command to store the bytes in a file, the shell command to run hexdump on the file, and the define command, we can make our own new hexdump command to use hexdump to display the contents of memory.

(gdb) define hexdump
Type commands for definition of "hexdump".
End with a line saying just "end".
>dump binary memory /tmp/dump.bin $arg0 $arg0+$arg1
>shell hexdump -C /tmp/dump.bin
>end

Those commands can even go in the ~/.gdbinit file to define the hexdump command permanently. Here it is in action:

(gdb) hexdump mydata sizeof(mydata)
00000000  02 01 00 02 00 00 00 01  01 47 00 12 61 74 74 72  |.........G..attr|
00000010  69 62 75 74 65 73 2d 63  68 61 72 73 65 75 00 05  |ibutes-charseu..|
00000020  00                                                |.|
00000021

Inline disassembly

Sometimes you want to understand more about what happened leading up to a crash, and the source code is not enough. You want to see what's going on at the CPU instruction level.

The disassemble command lets you see the CPU instructions that implement a function. But sometimes the output can be hard to follow. Usually, I want to see what instructions correspond to a certain section of source code in the function. To achieve this, use the /s modifier to include source code lines with the disassembly.

(gdb) disassemble/s main
Dump of assembler code for function main:
prog.c:
11    {
   0x0000000000401158 <+0>:    push   %rbp
   0x0000000000401159 <+1>:    mov	%rsp,%rbp
   0x000000000040115c <+4>:    sub	$0x10,%rsp

12    	int n = 0;
   0x0000000000401160 <+8>:    movl   $0x0,-0x4(%rbp)

13    	sometimes_crashes(&n);
   0x0000000000401167 <+15>:    lea	-0x4(%rbp),%rax
   0x000000000040116b <+19>:    mov	%rax,%rdi
   0x000000000040116e <+22>:    callq  0x401126 <sometimes_crashes>
[...snipped...]

This, along with info registers to see the current values of all the CPU registers and commands like stepi to step one instruction at a time, allow you to have a much more detailed understanding of the program.

Reverse debug

Sometimes you wish you could turn back time. Imagine you've hit a watchpoint on a variable. A watchpoint is like a breakpoint, but instead of being set at a location in the program, it is set on an expression (using the watch command). Whenever the value of the expression changes, execution stops, and the debugger takes control.

So imagine you've hit this watchpoint, and the memory used by a variable has changed value. This can turn out to be caused by something that occurred much earlier; for example, the memory was freed and is now being re-used. But when and why was it freed?

The GNU Debugger can solve even this problem because you can run your program in reverse!

It achieves this by carefully recording the state of the program at each step so that it can restore previously recorded states, giving the illusion of time flowing backward.

To enable this state recording, use the target record-full command. Then you can use impossible-sounding commands, such as:

reverse-step, which rewinds to the previous source line
reverse-next, which rewinds to the previous source line, stepping backward over function calls
reverse-finish, which rewinds to the point when the current function was about to be called
reverse-continue, which rewinds to the previous state in the program that would (now) trigger a breakpoint (or anything else that causes it to stop)

Here is an example of reverse debugging in action:

(gdb) b main
Breakpoint 1 at 0x401160: file prog.c, line 12.
(gdb) r
Starting program: /home/twaugh/Documents/GDB/prog
[...]

Breakpoint 1, main () at prog.c:12
12    	int n = 0;
(gdb) target record-full
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000401154 in sometimes_crashes (f=0x0) at prog.c:7
7      return *f;
(gdb) reverse-finish
Run back to call of #0  0x0000000000401154 in sometimes_crashes (f=0x0)
	at prog.c:7
0x0000000000401190 in main () at prog.c:16
16    	sometimes_crashes(0);

These are just a handful of useful things the GNU Debugger can do. There are many more to discover. Which hidden, little-known, or just plain amazing feature of gdb is your favorite? Please share it in the comments.