User Tools

Site Tools


Debugging Code

The following page gives some examples of how to debug C code.


The Gnu DeBugger is a very powerful tool, but not easy to use. Beginners may first try insight, a very nice GDB-Gui.

KGDB over Serial Console

Debugging Kernel Modules

To make debugging of kernel modules useful at all, one needs to fill GDB with information about the symbols each loaded module contains. Since kernel modules are no regular executables, GDB doesn't find the relevant sections itself. But beware: we need to get rid of the relocation done to a module when it's being loaded. The easiest way to extract the section locations is via sysfs when the module has already been loaded:

$ cd /sys/module/gdb1/sections
 $ ls -A1
 .bss                       (where the BSS section was loaded)
 .data                      (where the data section was loaded)
 .text                      (where the text section was loaded)
 $ cat .text .data .bss     (the section addresses I care about)
 0xffffffffa00f4000         (address of module's text section ...)
 0xffffffffa00f4568         (... and data ...)
 0xffffffffa00f47c0         (... and BSS) 

then we can load the shared-object into gdb specifying the correct addresses:

(gdb) add-symbol-file .../gdb1.ko 0xffffffffa00f4000 \
 -s .data 0xffffffffa00f4568 \
 -s .bss 0xffffffffa00f47c0
 add symbol table from file ".../gdb1.ko" at
 	.text_addr = 0xffffffffa00f4000
 	.data_addr = 0xffffffffa00f4568
 	.bss_addr = 0xffffffffa00f47c0
 (y or n) y
 Reading symbols from .../gdb1.ko...done.

Printing Non-Global Variables

While global variables can easily be addressed by their (unique) name, things get more complicated with file- or function-local variables due to their limited scope. gdb though allows to qualify the variable name by prepending the file or function name:

(gdb) p 'test.c'::static_var
$2 = 42
(gdb) p main::local_var
$1 = 1337

Simple Kernel Debugging

gdb vmlinux /proc/kcore

This does not allow for run-time analysis (breakpoints et al.), but printing variables and the like.

In order to see changed variable values, one has to reload the core file first:

(gdb) core-file /proc/kcore

Enable Core-Dumps

If running a segfaulting program with gdb directly is not possible or not feasible, enabling core-dumps and analyzing the dump later may be an option. First, enable core dumps by setting maximum dump file size to something larger than zero (or, as shown here, unlimited):

$ ulimit -c unlimited

Dump output is controlled via sysctl settings kernel.core_pattern and kernel.core_uses_pid. The latter causes .PID appendix in whatever the further specified and is in fact a bit outdated since kernel.core_pattern supports format specifiers. These are:

%<NUL> '%' is dropped
%% output one '%'
%p pid
%P global pid (init PID namespace)
%i tid
%I global tid (init PID namespace)
%u uid (in initial user namespace)
%g gid (in initial user namespace)
%d dump mode, matches PR_SET_DUMPABLE and /proc/sys/fs/suid_dumpable
%s signal number
%t UNIX time of dump
%h hostname
%e executable filename (may be shortened)
%E executable path
%<OTHER> both are dropped

It is highly advisable to change kernel.core_pattern from its default of core because the segfaulting process may not be allowed to write to its current working directory (or it is not known). So better set an absolute path, such as /tmp/core.%p but keep security considerations in mind, dump files may contain sensitive data!


code/debugging.txt · Last modified: 2020/03/29 12:32 (external edit)