DynamoRIO
Debugging

General Tips

  • Use debug builds (pass "-debug" to drrun) with notifications turned on to diagnose errors. In general, it is much easier to debug with the debug build of DynamoRIO. A release build crash may show up as an earlier, easier-to-diagnose assert in debug build.
  • Try running without any client to isolate where the error is
  • Asserts can be suppressed with the -ignore_assert_list * option or with -checklevel 0.
  • Logging can be enabled with -loglevel N. Logs are written to DR-install/logs/. See our logging documentation for information on what is contained in the logs.
  • Use the load_syms or load_syms64 script to locate client and private libraries on Windows (-no_hide is no longer necessary (it never helped with the client library anyway)). Pre-install these are in tools/windbg-scripts/load_syms and tools/windbg-scripts/load_syms64. I simply change my windbg shortcuts to point at these via -c (please note: these are meant for attaching to a running process that is under DynamoRIO control):

    "C:\Program Files (x86)\Debugging Tools for Windows\windbg.exe" -pt 1 -c "$><E:\derek\dr\git\src\tools\windbg-scripts\load_syms"
    "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe" -pt 1 -c "$><E:\derek\dr\git\src\tools\windbg-scripts\load_syms64"

    For debugging a 32-bit process with the 64-bit windbg, use the load_symsWOW64 variant. However, this is less well-tested than using a 32-bit windbg on a 32-bit process.

  • To locate client and private libraries on Linux, use the add-symbol-file commands printed out at start time (see below for more information).
  • Use read watchpoints instead of breakpoints in application code, as the trap instruction inserted by the debugger into the application code can end up copied into DynamoRIO's code cache, resulting in an unhandled trap.
  • On Windows, if an application invokes OutputDebugString() while under a debugger, DynamoRIO can end up losing control of the application.
  • A SIGSEGV or Access Violation observed in a debugger does not necessarily indicate a problem. DynamoRIO uses "safe read" operations to access untrusted application addresses and can incur faults which are handled and continued past.
  • DynamoRIO disables itself when Windows is booted in safe mode (without networking). Thus, if a crash occurs in a Windows service under DynamoRIO, rebooting in safe mode will allow recovery.
  • If a client library doesn't seem to function for a given process, it is possible that the client library wasn't loaded due to permissions errors.

    One of the common situations where this happens is when the target application runs as a different user than the user who created the client library. This results in the application process not having the right permissions to access the client library.

    Try running the process under the debug mode of DynamoRIO (see dr_register_process()), where diagnostic messages are raised on errors like client library permissions. To see all messages, set the notification options like -msgbox_mask and -stderr_mask options to 0xf (see DynamoRIO Runtime Options). This will alert you to the problem.


Debugging On Linux

On Linux we use gdb for debugging. Note that we now use debug information files that are separate from their corresponding shared libraries: e.g., libdynamorio.so and libdynamorio.so.debug. There is a section inside the shared library that identifies the debug file.

Expected Signals

On Linux, DR sends itself a SIGILL signal during initialization, in order to measure the size of the signal frame used by the kernel. If you are under gdb, simply continue past this signal. SIGILL (SIGFPE on MacOS; SIGSTKFLT inside QEMU where the others crash QEMU (but note the poor support for SIGSTKFLT in gdb)) is also sent to other threads when attaching to a multi-threaded application and to suspend or terminate them later on during execution.

Additionally, DR uses various "safe read" strategies where it may raise a SIGSEGV or SIGBUS while examining application memory. Load symbols and look for safe_read variants on the call stack (such as safe_read_tls_magic, safe_read_asm_pre, safe_read_fast) to identify these. Just like with the init-time SIGILL, simply continue past these.

Attaching

Use the DynamoRIO runtime option -msgbox_mask with a desired mask, optionally along with -pause_via_loop if the application uses stdin, to cause a target application to wait and let you attach gdb. To attach early on, use a debug internal build and set the mask for informational messages. See the option descriptions in the API documentation.

Launching From GDB

For fast iterative debugging of small applications, it's nice to be able to launch the app under DR under gdb with one command:

$ gdb --args bin64/drrun -ops... -- ./path/to/myapp

However, be sure to run this command within gdb prior to running the app to ensure success:

set disable-randomization off

One drawback over attaching is that gdb's breakpoints in the loader can interfere with the dlopen() execution, and you will have to continue through them. DR's safe_read faults may also show up. It's best to ignore them via:

handle SIGSEGV nostop pass
handle SIGBUS nostop pass

If you have control of the target application you can build it with the start/stop interface so that it invokes DynamoRIO and then run it under gdb just like a native application.

Loading Client Symbols

To isolate clients and their libraries from the application, DR uses its own private loader to load clients. By default, gdb can only see libraries loaded by the glibc loader (ld-linux.so). To work around this problem, DR prints the gdb commands necessary to load symbols in a debug build. It looks like this:

<Starting application /usr/local/google/home/rnk/dynamorio/build/suite/tests/bin/client.events (10801)>
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/user/dr/suite/tests/bin/libclient.events.dll.so' 0x00007f14e46143e0
add-symbol-file '/home/user/dr/lib64/debug/libdynamorio.so' 0x00007f152865c000
add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007f1528280320
add-symbol-file '/lib64/ld-linux-x86-64.so.2' 0x00007f15285b6090
>

To pause the program and allow entering these commands when they are first printed, the -msgbox_mask 15 option can be used (see the example below).

In a release build, that message is not printed. However, the same string of commands is available in the global variable gdb_priv_cmds, which can be displayed within gdb with something like this:

x/3s gdb_priv_cmds

You can also use the -no_private_loader option to use the system loader to load the client, although this will break many clients and apps and is no longer officially supported.

To manually generate the symbol file loading commands, you want to tell gdb where the .text segment is.

echo add-symbol-file '/home/user/dr/lib64/debug/libdynamorio.so' 0x$(objdump -h /home/user/dr/lib64/debug/libdynamorio.so | grep text | awk '{print $4}')

You will need to adjust that address based on where the library was actually loaded: simply add the current base address from the maps file or DR's diagnostic printout and subtract the preferred base.

Loading DynamoRIO Symbols

The add-symbol-file commands shown in the prior section include symbols for the DynamoRIO library. If you need symbols for the DR library itself before DR sets up those commands, in some cases the debugger loads them properly for you and you do not need to do anything special. However, some versions of gdb load DR's symbols at the wrong address when you launch a process from within the debugger and you may need to clear the symbols first, before executing any add-symbol-file ... commands, by running:

symbol-file

The debugger also gets DR's symbols wrong when DR reloads itself to avoid gaps between its segments. The repository contains a gdb python script to load the libdynamorio.so symbols which is the simplest way to automagically get the correct symbols.

Example Gdb Session

Below is a sample gdb session where we break on the drtable_create function in the drcov tool. We use -msgbox_mask 15 in debug build to get a pause at each message including the one with all the symbol commands. At that message, we hit Control-C to interrupt into the debugger where we paste the symbol loading commands. Then we can use regular gdb breakpoint commands. Upon continuing, we have to remember to hit enter as the message is still waiting. We also disable SIGBUS to avoid safe-read faults from adding noise to the picture.

derek@dynamorio:~/dr/build$ gdb --args bin64/drrun -msgbox_mask 15 -t drcov -- suite/tests/bin/simple_app
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
...
Reading symbols from bin64/drrun...
Reading symbols from /home/derek/dr/build/bin64/drrun.debug...
(gdb) r
Starting program: /home/derek/dr/build/bin64/drrun -msgbox_mask 15 -t drcov -- suite/tests/bin/simple_app
process 3239276 is executing new program: /home/derek/dr/build/lib64/debug/libdynamorio.so
<Starting application /home/derek/dr/build/suite/tests/bin/simple_app (3239276)>
<press enter to continue>
<Initial options = -no_dynamic_options -client_lib '/home/derek/dr/build/bin64/../clients/lib64/debug/libdrcov.so;0;' -client_lib64 '/home/derek/dr/build/bin64/../clients/lib64/debug/libdrcov.so;0;' -code_api -msgbox_mask 15 -stack_size 56K -signal_stack_size 32K -nop_initial_bblock -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<press enter to continue>
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/derek/dr/build/bin64/../clients/lib64/debug/libdrcov.so' 0x0000ffffb3f8ba70
add-symbol-file '/home/derek/dr/build/lib64/debug/libdynamorio.so' 0x0000000071019430
add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrcovlib.so' 0x0000ffffb3fc7f70
add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrx.so' 0x0000ffffb3fe43b0
add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrreg.so' 0x0000ffffb4000130
add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrmgr.so' 0x0000ffffb401f470
>
<press enter to continue>
^C
Program received signal SIGINT, Interrupt.
0x000000007145e744 in ?? ()
(gdb) set confirm off
(gdb) add-symbol-file '/home/derek/dr/build/bin64/../clients/lib64/debug/libdrcov.so' 0x0000ffffb3f8ba70
add symbol table from file "/home/derek/dr/build/bin64/../clients/lib64/debug/libdrcov.so" at
.text_addr = 0xffffb3f8ba70
Reading symbols from /home/derek/dr/build/bin64/../clients/lib64/debug/libdrcov.so...
Reading symbols from /home/derek/dr/build/clients/lib64/debug/libdrcov.so.debug...
(gdb) add-symbol-file '/home/derek/dr/build/lib64/debug/libdynamorio.so' 0x0000000071019430
add symbol table from file "/home/derek/dr/build/lib64/debug/libdynamorio.so" at
.text_addr = 0x71019430
Reading symbols from /home/derek/dr/build/lib64/debug/libdynamorio.so...
Reading symbols from /home/derek/dr/build/lib64/debug/libdynamorio.so.debug...
(gdb) add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrcovlib.so' 0x0000ffffb3fc7f70
add symbol table from file "/home/derek/dr/build/ext/lib64/debug/libdrcovlib.so" at
.text_addr = 0xffffb3fc7f70
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrcovlib.so...
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrcovlib.so.debug...
(gdb) add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrx.so' 0x0000ffffb3fe43b0
add symbol table from file "/home/derek/dr/build/ext/lib64/debug/libdrx.so" at
.text_addr = 0xffffb3fe43b0
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrx.so...
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrx.so.debug...
(gdb) add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrreg.so' 0x0000ffffb4000130
add symbol table from file "/home/derek/dr/build/ext/lib64/debug/libdrreg.so" at
.text_addr = 0xffffb4000130
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrreg.so...
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrreg.so.debug...
(gdb) add-symbol-file '/home/derek/dr/build/ext/lib64/debug/libdrmgr.so' 0x0000ffffb401f470
add symbol table from file "/home/derek/dr/build/ext/lib64/debug/libdrmgr.so" at
.text_addr = 0xffffb401f470
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrmgr.so...
Reading symbols from /home/derek/dr/build/ext/lib64/debug/libdrmgr.so.debug...
Breakpoint 1 at 0xffffb3fcc408: file /home/derek/dr/src/ext/drcontainers/drtable.c, line 171.
(gdb) handle SIGBUS nostop noprint pass
Signal Stop Print Pass to program Description
SIGBUS No No Yes Bus error
(gdb) c
Continuing.
Breakpoint 1, drtable_create (capacity=4096, entry_size=8, flags=0, synch=1 '\001', free_entry_func=0x0) at /home/derek/dr/src/ext/drcontainers/drtable.c:171
171 DR_ASSERT(entry_size > 0 && entry_size < MAX_ENTRY_SIZE);
(gdb) bt
#0 drtable_create (capacity=4096, entry_size=8, flags=0, synch=1 '\001', free_entry_func=0x0) at /home/derek/dr/src/ext/drcontainers/drtable.c:171
#1 0x0000ffffb3fc84ac in bb_table_create (synch=1 '\001') at /home/derek/dr/src/ext/drcovlib/drcovlib.c:188
#2 0x0000ffffb3fc8778 in thread_data_create (drcontext=0x0) at /home/derek/dr/src/ext/drcovlib/drcovlib.c:258
#3 0x0000ffffb3fc88b0 in global_data_create () at /home/derek/dr/src/ext/drcovlib/drcovlib.c:282
#4 0x0000ffffb3fc930c in event_init () at /home/derek/dr/src/ext/drcovlib/drcovlib.c:543
#5 0x0000ffffb3fc953c in drcovlib_init (ops=0xffffffffdec8) at /home/derek/dr/src/ext/drcovlib/drcovlib.c:599
#6 0x0000ffffb3f8c0fc in dr_client_main (id=0, argc=1, argv=0xfffdb3fcda28) at /home/derek/dr/src/clients/drcov/drcov.c:179
#7 0x00000000711e1f5c in instrument_init () at /home/derek/dr/src/core/lib/instrument.c:772
#8 0x000000007102facc in dynamorio_app_init_part_two_finalize () at /home/derek/dr/src/core/dynamo.c:716
#9 0x000000007144c490 in privload_early_inject (sp=0xfffffffff420, old_libdr_base=0xfffff7a64000 <error: Cannot access memory at address 0xfffff7a64000>,
old_libdr_size=5881856) at /home/derek/dr/src/core/unix/loader.c:2254
#10 0x000000007140cc3c in _start () at /home/derek/dr/src/core/arch/aarchxx/aarchxx.asm:68

Memory Querying

Another useful script provided in the repository is a memory query script to print the line in the maps file matching a given address.

(gdb) memquery $rsp
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]

Call Stacks

gdb has trouble with generating call stacks at various points during DR execution, such as when at a system call in generated code. Manually setting the stack and frame pointers can solve this:

# optional args: $arg0=frame count, $arg1=start frame ptr
define setbt
if $argc < 1
set var $max=5
else
set var $max=$arg0
end
if $argc < 2
set var $myfp=$ebp
else
set var $myfp=$arg1
end
set var $count=0
while $count < $max && $myfp != 0
if $count == 1
set var $esp = $myfp
end
if $count == 2
set var $ebp = $myfp
end
printf "\nframe %d\n", $count
x/4dx $myfp
info line * *((long*)($myfp+sizeof(long)))
x/2i *((long*)($myfp+sizeof(long)))
set var $myfp=*((long *)$myfp)
set var $count=$count+1
end
bt
end
# optional args: $arg0=frame count, $arg1=start frame ptr
define setbt64
if $argc < 1
set var $max=5
else
set var $max=$arg0
end
if $argc < 2
set var $myfp=$rbp
else
set var $myfp=$arg1
end
set var $count=0
while $count < $max && $myfp != 0
if $count == 1
set var $rsp = $myfp
end
if $count == 2
set var $rbp = $myfp
end
printf "\nframe %d\n", $count
x/4gx $myfp
info line * *((long*)($myfp+sizeof(long)))
x/2i *((long*)($myfp+sizeof(long)))
set var $myfp=*((long *)$myfp)
set var $count=$count+1
end
bt
end

For examining the stack, I find this function useful:

# Equivalent to windbg's dps command.
# Pass it the stack pointer range: [start, end)
define dps
set var $start = (char *) $arg0
set var $end = (char *) $arg1
set var $ptr = $start
while $ptr < $end
set var $retaddr = *((unsigned long *)$ptr)
# I don't like how %p prints "(nil)" so:
if sizeof(void*) == 8
printf "0x%016lx 0x%016lx ", $ptr, $retaddr
else
printf "0x%08x 0x%08x ", $ptr, $retaddr
end
# We have to cast to void* to avoid gdb failing to properly evaluate
# $retaddr (ends up using value from 1st iter on each iter).
#
# XXX: can we get the result of this into a var and not print it
# if it says "No symbol matches..."?
info symbol (void *)$retaddr
set var $ptr = $ptr + sizeof(void*)
end
end

Switching Stacks

The frame command has never worked for me: it always prints #0 0x00000000 in ?? (). Instead I set $ebp, and sometimes have to also set $esp and $eip (which can only be set from frame 0). Or you can manually walk frame pointers:

(gdb) info symbol **(sc->ebp+4)
get_memory_info + 647 in section .text
(gdb) info symbol **(**(sc->ebp)+4)
check_thread_vm_area + 7012 in section .text
(gdb) info symbol **(**(**(sc->ebp))+4)
check_new_page_start + 79 in section .text
(gdb) info symbol **(**(**(**(sc->ebp)))+4)
build_bb_ilist + 514 in section .text

To get line numbers use info line instead:

info line **(**(sc->ebp+4))

Viewing Segment Bases

There is no way to view segment bases directly within gdb, but you can create a core file and mine it to find the bases. Here is how to do it for gs for 64-bit stored in its MSR (rather than the GDT):

$ sudo apt-get install elfutils
...
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffdf2af5700 (LWP 1166202))]
(gdb) generate-core-file mycore
Saved corefile mycore
(gdb) shell eu-readelf --notes mycore | grep -A 12 'pid: 1166202' | grep gs\.base
fs.base: 0x00007ffdf2af5700 gs.base: 0x00007ffdf2af1000

Debugging On Windows

Debugging Tools

The Visual Studio debugger is completely inadequate for debugging applications running under DynamoRIO. It frequently fails due to our manipulation of its injected thread, it has no command-line interface, etc. We use WinDbg (and its counterparts ntsd and cdb) exclusively. WinDbg is a low-level debugger that provides symbolic debugging too.

Install the Debugging Tools for Windows to get the full WinDbg. For debugging 32-bit applications, we recommend using WinDbg version 6.3.0017, not the newer versions 6.4 through 6.11, as they have problems displaying callstacks involving DynamoRIO code. However, if you cannot obtain 6.3.0017 (it is no longer supported), get the latest version. You'll have to go to extra effort to get a callstack when attaching at a DynamoRIO messagebox midway through execution for a 32-bit process (see below). For 64-bit, use the most recent version of WinDbg.

Most of the information about how to use this WinDbg is available from its excellent help file. Some key commands include:

  • Display callstacks of all threads: ~**kb
  • Select thread number N: ~Ns
  • Display callstack of current thread with numbered frames: kn
  • Select frame number N: .frame N
  • Display local variables (only useful in debug builds): dv
  • Display local variable x: dt x
  • Display all symbols in module M starting with XYZ: x M!XYZ**
  • Display stack with symbolic references: dds esp
  • Switch context (e.g., to an exception context): .cxr <CONTEXT address>
  • Display loaded modules: lm
  • Open a log file: .logopen <filename>
  • Set a breakpoint: bp <address>
  • Continue execution: g

See the tools/windbg-scripts scripts and the WindDbg Tips section below for further examples.

You also need access to the symbol files both for DynamoRIO and for the operating system libraries. You can use the .symfix command to automatically download symbols from the Microsoft symbol server. You should specify a local cache directory so that WinDbg doesn't query the server every time: .symfix c:\my\symbol\dir. The symbol path and cache directory are stored in a global environment variable _NT_SYMBOL_PATH. Here is an example path:

c:\build12012\release;srv*c:\symbols*http://msdl.microsoft.com/download/symbols

Note that you may want to remove any network paths once you have the symbols of interest locally to speed up debugging.

Attaching

To attach to a process on Windows, use the -msgbox_mask option and attach the debugger while the dialog box has paused the application. Use -msgbox_mask 15 to attach a program startup, or -msgbox_mask 12 to attach at a later error.

In order to attach press F6; this will show a list of all processes available and you can choose your application. You can also view the attach list through the File->Attach menu item. After you have attached click 'ok' on the pop up window.

Launching Within WinDbg

To get control of DynamoRIO when it starts initializing, invoke WinDbg from the cygwin command line as follows. Make sure WinDbg is in your path.

% windbg bin32/drrun.exe -- c:\\path\\to\\targetapp.exe

Once the debugger command line comes up, type the following commands, substituting your path to the directory where dynamorio.dll is located:

> .childdbg 1
> bp drinjectlib!inject_gencode_mapped_helper
> g
> gu
> r $t0=poi(map)
> g
> .sympath c:\mybuild\lib32\debug
> .reload dynamorio.dll=@$t0
> bp dynamorio!dynamorio_earliest_init_takeover
> g

Automatically loading symbols

The dynamorio.dll library is not loaded by the system loader, and so WinDbg does not automatically find it. Once DynamoRIO has initialized enough, a script that we provide will automatically load the symbols for DynamoRIO and all client libraries in use. See the top of this file for how to load these scripts at WinDbg startup. From within WinDbg, for 32-bit:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_syms

For 64-bit:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_syms64

For a 32-bit application but a 64-bit windbg:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_symsWOW64

These scripts will fail if the process is not running under DynamoRIO or if it has not finished DynamoRIO initialization: thus, they will not work at the early attach point described under Launching Within WinDbg above.

To manually point WinDbg at the library, you will need its directory and its base address. Then use these two commands:

> .sympath c:\path\to\lib32\debug
> .reload dynamorio.dll=15000000

Private Libraries

Libraries loaded for a client are not on the system library list. To identify these modules use the following command:

!list -t dynamorio!privmod_t.next -x "dt" -a "dynamorio!privmod_t" @@(dynamorio!modlist)

Symbols for private libraries can be automatically loaded using the load_syms script. For 32-bit:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_syms

For 64-bit:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_syms64

For a 32-bit application but a 64-bit windbg:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_symsWOW64

32-bit DynamoRIO Callstacks with WinDbg 6.4+

Windbg versions from 6.4 onward refuse to show a callstack if it's not part of the main stack for that thread, for 32-bit processes. You'll see just the top frame, often ntdll!NtRaiseHardError if you attached at a message box. Something like this should show the right callstack although it won't let you examine frames:

knĀ =esp esp eip

Another trick is to clobber the TEB stack fields:

$><c:\path\to\dynamorio\sources\tools\windbg-scripts\load_syms
!teb
r $t0=poi(dynamorio!tls_dcontext_offs)
r $t0=poi(fs:@$t0)
ed @$teb+4 @@(((dynamorio!dcontext_t*)@$t0)->dstack)
ed @$teb+8 @@(((dynamorio!dcontext_t*)@$t0)->dstack - dynamorio!dynamo_options.stack_size)
kn

Which will then result in the k commands working nicely (I've done this in 6.11.001.402; should work in all other versions).  Though of course only do this when not planning to continue app execution, or restore the TEB fields, just in case (that's what the !teb is for, to print out the original values for restoring).

Crash Callstacks

When DynamoRIO catches an unhandled exception, the callstack should have a dynamorio!intercept_exception frame. Navigate to that frame, where we have a CONTEXT* local var named cxt. Then change the thread's context and ask for a stack trace. So if you see in the initial callstack:

1. Child-SP RetAddr Call Site
00 00000000`bfc9df68 00000000`153f05f6 ntdll!NtRaiseHardError+0xa
01 00000000`bfc9df70 00000000`153a93ed dynamorio!nt_messagebox+0x176 [@ 3486](...\dr\git\src\core\win32\ntdll.c)
02 00000000`bfc9e010 00000000`15177168 dynamorio!debugbox+0x5d [@ 4570](...\dr\git\src\core\win32\os.c)
03 00000000`bfc9e040 00000000`153e132d dynamorio!notify+0x298 [@ 1956](...\dr\git\src\core\utils.c)
04 00000000`bfc9e8b0 00000000`15548cfc dynamorio!intercept_exception+0x27dd [@ 5521](...\dr\git\src\core\win32\callback.c)
05 00000000`bfc9ee60 00000000`bfc9ee80 dynamorio!interception_code_array+0xcfc

Execute these commands:

.frame 4
.cxr @@(cxt)
kn

And now you should see the callstack as of the exception itself.

Creating Core Dumps

You can create a core dump from windbg that allows others, or you at a later date, to re-analyze the bug:

.dump /ma c:\path\to\new\dump\file.dmp

It is good practice to always create a dump file when attaching windbg, just in case the bug is not reproducible.

Debugging .ldmp Files

.ldmp files are generated automatically by the core in certain failure situations. The failure situations on which a .ldmp file is created are specified by the -dumpcore_mask option (see the enum in core/os_shared.h), which defaults to 0x1ff in debug builds and to 0x0 in release builds. An additional parameter -dumpcore_violation_threshold controls the maximum number of violation .ldmp files to generate.

Once you have a .ldmp file you can use it to recreate the process for debugging purposes. To do so use the ldmp.exe utility in the tools module. The syntax is:

% ./ldmp.exe <ldmp_file> <dummy_executable>

where ldmp_file is the ldmp you wish to view and dummy_executable is a fully qualified absolute windows style path to dummy.exe (also located in the tools module).

A cygwin example:

% ./ldmp.exe ../foo.ldmp c:\\foo\\bar\\tools\\dummy.exe

A cmd shell example:

% ldmp ../foo.ldmp c:\foo\bar\tools\dummy.exe

Ldmp will print out a process id and a bunch of mapping information. Use WinDbg to attach to the process id specified, making sure to attach NON-INVASIVELY** (you will crash WinDbg with an invasive attach). You should now be ready to debug.

Note that process ids and thread ids will differ from the original process as well as peb and teb addresses (though not contents, so you can still use !teb and !peb). Ldmp.exe provides mapping information from original thread ids and teb addrs to new thread ids and teb addrs as well as mappings for any other memory regions that were moved. Note that ldmp.exe can only recreate threads that were in our all_threads list at the time the ldmp was created, though ldmp.exe will try to detect teb regions associated with the missing threads. Handle information will not be available. MEM_TYPE information is also lost, but it is available in the human readable parts of the ldmp file. It is expected that ldmp.exe will be unable to copy over the shared_user_data/vsyscall page (so note that if debugging across os versions).

Once you are finished you will need to use task manager or DRkill.exe to kill the recreated process (will show up as dummy.exe unlike in WinDbg). Ldmp files are also somewhat human readable.

Kernel debugging

Some debugging scenarios include services that start very early in the system initialization before we are able to start any programs - including debuggers. Also, sometimes even at a later stage a machine becomes completely unresponsive. Our only resort to getting control over such a machine is with a kernel debugger. Later we'll add notes here on how to use a kernel debugger.

System Dumps

If a Windows machine is hung without a chance of attaching a user mode debugger we can still get a full system dump with a keyboard command.

This dump file requires a pagefile on your boot drive that is at least as large as your main system memory. Also verify that you have free space on your drive for the dump file itself. This means you need at least 2xRAM to be able to do this.

How to set it up:

  • Go to Control Panel | System | Advanced | Startup and Recovery Settings
  • Choose a Complete Memory Dump and note the location: it should be SystemRoot%\MEMORY.DMP
  • Keep the overwrite setting checked, but remember after getting a dump to rename it with a meaningful name if you want to keep it
  • Go to Performance Settings | Advanced | Virtual Memory and make sure you have a large enough pagefile

To set up the key combination, set this registry key:

["CrashOnCtrlScroll"=dword:00000001

The magic key combination is [[Right|Ctrl]]+[[Scroll|Lock]] [[Scroll|Lock]]

See http://support.microsoft.com/kb/244139/ for details on changing which keys are used, in case you're using a KVM that messes it up.

Remote Debugging

There is also an easy way to debug remotely with WinDbg when you can actually start it on the target machine. This saves you the trouble of VNCing or to the target and we should give it a try. Of course, in this mode it is not secure at all - read up on docs for better ways of doing this if you like it.

% windbg -server tcp:port=1099 -p PID
% windbg -remote tcp:server=SERVER-NAME,port=1099
% "C:\Progra~1\Debugg~1\ntsd.exe" -server tcp:port=1227 -g -x "C:\WINNT\System32\notepad.exe"

You can also set this in

HKLM\Software\Microsoft\Windows NT\Current version\Image File Options

However, be careful not to try this on services like winlogon.exe on which it doesn't work. You will need a recovery CD if you mess it up.

Note that if you have already started a debug session you can easily convert it into a remote server with

> .server tcp:port=1099

and such debugging is much faster than over RDP or VNC, and helpful for sharing a debug session with someone else.

WinDbg Tips

These apply only to debug builds:

  • Dump stats: dt stats
  • See info on all locks - most importantly contention stats: dt innermost_lock -l next_process_lock

These work in both release and debug:

  • Always save to a logfile via the .logopen command. After debugging, copy the logfile to the bug directory to keep a record of your analysis. This is particularly important for a bug you gave up on or didn't have time to complete analysis of.
  • Dump all callstacks:
    • In user mode debugger: ~*kp
    • In kernel mode debugger on XP+: !process 0 1f [lsass.exe](HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters])
  • Check which build you're using in the target: lm vm dynamorio
  • Dump heap units:

    You need to obtain the heap.units address and then dump the first two fields in the !list command:

    ``` > dt dynamorio!heap units . +0x000 units : 0x1a151000 +0x000 start_pc : 0x1a15101c "1X@" +0x004 end_pc : 0x1a160fff "" ```

    Alternatively:

    ``` > ?? heap.units struct theHeapUnit * 0x1a151000

    > !list -t theHeapUnit.next_global -x "dd" -a "L2" 0x1a151000 dd 1a151000 L2 1a151000 1a15101c 1a160fff

    dd 1a131000 L2 1a131000 1a13101c 1a140fff ```

  • See current memory state with initial allocation information: !vadump -v
  • Use the scripts in the tools/windbg-scripts module to simplify analyses of DynamoRIO data structures.

    Invoke by setting up parameters in the pseudo-registers and then using the $>< command:

> r $t0=18345270
> $><c:\path\to\tools\windbg-scripts\fragment_flags

Addr2Line

The address_query.pl script in the tools module can be used for a command-line address-to-line utility. It requires that you have already built DRload.exe in the tools module. It will use cdb if you've installed the Debugging Tools for Windows in the standard location; otherwise it uses the ntsd present on every machine and goes through a temporary log file.

It can take as input stdin, a file, or command-line arguments listing the addresses to be queried. Here's the usage:

Usage: /i/dr/tools/address_query.pl [[-f <addrfile>](-raw]) [<debuggerpath>](-d) <DRdllpath> [... <addrN>](<addr1>)

Here's an example using a command-line argument:

% address_query.pl /i/dr/builds/11105/exports/x86_win32_rel/dynamorio.dll 0x7000ddb9
0x7000ddb9:
vmareas.c(2420)+0x1
(7000dc40) dynamorio!check_thread_vm_area+0x179 | (7000e1b0) dynamorio!prepend_fraglist

As another example, if you have a callstack in the file "cstack" with two columns (frame pointers followed by addresses), you could do something like this:

% awk '{print $2}' < cstack | address_query.pl /c/derek/dr/builds/11087/exports/x86_win32_rel/dynamorio.dll
77f830e7:
(77f82f56) ntdll!RtlMultiByteToUnicodeN+0x190 | (77f8e415) ntdll!RtlpExecuteHandlerForException
77f8d96b:
(77f8d86d) ntdll!LdrpResolveDllName+0x11c | (77f912cf) ntdll!RtlpInitDeferedCriticalSection
77f8da9e:
(77f8da40) ntdll!LdrpCreateDllSection+0x60 | (77f912cf) ntdll!RtlpInitDeferedCriticalSection
77f88a29:
(77f8b3cf) ntdll!RtlAllocateHeapSlowly+0x84f | (77f85a4e) ntdll!RtlFreeHandle

See the comments at the top of the script for more information.

#define DR_ASSERT(x)
Definition: dr_tools.h:114
void * drtable_create(ptr_uint_t capacity, size_t entry_size, uint flags, bool synch, void(*free_entry_func)(ptr_uint_t idx, void *entry, void *user_data))