Using Dr. Memory

This section gives an overview of how to use Dr. Memory, divided into the following sub-sections:

See also the full reference of Dr. Memory's runtime options:

Dr. Memory Runtime Option Reference

Setup on Linux

This section describes requirements, setup instructions, and other concerns prior to running Dr. Memory on Linux.

Utilities

Ensure your Linux machine has the following utilities installed:

perl
objdump
addr2line

Setup on Windows

This section describes requirements, setup instructions, and other concerns prior to running Dr. Memory on Windows.

Dr. Memory displays results using notepad by default. If it is not on the PATH a message pointing to the file containing the results will be displayed. The option -batch can be used to disable the launch of notepad at application exit.

Dr. Memory and Cygwin

If you wish to run Cygwin applications under Dr. Memory, you will need to use the separate Cygwin build of Dr. Memory as the regular Windows build is not able to read Cygwin symbols. Also, make sure you have the following Cygwin utilities installed:

perl
objdump (binutils package)
addr2line (binutils package)

For this Alpha release, on Windows it is recommended to use native Windows applications (i.e., built via cl.exe or Visual Studio rather than g++) as there are many false positives in the Cygwin libraries.

The regular Dr. Memory build can be executed just fine from a Cygwin shell when targeting non-Cygwin applications.

Running Dr. Memory

The first step is to unzip or untar the Dr. Memory package.

Prepare Your Application

In order to obtain line number information, compile your target application with debugging information enabled (e.g., "/Zi" for cl.exe or "-g" for gcc or g++).

If you want to run your application with stripped libraries or binaries and have the unstripped versions in separate locations, you can point at those locations using the DRMEMORY_LIB_PATH environment variable, which is a colon-separated list of directories.

On Windows, when using static debug versions of msvcrt (e.g., building with the "/MTd" flag), if your application does not have symbols then Dr. Memory will not be able to identify and avoid conflicts with msvcrt's own heap debugging facilities, which may lead to false positives and/or false negatives. Be sure to build with debugging information included.

Invoking Dr. Memory

Run your application as you normally would from a command prompt (on Windows, either the cmd shell or a Cygwin prompt), with drmemory.exe and "--" prefixed to the command line (the "--" separates any arguments to Dr. Memory from the application being run).

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -- c:/path/to/my/app args to my app

On Windows, if you used the installer and checked the box to add drmemory.exe to your PATH, you can simply type into your shell:

: drmemory.exe -- c:/path/to/my/app args to my app

You can also drag-and-drop your application onto drmemory.exe in Windows Explorer or on the Desktop if you requested a Desktop shorcut upon installation.

When running scripts it is best to explicitly invoke the interpreter rather than invoking the script directly. For example:

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -- /usr/bin/perl ./myscript.pl

Examining the Results

Dr. Memory's results are written to a file called results.txt inside a subdirectory created for each invocation of Dr. Memory, inside Dr. Memory's log directory. The log directory by default is the logs subdirectory of the unpacked installation. It can be changed with the -logdir option:

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -logdir c:/logs -- myapp

When an application is executed under Dr. Memory's control, it creates a subdirectory in the base log directory named "DrMemory-<appname>.<pid>.NNN", where NNN is a counter that is incremented to ensure unique names.

For full details on the errors reported by Dr. Memory, see Error Types Reported by Dr. Memory.

If you would like to attach a debugger at the point an error is first detected, use the -pause_at_unaddressable or -pause_at_uninitialized options (see Dr. Memory Runtime Option Reference).

Aggregating Results From Multiple Processes

Dr. Memory supports aggregating results from multiple log directories, whether from multiple processes of a multi-process application or from multiple runs of the same application. Simply use the -aggregate option and supply the list of log directories, or a single directory containing the log directories as sub-directories to aggregate all of them:

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -aggregate DrMemory-myapp.1234.000 DrMemory-myapp.1235.000

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -aggregate /parent/logdir/

Applications That Do Not Exit

While Dr. Memory updates the results.txt file as the application runs, it does not perform leak checking or produce a summary of errors until the end of the run. For applications that do not have a normal exit, such as daemons, Dr. Memory provides a method of forcing end-of-run actions.

Run drmemory.exe with the -nudge option and the process identifier of the application in order to request leak checking and other end-of-run actions:

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -nudge processid

Post-Run Symbol Processing

Some applications have very large symbol files. To reduce resource usage during an application run, symbol processing can be disabled while the application executes and instead be performed after the run. Use the -skip_results option when running the application. Once finished, re-run Dr. Memory with the -results option and the log directory created during the run (which contains the raw data):

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -skip_results -- myapp myargs

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -results DrMemory-myapp.9876.000

If -results is invoked on a different machine where the application executable is not located at the same path as when it was executed with -skip_results, use the -results_app parameter to specify the executable along with the -results parameter.

The -skip_results option is not currently available on Windows.

Suppressing Errors

Dr. Memory generates a file named suppress.txt alongside the results.txt file. To suppress errors from being reported in future runs, edit this file to contain just the errors you wish to suppress. Then pass the file to drmemory.exe with the -suppress option:

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -suppress c:/suppress-custom.txt -- myapp

The suppress.txt generated by Dr. Memory will contain suppression information only for those errors that weren't suppressed using the -suppress option. For each error resported in suppress.txt, there will be two types of call stacks, one showing <module+offset> type frames and the other module!function types frames. Either type of call stack can be used and it is enough if one of those two is specified. The module!function are more general and more robust across different versions of a module. Also, the module!function callstacks support "*" and "?" wildcards in either the module or function name, and they support prefix matching. Prefix matching means that the callstack in the suppression file is considered a match if it matches the top of any actual callstack during a run. This allows specifying only the first few frames of a callstack and have it match any callstack whose first frames match those frames, regardless of subsequent frames. Also, you can use "..." lines in suppressions. Such lines match zero or more frames in the callstack. Currently the <module+offset> frames do not support either wildcards or prefix matching on Linux or Cygwin: on Windows they do support both just like module!function callstacks do.

Suppression files can have empty lines and comment lines (begining with #). There should be no leading white space for the actual suppression information. Suppressions should not end with "...", which is unnecessary due to the implicit prefix matching.

Another method is to filter the reported errors to focus on particular source files. Use the -srcfilter option to do this:

: ~/DrMemory-Windows-1.3.0-1/bin/drmemory.exe -srcfilter hello.c -- ./hello

The -srcfilter option is not available in the Windows version of Dr. Memory.

Dr. Memory comes with a set of default suppressions to avoid known false positives in system libraries. These can be disabled with the option -no_default_suppress.

Replaced Routines

Dr. Memory replaces several string and memory routines in the C library, executable, and other libraries, in order to avoid false positives from extremely-optimized versions of these routines. So if an application expects a crash inside a routine like strlen, do not be alarmed if it shows up as replace_strlen in the Dr. Memory library rather than strlen in the C library.

Error Types Reported by Dr. Memory

Each error found by Dr. Memory is listed in the results.txt file (see Examining the Results). The error report includes an error number, the type of error, the address and size of the memory access in question (if applicable to the error type), a timestamp and thread identifier indicating when and where the error was detected, and a callstack of the application at the point of the error.

The following subsections describe each type of error. Leaks are described in Memory Leaks.

Unaddressable Access

Dr. Memory considers any read or write of a memory location that was not allocated as an "unaddressable access". An allocation is:

A call to mmap (or mremap) (or Windows equivalents VirtualAlloc or MapViewOfFile) outside of a heap or stack region.
A call to malloc (or calloc or realloc, or Windows equivalents HeapAlloc or HeapReAlloc or the Local or Global versions) inside of a heap region
An extension of the stack in a stack region, either via raw instructions or via alloca.

An unaddressable access is an access to an invalid memory address. Examples include a buffer overflow, reading off the end of an array, reading or writing to memory that has been freed (often referred to as a "use-after-free" error), reading beyond the top of the stack, etc.

The heap allocator may pad the size requested by the application. The padded region beyond what the application asked for is considered unaddressable by Dr. Memory.

Consider this example code:

char *x = malloc(8);
char c = *(x+8);

Here is a diagram of the resulting heap allocation:

malloc header: unaddressable

x: uninitialized

x + 8: unaddressable

Dr. Memory will report an unaddressable access error on the second line of the example code, because x+8 is not an addressable location.

When an unaddressable access occurs in a heap region, Dr. Memory attempts to provide additional information about nearby valid allocations and whether the target address is inside a region that has been freed. For example, here is a sample error report:

Error #1: UNADDRESSABLE ACCESS: reading 0x098893f3-0x098893f4 1 byte(s)
Elapsed time = 0:00:00.136 in thread 18291
Note: prev lower malloc:  0x098893f0-0x098893f2
0x08048839 <malloc+0x839> malloc!main
    /work/dr/misc/internal/clients/drmemory/tests/malloc.c:95
0x0092dbb6 <libc.so.6+0x16bb6> libc.so.6<nosyms>!__libc_start_main
    ??:0
0x08048691 <malloc+0x691> malloc!_start
    ??:0

Note how the "prev lower malloc" ends at 0x098893f2 just before the invalid address 0x098893f3, indicating a read that went one byte too far.

Here is another example:

Error #2: UNADDRESSABLE ACCESS: reading 0x001337a8-0x001337ac 4 byte(s)
Elapsed time = 0:00:01.328 in thread 2072
Note: next higher malloc: 0x001337e0-0x00133830
Note: prev lower malloc:  0x00133708-0x0013373a
Note: 0x001337a8-0x001337ac overlaps freed memory 0x001337a8-0x001337b8
0x00401272 <suppress.exe+0x1272> suppress.exe!unaddr_test1+0x22
    e:\derek\dr\misc\internal\clients\drmemory\tests\suppress.c:79+0x3
0x004010dc <suppress.exe+0x10dc> suppress.exe!test+0x7c
    e:\derek\dr\misc\internal\clients\drmemory\tests\suppress.c:177+0xc
0x00401042 <suppress.exe+0x1042> suppress.exe!main+0x32
    e:\derek\dr\misc\internal\clients\drmemory\tests\suppress.c:199+0x0
0x00401bcd <suppress.exe+0x1bcd> suppress.exe!__tmainCRTStartup+0x15f
    f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c:327+0x12
0x7d4e9982 <KERNEL32.dll+0x29982> KERNEL32.dll!BaseProcessInitPostImport+0x8d
    ??:0

In this case Dr. Memory is able to report that the target address is inside a malloc area that has been freed and has not been re-allocated since.

This additional information, and the addresses accessed, only apply to the first error with that callstack that Dr. Memory sees. Any subsequent errors with the same callstack will increment the duplicate count for that error but further individual information about each duplicate is not provided.

Uninitialized Read

If the application reads from addressable memory that has not been written to since it was allocated, Dr. Memory reports an "uninitialized read" error. In order to avoid false positives, Dr. Memory does not report the use of uninitialized memory until something "meaningful" is done with that memory, such as a comparison or conditional branch or passing it to a system call. Variables or fields smaller than a word are often initialized without their containing word (variables and fields are typically word-aligned) being initialized. When these variables or fields are then copied, the uninitialized portion of each word is technically being read as an uninitialized value, but reporting such reads as errors would result in far too many errors.

When passing data structures to a system call, if the structure is initialized field-by-field then padding bytes may be left uninitialized. Dr. Memory will report errors on these as it does not know whether the kernel or a receipient on the other end might read it. To avoid these errors, memset the entire structure, or use a Dr. Memory error suppression (see Suppressing Errors) to ignore the error.

Here is an example of an uninitialized read error:

Error #2: UNINITIALIZED READ: reading 0xffbae108-0xffbae114 12 byte(s) within 0xffbae100-0xffbae114
Elapsed time = 0:00:00.214 in thread 19298
system call socketcall setsockopt args
    <system call>
0x08049a65 <my-socket-test+0x1a65> my-socket-test!main
    ??:0
0x0092dbb6 <libc.so.6+0x16bb6> libc.so.6<nosyms>!__libc_start_main
    ??:0
0x080489b1 <my-socket-test+0x9b1> my-socket-test!_start
    ??:0

When only part of a larger region is uninitialized, Dr. Memory reports the containing range to make it easier to track down the problem. This typically happens with buffers or structures passed to system calls. Note also in this example how Dr. Memory reports which part of the socketcall system call checks discovered this error.

Invalid Heap Argument

Whenever a pointer that does not refer to a valid malloc region is passed to free() or other malloc-related routines, Dr. Memory reports an "invalid heap argument" error. Here the problem is immediately apparent as 0x00001234 is not a valid heap address at all:

Error #4: INVALID HEAP ARGUMENT: free 0x00001234
Elapsed time = 0:00:00.263 in thread 19755
0x08048a71 <malloc+0xa71> malloc!main
    /work/dr/misc/internal/clients/drmemory/tests/malloc.c:161
0x0092dbb6 <libc.so.6+0x16bb6> libc.so.6<nosyms>!__libc_start_main
    ??:0
0x08048691 <malloc+0x691> malloc!_start
    ??:0

Memory Leaks

Dr. Memory divides all memory that is still allocated at the time it does its leak scan into 3 categories:

Memory that is still reachable by the application. This is NOT considered a leak. It is reported as "still-reachable allocation(s)" in Dr. Memory's summary. Many applications do not explicitly free memory whose lifetime matches the process lifetime and this is not considered an error by Dr. Memory.
Memory that is definitely not reachable by the application (at least, not by an aligned pointer to the start or middle of the allocated block). This is called a "leak" by Dr. Memory, as there is no way for the application to free this memory: it has lost any handle it had to the memory.
Memory that is reachable only via pointers to the middle of the allocation, rather than the head. This is called a "possible leak" by Dr. Memory. These may or may not be legimitate pointers to that allocation. There are several cases of known legitimate pointers that Dr. Memory recognizes using heuristics. None of these are listed as possible leaks unless the corresponding option is turned off:
- One class of legitimate mid-allocation pointers includes C++ arrays allocated via new[] whose elements have destructors. The new[] operators adds a header but returns to the caller the address past the header. You can suppress Dr. Memory's identification of such mid-allocation pointers, causing them to show up as possible leaks, with the -no_midchunk_new_ok runtime option.
- Another class, for some C++ compilers, includes instances of a pointer to a class with multiple inheritance that is cast to one of the parents: it can end up pointing to the subobject representation in the middle of the allocation. You can suppress Dr. Memory's identification of such mid-allocation pointers, causing them to show up as possible leaks, with the -no_midchunk_inheritance_ok runtime option.
- The std::string class stores a char[] array in the middle of an allocation but points to the start of the allocation. You can suppress Dr. Memory's identification of such mid-allocation pointers, causing them to show up as possible leaks, with the -no_midchunk_string_ok runtime option.
- A final example is a custom malloc wrapper that uses a header and passes a pointer offset into the result from malloc. The free wrapper subtracts from the passed-in pointer in order to compute the pointer to the head of the allocated block. You can suppress Dr. Memory's identification of such mid-allocation pointers, causing them to show up as possible leaks, with the -no_midchunk_size_ok runtime option.

The two categories of leaks ("leaks" and "possible leaks") are further broken down into direct and indirect leaks. An indirect leak is a heap object that is reachable by a pointer to its start address, but with all such pointers originating in leaked objects. Leaks can be thought of as trees, with the top-level object the direct leaks and all child objects indirect leaks. Dr. Memory reports the size of all the child indirect leaks for each direct leak, but does not report a detailed callstack for indirect leaks. If the direct leak is addressed, the indirect leaks should no longer be leaked, making their details unnecessary.

By default, Dr. Memory performs leak checking at application exit, or when a mid-run check is requested via -nudge (see Applications That Do Not Exit). Nudges can be used to help determine when the last pointer to an allocation was lost, if the callstack of the allocation is not sufficient to pinpoint the error in the source code. Each nudge will perform a full leak scan and by nudging periodically the first instance of a particular leak can be used to identify when the leak occurred.

On Windows, when HeapDestroy is called, any live allocations inside are reported as possible leaks. This can be disabled, since some applications may consider it correct behavior, with the -no_check_leaks_on_destroy option.

Dr. Memory reports the number of leaks, possible leaks, and still-reachable allocations. The callstack for the allocation of each leak and possible leak is gathered by default and listed with other errors in the results file. To also see all of the reachable allocations, use the -show_reachable runtime option. Reachable allocations will be in the global.pid.log file inside the log directory, labeled as REACHABLE LEAK.

Here are example lines from Dr. Memory's output summary:

:::Dr.Memory:::       9 unique,   282 total,   5860 byte(s) of leak(s)
:::Dr.Memory:::       0 unique,     0 total,      0 byte(s) of possible leak(s)
:::Dr.Memory:::     125 still-reachable allocation(s)
:::Dr.Memory:::          (re-run with "-show_reachable" for details)

There are known sources of false positives where a memory allocation may actually be reachable by the application, but Dr. Memory's algorithm will determine that it is unreachable. These cases include:

The only stored pointers to an allocation are not aligned to 4 bytes.
The only stored pointers to an allocation are encrypted.
There is some custom mechanism to reach these allocations that does not involve a direct stored pointer into any part of the allocated block.

One known problem comes from using glib's memory allocator. To avoid false positives when using glib, set the following environment variables when running your application:

G_SLICE=always-malloc
G_DEBUG=gc-friendly

Warning

Dr. Memory reports warnings about unusual memory-related conditions that an application developer might like to know about. For example:

Error #5: WARNING: heap allocation failed 
Elapsed time = 0:00:04.562 in thread 2668
0x00401366 <malloc.exe+0x1366> malloc.exe!main+0x356
    e:\derek\dr\linux\internal\clients\drmemory\tests\malloc.c:172+0x1e
0x0040219e <malloc.exe+0x219e> malloc.exe!__tmainCRTStartup+0x15f
    f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c:327+0x12
0x7d4e9982 <KERNEL32.dll+0x29982> KERNEL32.dll!BaseProcessInitPostImport+0x8d
    ??:0