The drpttracer DynamoRIO Extension provides clients with tracing functionality via Intel's PT instruction tracing feature.

Note: This extension only works on x86_64 Linux.

Setup

To use drpttracer with your client simply include this line in your client's CMakeLists.txt file:

use_DynamoRIO_extension(clientname drpttracer)

That will automatically set up the include path and library dependence.

Initialize and clean up drpttracer by calling drpttracer_init() and drpttracer_exit().

Usage

Intel PT (Processor Trace) only logs control flow changes. Therefore, when decoding a trace, the decoder needs to get raw bits of every instruction from images. drpttracer provides APIs to use Intel PT to generate trace data and some auxiliary data to help the decoder (e.g. libipt) to decode the trace. One type of auxiliary data is sideband data. This sideband data stores perf event records, which contain the image change messages necessary for decoding the trace. Another type of auxiliary data is the metadata for a trace. The metadata contains the CPU information and some other information that can be used to synthesize the time of PT trace and the perf event records. The auxiliary data enables the decoder to find the correct raw bits for every instruction.

drpttracer uses the pttracer handle to manage the tracing sessions. The handle is a pointer to drpttracer's internal data structure, which contains the tracing's PT data buffer, sideband data buffer, and metadata. To generate a pttracer handle, the client needs to call:

drpttracer_create_handle()

The create function lets the client specify the following parameters:

trace_mode : The tracing mode.
pt_size_shift: The size shift of PT trace's ring buffer. It must be greater than 0, and the buffer size is 2^pt_size_shift * PAGE_SIZE.
sideband_size_shift: The size shift of PT sideband data's ring buffer. It must be greater than 0, and the buffer size is 2^sideband_size_shift * PAGE_SIZE.

Note: Linux perf sets the buffer size to 4MiB by default. Therefore, it is best for clients to set trace and sideband buffers larger than 4Mib.; For one thread, only one tracing can execute at the same time. So the client needs to ensure one thread only owns one pttracer handle.

The client must ensure that the PT trace's ring buffer and the sideband data's ring buffer is large enough to hold the PT data. Insufficient buffer size will lead to lost data, which may cause issues in dynamorio::drmemtrace::pt2ir_t decoding.

After creating a pttracer handle successfully, the client can start or stop a tracing session by calling drpttracer_start_tracing() and drpttracer_stop_tracing().

The client must pass a valid trace handle to the start and stop function. And the client can use one handle for multiple tracing sessions. When the tracing is stopped, the stop function will allocate an instance of drpttracer_output_t, and copy the PT data, sideband data, and metadata to a drpttracer_output_t instance which is returned to the client. If drpttracer_stop_tracing() detects an overflow, it will return an error status code:

Return DRPTTRACER_ERROR_OVERWRITTEN_PT_TRACE if the PT trace is overwritten.
Return DRPTTRACER_ERROR_OVERWRITTEN_SIDEBAND_DATA if the sideband data is overwritten.

After all the tracing sessions are stopped, the client needs to call the following function to free the pttracer handle:

drpttracer_destroy_handle()

Additionally, to avoid memory leak, the client must call the following function to destroy the output instances after using them:

drpttracer_destroy_output()

Also, the client can dump the data in the output to different files for offline post-processing. The user can use the library dynamorio::drmemtrace::pt2ir_t in drcachesim to convert the PT trace to DynamoRIO's IR.

Tracing Mode

drpttracer provides three types of tracing modes:

When starting tracing, the client can choose the tracing mode by passing the appropriate flag to drpttracer_start_tracing().

Note: For kernel PT traces, the trace data's raw bits are all from kcore. So the output data from drpttracer doesn't contain sideband data; it only contains the trace data and metadata.

Unit Tests

We have some unit tests that verify the kernel tracing feature in drmemtrace, which also uses the drpttracer DynamoRIO extension. These tests are not built and run by default because they require superuser permission. They are also disabled automatically if the host system does not support the Intel-PT feature.

To run these tests, pass -DRUN_SUDO_TESTS=ON to cmake when building DynamoRIO. E.g.,

$ cmake -DRUN_SUDO_TESTS=ON -DBUILD_TESTS=ON <dynamorio-src-dir>
$ make -j
$ ctest -R 'drpttracer|drcacheoff.kernel'

On some systems, one may see errors like the following:

408: *** postcmd failed (1): drpt2ir: [28430, IP:ffffffffc11dd000] get next

408: instruction error: no memory mapped at this address

This is because our kcore logic copy may have missed copying some instructions from /proc/kcore. We rely on /proc/modules and /proc/kallsyms to point to relevant kernel code regions. Symbols for JIT code like eBPF are not included by default. The following workaround may help in cases where the missing memory region belongs to BPF JIT code. They make the BPF JIT code symbols visible in /proc/kallsyms.

$ sudo bash -c "echo 0 > /proc/sys/net/core/bpf_jit_harden"

$ sudo bash -c "echo 1 > /proc/sys/net/core/bpf_jit_kallsyms"

You may want to record the existing values in these configs so you can revert them after running the tests. See https://docs.kernel.org/admin-guide/sysctl/net.html#proc-sys-net-core-network-core-options for more details.