Testing

Automated Test Machines

Continuous Integration (CI) via Github Actions

We have Github Actions set up to run our short test suite (the same as the pre-commit suite) for every push to the master branch and for every pull request, for 32-bit and 64-bit x86 on Linux and Windows, for 64-bit Mac, and for AArch64 on Linux. ARM (32-bit) and Android builds are performed but no tests are run at this point. The Github Actions CI can also be triggered manually, for master or any other branch, on the DynamoRIO Github Actions page.

An email is sent whenever there is a (non-flaky-test) failure. If your commit causes a failure on a merge to master, please forward this email to the dynamorio-devs mailing list explaining the failure and whether you will be reverting your commit (which you should do unless you have a trivial fix ready immediately). If any tests become flaky, please contribute to addressing them to keep our continuous integration testing clean and green.

The Github Actions configuration is specified in the workflow files in .github/workflows.

We have a default test suite verbosity of ctest -VV in suite/runsuite_wrapper.pl (ctest --output-on-failure -VV -S) which shows build warnings and failing test output. The Actions integrated output viewer is slow; it is often better to use the right-hand three-dot menu to select View raw logs or Download log archive.

CI Job Auto-Cancellation

We have configured the CI such that if a new commit is pushed to a pull request before the prior CI jobs have finished running, the prior jobs will be cancelled (with an email sent for each cancellation). This is to conserve CI resources under the assumption that the old results are no longer needed.

CI for AArch64 & AArch32

We use Jenkins for pre- and postcommit testing on AArch64. Jenkins is set up to run our short test suite (the same as the pre-commit suite) for every push to the master branch and for every pull request. We also run a subset of the test suite under QEMU emulation on both AArch32 and AArch64, but not all tests are able to run that way.

CI Tree Closures

If the CI on master turns red, the tree is "closed": meaning that there should be no commits to the repository unrelated to fixing the problem until it is green again.

Trybots

Our CI setups provide "trybot" functionality for nearly every platform via pull request testing. To test a change on platforms you don't have locally, or test whether your change will pass the CI, you can create a pull request. The CI will then automatically run the test suite on your change.

Debugging Tests on Github Actions Runner

Test failures that happen only on Github Actions and are not reproducible locally can be hard to debug. Fortunately, there's a way to SSH into a Github Actions runner to debug the test. This can be done using tmate: https://github.com/marketplace/actions/debugging-with-tmate.

Using tmate requires sudo on the Actions runners which is only available with pull requests on branches within the repository; it will not work with pull requests created from external forks of the repository.

First, identify the Actions workflow file which contains the job with the failure. From the Actions run page with the failing test (usually reached from the links in the job runs for a pull request), click on "Workflow file" in the bottom of the left sidebar. It will present the file contents with its path at the top. It will be something like ".github/workflows/ci-windows.yml". That is the path within the git repository.

Next, go and edit that file in your branch. Delete all jobs except the failing one (just remove those lines from the file). For the failing one, add these 3 lines as a new step right after the "Run Suite" step but before the "Send failure email step". Be sure to match the surrounding indentation as indentation matters for .yml files.

- name: Setup tmate session
  if: failure()
  uses: mxschmitt/action-tmate@v3

Next, delete all the other workflow files in the ".github/workflows/" directory. This will save you time and save resources in general, to only run the single target job.

You can see an example of these workflow file deletions and edits in this commit: https://github.com/DynamoRIO/dynamorio/pull/6414/commits/08b96200cdb9fd4d39a4c89e2aa9eafed92027f4

If you want to focus on just one test, you can use a label like the TMATE_DEBUG label in the linked commit to run only that one test, but that is not necessary. Pasting the key lines for that here:

In runsuite.cmake after the arg parsing:

set(extra_ctest_args INCLUDE_LABEL TMATE_DEBUG) # TEMPORARY

At the bottom of suite/tests/CMakeLists.txt, add the label to the target test:

set_tests_properties(code_api|tool.drcacheoff.burst_traceopts PROPERTIES
    LABELS TMATE_DEBUG) # TEMPORARY

Commit these changes with a title that starts with "DO NOT COMMIT" so it’s clear these are temporary debugging changes, and send to Github with git review.

Now go to your pull request page and click on the details for the target workflow. Wait for it to reach the "Setup tmate session" step (this may take from a few minutes to 15-20 minutes depending on the job; you can look at prior instances of jobs to see how long they typically take). It will print a command like " ssh JhJp879nThXEKUAPWEGuatJ3J@sfo2.tmate.io". Run that command and you will have an interactive shell in the base build directory.

By default, it may start the ssh in tmux review mode. You would need to quit out of that to get to the terminal. The shell should last until the action times out which is after 6 hours, or until you terminate the initial connection.

The connection is through tmux, so you can create new panes and shells using tmux commands. You can install gdb if needed.

tmate also allows web shell access; note that you may need to press q one time if the web page doesn't show anything.

Regression Test Suite

The black box test suite for DynamoRIO resides in the suite/ directory. The tests are applications that we execute under control of DynamoRIO to ensure we're covering corner cases.

Test Organization

The tests in suite/tests/ are divided into different directories according to type of test:

• client-interface/ = tests of the regular client API
• api/ = tests using the IR to test decoding and encoding, and a test of the start/stop interface
• common/ = cross-platform basic test applications
• linux/ = Linux-specific tests
• pthreads/ = more Linux tests, using pthread
• win32/ = Windows-specific tests
• runall/ = Windows-specific tests using AppInit or follow-children injection

There are also tests that are less directly focused on the API of this open-source project, but that are still useful as they contain some crazy behavior that is good to test:

• security-common/ = tests of security policies
• security-linux/ = Linux-specific tests of security policies
• security-win32/ = Windows-specific tests of security policies

Building and Running Tests

CMake is used to build the tests. They are not built by default. To enable them, turn on the BUILD_TESTS CMake variable. For example, from a build directory next to the dynamorio source tree:

cmake -DBUILD_TESTS=ON ../dynamorio

CTest is used to run the tests with various runtime options. Each test is named to indicate the runtime options and the executable. The options are first, separate by commas, with a pipe delimiting the end of the options. The executable has a dot instead of a slash. Here are some examples:

code_api|security-common.decode-bad-stack
code_api,disable_traces|client.events

Note that CTest versions prior to 2.8 truncate the test names to 30 characters, so the second example above may show up as:

code_api,disable_traces|client

From a build directory that was built with BUILD_TESTS you can run the set of tests for that build with the test Makefile target. You need to build first with a plain make command as the test target does not check for that.

make -j6
make test

You can also invoke ctest directly, which gives greater control over which subsets of the tests are run. Use ctest -N to see the list of tests and which number is assigned to each. Then you can use ctest -I x,y to run all tests from number x to number y, and the -V parameter to display the command line and the output of the test. For example:

ctest -V -I 49,49

You can also specify tests using inclusion and exclusion regular expressions. For example:

ctest -R 'strace|signal'

will run all tests with the string "strace" or the string "signal" in their names, while

ctest -E security-common

will run all tests except those with security-common in their names.

If you are using CTest version 2.8 or later, you can run tests in parallel by passing -jN to ctest on the command line:

ctest -j5

As an alternative to using -V to display the test command line and output during executing, CTest stores that information (even when not run with -V) in the Testing/Temporary/LastTest.log file in the build directory.

Testing AArchXX

Currently we have access to the following machines for contributors to do testing on actual AArchXX hardware (please contact @AssadHashmi if you want access):

• one AArch64 ThunderX1, hosted on packet.net

We also have support for running tests under QEMU emulation. This is automatically set up if qemu-user is installed:

$ sudo apt-get install qemu-user qemu-user-binfmt

For instructions on how to manually run under QEMU, see Running Under QEMU.

Test Output

Each test produces output to stderr and/or stdout that is diffed against an expected output. There are three primary ways we calculate the expected output. The first is from a literal file with an .expect extension: e.g., suite/tests/common/segfault.expect.

The second is from a .template file that is evaluated with respect to the current build and runtime option parameters to produce an .expect file for literal diffing: e.g., suite/tests/common/decode.template. The C preprocessor is used to convert the .template file to a literal string for matching. Runtime options are converted to compiler definitions and added to those set for the build when running the preprocessor. This allows the test output to be conditional on the build defines and runtime parameters.

The third is a .templatex file, which is like a .template file but is also evaluated as a regular expression instead of a literal string after passing through the pre-processor.

All three of those options use the CTest property PASS_REGULAR_EXPRESSION (with regular expression characters escaped when not using .templatex). That property has a size limit. There are other methods of comparing output. One is to use programmatic comparisons inside the test. Another is to use our script runcmp.cmake which has no size limit.

Pre-Commit Test Suite

The suite/runsuite.cmake script is used to execute a series of builds and test runs. Running make test in a single build directory is not sufficient to test all of the configurations that we support. The test suite requires a development environment that is capable of compiling for both 64-bit and 32-bit (see [How To Build](How-To-Build) for instructions on setting up Ubuntu appropriately).

We have two different suites: the short suite, which is required to be run prior to committing code changes (see CommitCriteria), and the long suite, which is meant to be run as a nightly test suite on different platforms (see Nightly Suite). The suite/tests/CMakeLists.txt file controls which runs are executed for each build, while suite/runsuite.cmake lists the builds. The long suite can be executed by using the suite/runsuite_long.cmake wrapper script.

On Windows, runsuite.cmake assumes that the environment variables are set up for building, just like when configuring a single build directory. The script will change from 32-bit to 64-bit and vice versa when necessary; to do so, it assumes the typical Visual Studio layout of 64-bit subdirectories.

The runsuite.cmake script is meant to be executed from an empty directory. It creates a subdirectory for each build in the suite. Use ctest -S to execute the script. Here is an example:

mkdir ../build_suite
cd ../build_suite
ctest -S ../dynamorio/suite/runsuite.cmake

We recommend using Ninja, in which case pass the use_ninja parameter:

mkdir ../build_suite
cd ../build_suite
ctest -S ../dynamorio/suite/runsuite.cmake,use_ninja

You can pass -V (or -VV for more output) to ctest to see the results as the test suite runs. Note that it is normal to have ctest output strings such as "No tests were found!!!" as we have builds for which we run no tests (particularly in the short suite). It is also expected to have an error at the end of the suite (if run with -V):

CMake Error: Some required settings in the configuration file were missing:
CTEST_SOURCE_DIRECTORY = E:/derek/dr/suite/..
CTEST_BINARY_DIRECTORY = (Null)
CTEST_COMMAND = c:/PROGRA~2/CMAKE2~1.6/bin/ctest.exe

This warning is an artifact of how CTest assumes we've set things up and can be ignored.

At the end of the suite a file called results.txt will be created and displayed. It shows configure failures, build failures, and test failures. It also looks for DynamoRIO crashes, asserts, and curiosities and displays those.

For details on configure, build, or test failures, look in the Testing/Temporary/Last* log files inside the appropriate build subdirectory of the top-level suite directory. For example:

cd ~/dr/build_suite
less build_debug-internal-32/Testing/Temporary/LastTest_20150708-1437.log

Note that the build directories are quite large. For a short suite on Linux they occupy about 600MB altogether.

Cross-Compilation and Android Testing

The test suite includes cross-compilation tests to ensure that the ARM and Android builds are not broken. If a cross-compiler is not found on the PATH, these builds will fail, but they are considered optional, so the whole suite will not be considered a failure. However, we recommend installing the cross-compilers and placing them on your PATH for more thorough testing.

If you have an Android device set up for access through the adb shell utility, the Android build is capable of automatically copying binaries to the device and running tests. If both the Android cross-compiler and adb are on your PATH, and adb status indicates an attached device, the tests will be run.

Pre-Commit Test Suite Over Ssh

The runsuite_ssh.cmake script can be used to launch the pre-commit test suite on a remote machine over ssh. Remote Windows machines are supported when using cygwin sshd on the remote machine. Both rsync and ssh must be installed on the local and remote machines, and RSA-key passwordless authentication is recommended.

cd /my/checkout && cmake -DHOST=dr-ubuntu -P suite/runsuite_ssh.cmake

For a remote Windows machine, cygpath must be on the path, and the REMOTE_WINDOWS variable must be set to properly convert paths:

cmake -DHOST=mylaptop -DUSER=derek -DREMOTE_WINDOWS=ON -P /my/checkout/suite/runsuite_ssh.cmake

The comments at the top of runsuite_ssh.cmake describe additional options.

Test Failures

Unfortunately our test suite is not as clean as it could be. Some tests can be flaky and while they pass on the machines of the existing developers and on our automated test machines, they may fail occasionally on a new machine. Please search the issue tracker before filing a new issue to see if a test failure has already been seen once before. We welcome contributions to fix flaky tests.

Flaky tests are marked in one of two ways:

• Append "_FLAKY" to the test's name
  • See suite/tests/CMakeLists.txt for examples
• Mention the test in runsuite_wrapper.pl
  • See suite/runsuite_wrapper.pl for examples

In both cases make sure an issue is filed to fix the test and mention the issue at the place the test is marked as flaky.

The latter is preferred for tests that should be fixed first.

Missing Tests

Some features that were tested in our pre-cmake infrastructure have not been ported to cmake. We welcome contributions in this area:

• Issue 120: port runall test infrastructure to cmake. This is needed to properly run all tests that use notepad or calc on Windows (these tests are currently disabled so they do not show up as failures):
  • client-interface/cbr2
  • client-interface/cbr
  • client-interface/custom_traces
  • client-interface/decode-bb
  • client-interface/nudge_test
  • client-interface/pc-check
  • all runall/ tests
• Issue 16: tests not building or updated properly for X64. Such tests include (these tests are currently disabled so they do not show up as failures):
• Issue 125: re-enable tests: linux.vfork, linux.vfork-fib, win32.debugger, security-common.selfmod-big
• Issue 1670: port rest of test suite to ARM

Any effort put into cleaning up the test failures and moving toward zero expected failures is greatly appreciated.

Adding New Tests

In order to add any tests to the regression suite, first choose the sub directory in which you are going to add your tests (see Test Organization). Pick a test from that directory based on which you are going to model yours and study it well, especially the .template file associated with that test case, because that determines what the expected output will be for this particular test in different runs of the regression suite where the options will be different.

Make sure that your test case has the statement INIT(); in it; this is necessary to catch unhandled exceptions on Windows. Lack of adding this line can cause the regression suite to break. Also, use print() rather than printf() because the former flushes the output (a problem on cygwin). Include tools.h in your test.

Once the basic test has been set up and tested, following the process for checking in code at CommitCriteria to add your test.