This page contains a record of some design decisions for the port to AArch64. The AArch64 master issue, #1569, has a list of commits and some more up-to-date information on the status of this port.

Introduction to AArch64

AArch64 is the ARM architecture's 64-bit execution state, which was introduced in version 8 of the architecture, ARMv8, announced in 2011. There have been subsequent updates to the architecture: ARMv8.2 was announced in 2016.

ARM defines three architecture "profiles" (A, R and M), representing architecture configurations and subsets appropriate to different market segments. For DynamoRIO we are only concerned with the "application profile", ARMv8-A, which includes virtual memory.

ARMv8 also defines the 32-bit execution state, AArch32, which uses the A32 ("ARM") and T32 ("Thumb") instruction sets familiar from previous versions of the ARM architecture. It is only possible to switch between AArch32 and AArch64 on an exception. A system that runs AArch64 software may or may not also be able to run AArch32 software. Although there are many similarities between AArch32 and AArch64 there are also some fundamental differences, so for many purposes it is helpful to think of AArch32 and AArch64 as separate architectures and this is the approach taken by DynamoRIO with the preprocessor macros AARCH64, ARM, X86, and subdirectories in the source code with the same names in lower case. However, there is also a preprocessor macro AARCHXX, and a corresponding subdirectory, to facilitate sharing of code between AArch32 and AArch64 where this is convenient.

Note that in DynamoRIO's source code, as in many other places, "ARM" is used to mean AArch32.

Linux uses the name "arm64" for its AArch64 architecture (which includes an ABI and other things not specified by the ARM Architecture). GCC and other tool chains use "aarch64" (lower case). So there is a Debian package called "gcc-aarch64-linux-gnu", which is the "GNU C compiler for the arm64 architecture".

The AArch64 user-mode execution state consists of:

X0-X30: 31 64-bit general-purpose registers. X30 is used as the procedure link register.
A 64-bit program counter (PC) and stack pointer (SP). Unlike in AArch32, these are distinct from the numbered registers.
V0-V31: 32 128-bit registers for floating-point and SIMD.
NZCV: Condition Flags (the top bits of a 32-bit register).
FPCR: Floating-Point Control Register (32 bits, some unused).
FPSR: Floating-Point Status Register (32 bits, some unused).
Under Linux, the 64-bit system register TPIDR_EL0 that is readable and writable in user mode and used for thread-local storage (TLS).

The ARM architecture is bi-endian: the operating system can switch between little-endian and big-endian handling of data, with little-endian as the default. The Linux arm64 kernel can be configured as big-endian but all major Linux arm64 distributions are little-endian.

IR decisions

AArch64 has 31, not 32, general-purpose registers. Depending on the context, the value 31 in an encoding may refer either to the stack pointer or, more often, to the "zero register", which is read as zero and unaffected by a write (it is a pseudo-register). DynamoRIO's internal representation (IR) distinguishes between XSP and XZR. In the enum, DR_REG_XSP follows DR_REG_X30 and is included in the range DR_REG_START_GPR to DR_REG_STOP_GPR even though XSP is not usually interchangeable with other X registers. DR_REG_XZR is not included in the "GPR" range.

The IR distinguishes between the "X" registers and the "W" registers, which are aliases for the lower 32 bits of an X register. Writing to a W register sets the top half of the corresponding X register to zero. Similarly, there are aliases for the lowest part of an FP/SIMD register: DR_REG_B0 (8 bits), DR_REG_H0 (16 bits), DR_REG_S0 (32 bits), DR_REG_D0 (64 bits), and DR_REG_Q0 (all 128 bits). (This is a noteworthy difference from AArch32: in AArch32, S3 is the highest word of D1 and of Q0; in AArch64, S3 is the lowest word of D3 and of Q3.)

There are the expected differences between DynamoRIO's IR and the standard assembly language. In particular, DynamoRIO lists source and destination registers separately. A register operand that is both read and written must appear in both lists, as must a register whose contents is only partly overwritten by an instruction. An example is MOVK, which overwrites part of a general-purpose register with a constant value.

Descriptions of the ARM architecture distinguish between "instructions" and "aliases". For example CMP X1, X2 is an alias for SUBS XZR, X1, X2: a flag-setting subtract that discards the result by specifying the zero register as the destination. A typical assembler accepts both of these forms, generating the same instruction, typically disassembled as CMP. However, DynamoRIO's AArch64 IR ignores aliases, so there is no OP_cmp. However, for convenience there are (or should be) macros in aarch64/instr_create_api.h corresponding to the standard aliases.

There is no DR_REG_PC for AArch64. Literal loads and instructions that generate PC-relative address are represented as in X86_64, using REL_ADDR_kind, not as in ARM/AArch32.

TBD: NZCV, FPCR, FPSR, SIMD instructions.

Encoder/decoder

AArch64 has a single instruction set, called "A64", in which all instructions have 32 bits. The encoding is relatively simple and consistent, which makes it possible in some cases to deduce properties of an instruction without fully decoding it. For example, a general-purpose register operand is encoded in one of four positions in the instruction word so it it may be possible to know that an instruction does not read or write a given register even without knowing anything else about the instruction. Similarly, it is possible to recognise a potential load/store instruction by examining just a few bits.

Encodings are described in "codec.txt", which is processed by "codec.py" to generate several C source files. In order to avoid adding Python as a build requirement these generated files are included in the source. A developer who modifies "codec.txt" should run "codec.py" manually.

Adding a new instruction to "codec.txt" will often require adding a new operand type, for which encoder and decoder functions must be added in "codec.c".

Currently the instruction bit patterns listed in "codec.txt" are not allowed to overlap. A possible extension would be to allow a more specific pattern (one with fewer 'x' bits) to override a less specific pattern. This would allow NOP, YIELD, WFE, WFI, SEV and SEVL to be defined as special cases of HINT, but there are other ways of handling HINT so this single case is not a strong argument for extending the notation. Also, there may be other ways of extending the notation that are inconsistent with the approach just described.

At the end of 2016, DynamoRIO's encoder/decoder handles all the load/store instructions, including load/store of FP/SIMD registers, and all the instructions that do not operate on FP/SIMD registers, up to ARMv8.2.

Because the decoder is incomplete, unrecognised instructions are decoded as instances of a generic instruction, OP_xx, which is regarded as reading and writing the general-purpose registers referenced in the four places in the instruction word where the number of a general-purpose register might appear. This ensures that undecoded FP/SIMD instructions are correctly (though perhaps inefficiently) handled when they might read or write the "stolen" register.

Stolen register

DynamoRIO uses a "stolen" register on AArch64 for the same reason as on AArch32: it is not possible to use TPIDR_EL0/TPIDRURO directly as an address for accessing memory. The stolen register may be specified on the command line at run time; by default it is X28.

If the fragment cache were not shared between threads it would be possible to avoid stealing a general-purpose register: borrow TPIDR_EL0 instead and spill registers, when necessary, by first spilling a general-purpose register into TPIDR_EL0 and then generating a memory address with ADRP. This way one could avoid the expense of mangling instructions that use a stolen general-purpose register, but instrumentation would be more expensive in some cases, so the value for DynamoRIO of this approach is unclear.

Reachability

An AArch64 unconditional immediate/direct branch (B or BL) has a range of +/- 128 MiB. If the fragment cache were restricted to a 128 MiB block of memory then it would be possible to branch from any fragment to any other fragment. DynamoRIO does not currently restrict the memory range used for the fragment cache so in general it is necessary to use a register/indirect branch when exiting from a fragment. There are opportunities for improvement in this area.

Self-modifying code

The X86 architecture requires hardware to detect when the instruction cache has been invalided by a write to memory, so DynamoRIO must detect when code that has already been rewritten into the fragment cache is subsequently modified, which is not trivial to implement efficiently.

The ARM architecture requires software to perform explicit synchronisation between writing instructions to memory and executing those instructions. In AArch32 this cannot be done in user mode, so 32-bit ARM Linux uses a system call (SYS_cacheflush), which DynamoRIO can easily intercept.

In AArch64 there are user-mode instructions for synchronising the instruction cache, so DynamoRIO must mangle these instructions so as to detect when a program may have legally modified itself.

The prescribed recipe for synchronising the instruction cache is implemented by clear_icache() in "drlibc.c". There are three scenarios we must consider:

Hardware without data to instruction cache coherence (\c CTR_EL0.DIC=0)

Software running on hardware without data to instruction cache coherence must perform explicit cache maintenance to invalidate icache lines operating memory that has been modified. DynamoRIO detects when an app has performed these operations by mangling the IC instructions to return to the C runtime with X0 set to linkstub_selfmod.

Hardware with data to instruction cache coherence (\c CTR_EL0.DIC=1)

Software does not have to perform explicit icache maintenance when modifying code. Some synchronisation is still required but the procedure is reduced to:

dsb ish

isb

Even though we could detect this sequence it does not give any indication of the memory range which has been modified so it can't be mangled in the same way we do for IC in the CTR_EL0.DIC=0 case. Instead we need to use sandboxing similar to the x86 port (see below).

Concurrent Modification and Execution of instructions (CMODX)

For a narrow class of instructions the ARM architecture allows one thread to modify instructions and guarantees that a second thread executing these instructions will either execute the original instruction, or the replacement and this does not require any explicit synchronisation. The only way to detect this is using sandboxing.

Sandboxing implementation

The AArch64 sandboxing implementation works in a similar way to the x86 version: Fragments marked FRAG_SELFMOD_SANDBOXED have a copy of the original app code appended to the fragment and extra mangling code is emitted to the beginning of the fragment which compares the app memory with the copy to detect changes since the last time this fragment was executed. If a difference is detected the fragment takes a special exit and the fragment is rebuilt from the new modified code. This check currently always compares all instructions in the fragment. As a future possible optimization this could be changed to just check CMODX-safe instructions in the CTR_EL0.DIC=0 case.

If -sandbox_writes is on, any store instructions in a sandboxed fragment will be mangled to check whether the destination range of the store overlaps with the app code for the current fragment. If it does, again it will exit and rebuild the fragment from the new modified code. Unlike x86, -sandbox_writes is off by default on AArch64. Even when running on hardware with icache coherence, software must execute a DSB followed by ISB to guarantee modified instructions are executed. We can take advantage of this and skip sandbox write mangling and instead just terminate any sandboxed block after an ISB instruction. Execution will continue to the next fragment which will begin with the top-of-fragment modification check. -sandbox_writes should not be required for typical AArch64 applications and is provided for experimentation and possible future use.