Assembly, the ABI & machine code

C++ cannot express "swap the stack pointer to an address in guest memory and jump". When the emulator needs that kind of control it writes assembly by hand. After mapping and NID resolution, read how C++ hands control to machine code. The examples cover guest entry, fiber context switching, generated thunks and instruction patches, preparing you for the instruction-patching lesson next.

Where assembly appears

These paths need explicit control of registers or instruction bytes:

x86-64 in ten lines

Everything below needs only a handful of registers. A fuller register tour is in Calling conventions & the ABI wall; the abbreviated list:

rax rbx rcx rdx        general purpose; rax also carries the return value
rsi rdi                argument registers (and the "S"/"D" asm constraints)
rbp rsp                frame pointer / stack pointer — the stack moves toward lower addresses
r8 r9 r10 r11 r12 r13 r14 r15
rip                    instruction pointer — you jump with call/jmp, never by writing it
rflags                 condition flags ("cc" in a clobber list)
gs                     segment base; on Windows it points at the per-thread TEB

Two habits to pick up from the tree's comments: call pushes the return address onto the stack, and rsp must stay 16-byte aligned at the point of a call — both rules come back in the JIT section below.

Inline asm — the grammar

GCC and Clang inline assembly is one statement with four parts:

asm volatile( "instructions, one string"
              : outputs      // "=r"(x)  — write result into x
              : inputs       // "r"(x)   — read x, any register;  "D"(x) pins rdi, "S"(x) pins rsi
              : clobbers);   // "cc" flags, "memory", and every register you destroyed

The rules that make it readable:

• %%rsp — a real register is written with a doubled %, because a single % starts an operand placeholder. Operands are %0, %1… by position, or %[guest_rsp] by the name given in the constraint.
• Clobbers are a promise to the compiler. If your asm writes rcx, the compiler must know — it is caller-saved, so it might be holding live data. Writing "memory" tells the compiler the asm can read or write any memory, which also stops it reordering your asm against other memory accesses (a compiler barrier).
• volatile prevents the compiler from treating the assembly as a disposable pure computation. It is not a general memory-ordering barrier: constraints and clobbers still have to describe the actual effects.

Now the real thing. This is the Windows branch of the guest-thread entry from runtimeLinker.cpp, abridged and annotated. The context: guest_rsp points into guest memory, and this block must run the game's entry point on that stack, then come back:

pushq %%r12 / %%r13 / %%r14 / %%r15   save the host callee-saved registers
movq  %%gs:0x08, %%r14                 read the Windows TEB stack limits...
movq  %%gs:0x10, %%r15
xorq  %%rcx, %%rcx
movq  %%rcx, %%gs:0x08                 ...and clear them, because stack probes during the
movq  %%rcx, %%gs:0x10                 guest run would otherwise check against host bounds
movq  %%rsp, %%r12                     park the host stack pointer
movq  %%rbp, %%r13                     and frame pointer
movq  %[guest_rsp], %%rsp              switch rsp to the guest stack
movq  %[guest_rbp], %%rbp              and rbp to the guest frame
callq *%[func]                         jump into the game (arguments already in rdi/rsi)
movq  %%r13, %%rbp                     restore host rbp
movq  %%r12, %%rsp                     restore host rsp
movq  %%r14, %%gs:0x08                 put the TEB stack limits back
movq  %%r15, %%gs:0x10
popq  %%r15 / %%r14 / %%r13 / %%r12   restore the callee-saved registers

Every line maps to a constraint on the right side of the statement. The inputs [guest_rsp] "r"(guest_rsp_reg) and [guest_rbp] "r"(guest_rbp_reg) feed the named %[...] operands; "D"(params) forces rdi and "S"(atexit_func) forces rsi, because that is where the System V caller placed them. The clobber list names every other register the block touches plus "cc" and "memory".

Explicit register variables

You can pin a C++ variable to a specific register, so the asm block can rely on it being there. Clang and GCC spell it as a register declaration with an asm name:

register entry_func_t func_reg asm("rbx")      = func;
register uintptr_t    guest_rsp_reg asm("r14") = guest_rsp;
register uintptr_t    guest_rbp_reg asm("r15") = guest_rbp;

Why bother? The comment right above it explains the exact bug this prevents: on macOS, Clang can allocate a plain "r" input to r12/r13 — registers the asm block itself clobbers before consuming the value. Pinning the inputs to rbx/r14/r15 (registers the System V guest preserves) makes the hand-off safe.

Control-flow attributes and generated thunks

These attributes describe control flow to the compiler:

AllocateUnresolvedImportThunk() emits a 34-byte diagnostic thunk in runtimeLinker.cpp. It passes a record ID to a System V helper, restores stack alignment, zeros the floating return register and returns. The helper logs and returns integer zero; that is an unresolved-import fallback, not an implementation of the missing API.

sub    rsp, 8
movabs rdi, <record_id>             patched at byte offset 6
movabs rax, <UnresolvedImportStub>  patched at byte offset 16
call   rax
add    rsp, 8
xorps  xmm0, xmm0
ret

And the fiber save, which is the same idea as a setjmp: capture every callee-saved register and both stack pointers into a struct, then return 0 — later FiberRestoreContext loads them back and jmps to the saved return address, making FiberSaveContext "return" a second time with a different value:

movq %[ctx], %%r10       struct pointer into a scratch register
movq %%rbx, 0(%%r10)     store every callee-saved register at a fixed offset
movq %%rbp, 8(%%r10)
movq %%r12, 32(%%r10)
...
leaq 8(%%rsp), %%r11     store rsp as it would be after this call returns
movq (%%rsp), %%r11      and the return address itself
xorl %%eax, %%eax        return 0

The ABI attribute is the C++ glue

The asm blocks above exist because guest code follows the System V ABI and the host build may follow Microsoft x64. The C++ side of that wall is two macros in src/common/abi.h — an attribute applied per function:

#define KYTY_MS_ABI   __attribute__((ms_abi))
#define KYTY_SYSV_ABI __attribute__((sysv_abi))

Any function the guest can call carries KYTY_SYSV_ABI, so the compiler emits a System V prologue for that one function. The attribute even travels on function pointers, which is how the JIT tells the compiler what convention a generated call must use:

using func_t = KYTY_MS_ABI uint8_t* (*)();
// "a pointer to a function, called with the Microsoft convention, returning uint8_t*"

And a pointer can be manufactured from an address with reinterpret_cast, the only sanctioned way to turn a number into something callable:

auto* func = reinterpret_cast<entry_func_t>(addr);
func(params, atexit_func);

System V variadic calls carry the vector-register argument count in al. A call-forwarding bridge must honor that ABI; current unresolved-import stubs instead log and return zero. guestPrintf.cpp interprets the guest argument list because it cannot be forwarded directly to Microsoft-convention vprintf.

Machine code as data — jit.h

The strangest and most instructive file to read is src/loader/jit.h. Its structs are not described by machine code — their member uint8_t code[N] arrays are the machine code. The emulator writes bytes into these buffers at runtime and then calls them.

Start with Call9, the nine-byte call replacement used by TLS patching:

struct Call9 {
    template <class Handler>
    void SetFunc(Handler func) {
        auto func_addr = reinterpret_cast<int64_t>(reinterpret_cast<void*>(func));
        auto rip_addr  = reinterpret_cast<int64_t>(&code[6]);
        auto offset32 = static_cast<uint32_t>(static_cast<uint64_t>(func_addr - rip_addr) & 0xffffffffu);
        *reinterpret_cast<uint32_t*>(&code[2]) = offset32;
    }
    uint8_t code[9] = {0x48, 0xE8, 0, 0, 0, 0, 0x48, 0x89, 0xC0};
};

• The bytes are instructions: 48 E8 rel32 calls the handler; 48 89 C0 initially copies RAX to itself. SetOutputReg() changes that destination register.
• The four-byte displacement starts at offset 2. It is the target minus the next instruction address, &code[6]. The allocator must keep the target within signed 32-bit reach.
• The REX.W prefix matters on AMD: it prevents a retained 0x66 prefix from turning the near call into callw.

Placement new is still used, but the old custom PLT classes are gone. In RuntimeLinker::SetupTlsHandler(), it constructs the TLS bridge in an existing allocation:

auto* code = new (reinterpret_cast<void*>(program->tls.handler_vaddr)) Jit::SafeCall;
code->SetFunc(TlsMainGetAddr);

new (address) Type runs a constructor on memory that already exists; it does not allocate that memory. The function also installs per-register TlsRegStub objects, protects the allocation as executable, and flushes the instruction cache.

The most elaborate one is SafeCall, a whole function hand-written in bytes, and it shows the two ABI rules from earlier being honoured by hand. Windows requires 32 bytes of shadow space and 16-byte stack alignment at a call:

sub    rsp, 0x80            make room to save every caller-saved register
pushfq                      preserve the flags
push   rcx / rdx / r8 / r9 / r10 / r11 / rbx / rdi / rsi
mov    rbx, rsp
and    rsp, 0xfffffffffffffff0    align the stack to 16 bytes
sub    rsp, 0x20                  reserve Win64 shadow space
movabs rcx, <handler>             the handler pointer is patched in at offset 0x22
call   rcx
mov    rsp, rbx                   restore, pop everything back, ret

Everything you need to read the tree is in that file: #pragma pack(1), reinterpret_cast writes, relative-offset arithmetic and function pointers carrying their own ABI. Read it top to bottom once and the rest of the loader stops being opaque.

The red zone — a guest promise that breaks on Windows

System V grants a leaf function 128 bytes below rsp — the red zone — to use freely without adjusting the stack pointer. Guest code was compiled assuming that promise. Windows never made it: a kernel-delivered exception or signal handler will use that exact space, silently overwriting the guest's data.

Kyty's answer is src/loader/guestInstructionPatcher.cpp, which reads .eh_frame to find function boundaries, identifies the functions that touch the red zone, and rewrites their memory instructions to route through a trampoline instead. The same patcher handles selected AMD instruction compatibility. Its result separates patched memory sites from per-family instruction outcomes; this excerpt omits unrelated counters:

struct GuestInstructionPatchResult {
    uint64_t function_count          = 0;
    uint64_t red_zone_function_count = 0;
    uint64_t patched_memory_instruction_count = 0;
    ...
    uint64_t unrelocatable_memory_instruction_count = 0;
    InstructionPatchCounts reciprocal_sqrt, extrq, insertq, rdpid, clwb;
};