Step 19 · Loading and execution

Patching guest instructions

Guest code assumes a working fs segment and a FreeBSD-shaped world. On Windows those assumptions are false, and since the code runs natively there is no interpreter to fix them up on the fly. So the loader edits the instructions themselves.

Before this page: Why there's no CPU emulator · Calling conventions & the ABI wall · Assembly, the ABI & machine code. View the learning route.

Read

Two instruction patterns in PS5 code cannot execute on Windows. Since there is no recompiler to fix them during translation, the loader rewrites the bytes in place before the code is ever run.

Problem 1 — thread-local storage through fs

PS5 code reads its thread control block with a single instruction:

64 48 8B 04 25 00 00 00 00     mov rax, qword ptr fs:[0]

On x86-64 Windows the host TEB is addressed through gs. The guest’s fs:[0] assumption does not provide its TCB on the host, so the loader redirects the recognized access to a guest TLS helper.

The fix replaces those exact nine bytes with a call to a generated stub, then moves the result into whatever register the original targeted:

src/loader/runtimeLinker.cpp — PatchProgram()
const uint8_t tls_pattern[5]       = {0x64, 0x48, 0x8B, 0x00, 0x25};
const uint8_t zero_displacement[4] = {};

for (auto* ptr = start_ptr; ptr <= end_ptr; ptr++) {
    auto*  inst_ptr     = ptr;
    size_t prefix_count = 0;
    // compilers pad this sequence with up to three 0x66 prefixes
    while (prefix_count < 3 && inst_ptr < start_ptr + size && *inst_ptr == 0x66) {
        inst_ptr++; prefix_count++;
    }

    const uint8_t modrm = inst_ptr[3];
    if (memcmp(inst_ptr, tls_pattern, 3) == 0 && (modrm & 0xc7u) == 0x04u &&
        inst_ptr[4] == tls_pattern[4] &&
        memcmp(inst_ptr + 5, zero_displacement, sizeof(zero_displacement)) == 0) {

        const auto reg = (modrm >> 3u) & 7u;      // which register was the destination
        EXIT_NOT_IMPLEMENTED(reg == 4u);          // rsp would be nonsense

        // the stub is written at inst_ptr, AFTER any 0x66 prefixes — they are left in place
        auto* code = new (inst_ptr) Jit::Call9;
        code->SetFunc(reg == 0 ? program->tls.handler_vaddr
                               : program->tls.handler_vaddr + Jit::TlsRegStub::GetOffset(reg));
        ptr += prefix_count + Jit::Call9::GetSize() - 1;
    }
}

Jit::Call9 is exactly nine bytes, so the replacement is size-neutral and no addresses shift:

src/loader/jit.h — struct Call9
struct Call9 {
    void SetFunc(Handler func) { // rel32 written at code[2], relative to &code[6] }
    void SetOutputReg(uint8_t reg) { code[8] = 0xc0u | (reg & 7u); }
    static uint64_t GetSize() { return 9; }

    // rex.w call func   48 E8 xx xx xx xx
    // mov  rax,rax      48 89 C0
    uint8_t code[9] = {0x48, 0xE8, 0, 0, 0, 0, 0x48, 0x89, 0xC0};
};

The mov rax,rax byte is rewritten by SetOutputReg so the value lands in the register the original instruction wanted. For registers other than rax the stub jumps to a per-register variant at a fixed offset (TlsRegStub::GetOffset(reg)), each of which does call handler; mov <reg>, rax.

Why the call starts with a REX.W byte

The replacement uses 48 E8 …: a REX.W prefix followed by the relative call. The patch can retain compiler padding prefixes such as 0x66. AMD processors can interpret a retained operand-size prefix as a 16-bit near call; the REX.W byte ensures the intended 64-bit call behavior. Read Jit::Call9 alongside the patch matcher so the replacement width and retained prefixes stay consistent.

What the handler does

src/loader/runtimeLinker.cpp — TlsMainGetAddr()
static KYTY_MS_ABI uint8_t* TlsMainGetAddr() {
    if (g_tls_cached_main_program == g_tls_main_program && g_tls_cached_main_tcb != nullptr) {
        return g_tls_cached_main_tcb;                       // fast path
    }
    g_tls_cached_main_program = g_tls_main_program;
    g_tls_cached_main_tcb =
        RuntimeLinker::TlsGetAddr(g_tls_main_program) + g_tls_main_program->tls.tcb_offset;
    return g_tls_cached_main_tcb;
}

Note KYTY_MS_ABI: this is called from generated code that has already switched conventions. The SafeCall stub of Calling conventions & the ABI wall sits between them, saving flags and volatile registers, aligning the stack and reserving shadow space.

Why this is expensive but acceptable

A single instruction has become a call plus register spills. Thread-local reads are common but not usually in the hottest loops, and the cached fast path keeps the handler short. The alternative — a full recompiler — costs vastly more.

Problem 2 — the stack canary

Compiler stack-protector code stores a guard value through fs:[0x28]. With an unsuitable FS base this can fault, including at address 0x28 when that base is zero. The loader recognizes this store pattern:

src/loader/runtimeLinker.cpp — the fs:[0x28] pass
const uint8_t fs_store_pattern[8] = {0x64, 0xc7, 0x04, 0x25, 0x28, 0x00, 0x00, 0x00};

for (auto* ptr = start_ptr; ptr <= end_ptr; ptr++) {
    if (memcmp(ptr, fs_store_pattern, sizeof(fs_store_pattern)) == 0) {
        LOGF("Patch fs:[0x28] store at addr: [%016" PRIx64 "]\n", (uint64_t)ptr);

        // recognise the specific "canary failed → ud2" epilogue
        if (ptr + 16 < start_ptr + size && ptr[12] == 0xcd && ptr[13] == 0x45 &&
            ptr[14] == 0x90 && ptr[15] == 0x0f && ptr[16] == 0x0b) {
            ptr[0] = 0x5d;                    // pop rbp
            ptr[1] = 0xc3;                    // ret
            std::memset(ptr + 2, 0x90, 15);   // nop the rest
        } else {
            std::memset(ptr, 0x90, 12);       // just nop the store
        }
    }
}

This is a compatibility rewrite of a recognized instruction pattern. It changes the guest’s stack-protector behavior; it is not a general implementation of the guest TLS or stack-canary contract.

Optional AMD patches and host instruction gaps

--amd-cpu enables the shared guestInstructionPatcher.cpp path on Windows, Linux and x86-64 macOS. PatchGuestInstructions(..., protect_memory, emulate_amd, host_features) uses function boundaries from .eh_frame and registered trampoline space. It patches supported VRSQRTPS forms and fills host-feature gaps for EXTRQ, INSERTQ, RDPID and CLWB, preferring native trampolines with safe trap fallbacks. Windows red-zone protection is controlled separately by --redzone.

The patch result records per-family counts for reciprocal square root, EXTRQ, INSERTQ, RDPID and CLWB, distinguishing native patches from trap fallback. Host features decide which compatibility path is needed; x86-64 macOS signal handling also preserves the SSE4a context. Native TLS keeps fiber context valid across host-thread migration, and fiber ownership preserves small guest stacks.

How to see this yourself

Run any title and grep the log for Patch tls at addr. Take one of those addresses into a debugger and look at the nine bytes: you will find 48 E8 … where the ELF file on disk has 64 48 8B 04 25 00 00 00 00. Seeing that once makes the whole mechanism concrete in a way no description does.

Watch

Byte by byte: an instruction the host cannot honour, replaced in place with one it can. Note that the replacement must be the same length or everything after it shifts.

And the newer, Windows-only pass: the 128 bytes System V promises a leaf function, which Windows feels free to overwrite.

Do

Find the patcher and read what it actually looks for.

Checkpoint