Page faults & CPU/GPU coherency

Who has the current bytes?

The CPU and GPU can access different representations of the same guest data. Coherency work ensures a consumer sees the required writes. Protection faults can notify the emulator about an access, but the handler must still find the owner, synchronize and copy or invalidate as needed.

First connect B5's completion rule to one CPU-write → GPU-read sequence. Page-manager internals, sparse backing and texture-alias cases are implementation reference.

Does detecting a write make the GPU see its bytes immediately?

No. Detection identifies work to do. Visibility still depends on the correct upload or shared-memory path, synchronization and ordering before the GPU reads the data.

Use the code-example guide: snippets below are source excerpts or labelled schematic sequences, not standalone programs.

The single hardest problem in the project, and the source of the bugs that are hardest to reproduce. A console has unified memory; a PC does not.

The problem, stated plainly

On a PS5, CPU and GPU share one pool of physical memory. A game writes vertex data with the CPU, then draws with it. Shared backing avoids a mandatory copy between separate CPU and GPU allocations, but ordering and visibility still require the guest’s synchronization and cache-coherency rules.

On a PC, the GPU has its own memory. Kyty's Vulkan resources are copies of guest memory. So the emulator must answer two questions that the game never asks:

• Has the CPU changed this data since I last copied it to the GPU? (→ upload)
• Has the GPU changed this data since the CPU last read it? (→ download)

And it must answer them without any cooperation from the game.

Two cooperating classes

enum class PageFaultAccess { Read, Write, Execute, Unknown };
class PageManager final {
public:
    template <bool track>
    void UpdatePageWatchers(uint64_t vaddr, uint64_t size);
    template <bool track, bool is_read = false>
    void UpdatePageWatchersForRegion(uint64_t base_addr, RegionBits& mask);
};

The upload path

Before a draw reads a texture, the renderer asks which sub-ranges the CPU dirtied:

template <typename RangeFunc, typename UploadFunc>
    void ForEachUploadRange(uint64_t vaddr, uint64_t size, bool is_written, RangeFunc&& range_func,
                            UploadFunc&& upload_func) {
        static_assert(std::is_nothrow_invocable_v<RangeFunc&, uint64_t, uint64_t>);
        static_assert(std::is_nothrow_invocable_v<UploadFunc&>);
        CheckNotInUploadCallback();
        Iterate<true>(vaddr, size, [](RegionManager*, uint64_t, uint64_t) {});
        const auto* previous_upload_owner = std::exchange(s_upload_owner, this);
        Iterate<false>(vaddr, size, [&](RegionManager* manager, uint64_t offset, uint64_t bytes) {
            manager->lock.lock();
            manager->ForEachModifiedRange<DirtySource::Cpu, true>(manager->GetCpuAddr() + offset,
                                                                  bytes, range_func);
            if (!is_written) {
                manager->lock.unlock();
            }
        });
        upload_func();
        if (is_written) {
            Iterate<false>(vaddr, size,
                           [](RegionManager* manager, uint64_t offset, uint64_t bytes) {
                               manager->template ChangeState<DirtySource::Gpu, true>(
                                   manager->GetCpuAddr() + offset, bytes);
                               manager->lock.unlock();
                           });
        }
        s_upload_owner = previous_upload_owner;
    }

Three things to notice:

• Only dirty sub-ranges are copied. The transfer granularity depends on the buffer/image path, tracking pages and tiling; a changed tile does not guarantee a one-tile copy.
• Locks are held across the callback. The user's range_func runs with region locks held, which is why it is required to be noexcept.
• Protection is re-armed afterwards. Otherwise the next CPU write would go unnoticed.

The download path

Download iteration visits the requested GPU-dirty ranges under each region lock. The callback is required to be noexcept; clear controls whether GPU-dirty state is cleared:

template <bool clear, typename Func>
    void ForEachDownloadRange(uint64_t vaddr, uint64_t size, Func&& func) {
        static_assert(std::is_nothrow_invocable_v<Func&, uint64_t, uint64_t>);
        CheckNotInUploadCallback();
        Iterate<false>(vaddr, size, [&](RegionManager* manager, uint64_t offset, uint64_t bytes) {
            std::scoped_lock lock(manager->lock);
            const auto       address = manager->GetCpuAddr() + offset;
            manager->template ForEachModifiedRange<DirtySource::Gpu, false>(address, bytes, func);
            if constexpr (clear) {
                manager->template ChangeState<DirtySource::Gpu, false>(address, bytes);
            }
        });
    }

The separate extended-memory arena

Ordinary guest mappings keep their existing low address bands. Extended allocations use a separate 512 GiB virtual arena at 0x080000000000, an 8 TiB base, described by kExtendedMemoryBase and kExtendedMemorySize in kernel/memory.h. Reserving that address range does not allocate 512 GiB of physical RAM. Extended requests and ordinary mapping searches have separate allocation rules.

GPU tracking accepts the low 1 TiB region and that explicit extended interval; the space between them is not a valid resource range. Sparse image tracking covers 44-bit addresses. Dense buffer/BDA tables pack the low and extended intervals together through BufferCache::PageIndex() (guest address to packed page index) and GuestAddress() (packed byte offset to guest address), avoiding tables for the entire gap.

Automatic backing is recycled through physical-allocation ownership. Reserved mappings are published only after backing commit succeeds, and partial POSIX unmaps preserve still-live mappings. These are allocation/alias invariants tested by VirtualMemoryAllocationTests.cpp. Host primitives are declared directly in common/virtualMemory.h and implemented by sysWindowsVirtual.cpp / sysLinuxVirtual.cpp.

The invariants, enforced by hard failure

This code is unusually defensive, and the reason is worth internalising: a coherency bug produces corruption many frames later, with no traceable link to its cause. So Kyty converts each one into an immediate crash at the point of the mistake.

void CheckNotInUploadCallback() const noexcept {
    if (s_upload_owner == this) {
        EXIT("memory tracker re-entered from upload callback\n");
    }
}

And in InvalidateRegion, the state update runs twice — once to decide whether a flush is needed, once to verify the flush actually released every GPU-owned page:

template <typename Flush>
void InvalidateRegion(uint64_t vaddr, uint64_t size, Flush&& on_flush) {
    ...
    if (!update_cpu_state()) return;      // nothing GPU-owned; done
    std::forward<Flush>(on_flush)();      // flush GPU work
    if (update_cpu_state()) {
        EXIT("memory invalidation retained GPU-owned pages\n");
    }
}

How a CPU write is actually caught, end to end

1. Resource tracking installs write watchers through PageManager, protecting guest pages whose CPU changes need tracking.
2. The game later writes to the texture. Ordinary code — mov [rax], rbx.
3. The CPU faults. The OS delivers it to KytyExceptionHandler (this topic).
4. Memory::HandleGpuFault coordinates invalidation or readback as required, updates tracking and protection, and returns true.
5. Execution resumes; the write completes. The game never knew.
6. The next draw that reads the texture calls ForEachUploadRange, which reports exactly that page as dirty and copies it.

Because guest code normally runs natively, Kyty does not instrument every memory access. The CPU's own fault mechanism becomes the emulator's primary hook — two of the handler's three branches are load-bearing, not diagnostic.

The handler

static bool KytyExceptionHandler(const Common::HostException::ExceptionInfo& exception_info) {
    const auto* info = &exception_info;

    // 1 — an instruction this CPU does not have
    if (info->type == ExceptionType::IllegalInstruction &&
        Loader::X64InstructionEmulator::TryEmulate(info->native_context)) {
        return true;
    }

    if (info->type == ExceptionType::AccessViolation) {
        const auto access = /* map Read/Write/Execute */;

        // 2 — the GPU cares about this page
        if (Libs::LibKernel::Memory::HandleGpuFault(access, info->access_violation_vaddr)) {
            return true;
        }
    }

    // 3 — a genuine crash: print what can be read safely, then stop
    std::printf("--- Guest fault context ---\n");
    std::printf("thread: %s\n", thread_name);          // which guest thread, by its pthread name
    std::printf("rax=... rbx=... rcx=... rdx=...\n" ...);  // all sixteen registers
    // 96 code bytes around pc, 32 stack qwords from rsp — each range probed first
    EXIT("Unhandled host exception: type=%u code=%u pc=0x%016" PRIx64
         " access=%u address=0x%016" PRIx64 "\n", ...);
}

Returning true means "handled, resume execution." The guest never learns anything happened. On Linux and macOS the handler runs on a per-thread alternate signal stack (InitializeThreadSignalStack, installed for every guest thread) precisely because branch 2 can fire while the guest's own stack pages are write-protected by the GPU tracker; Windows delivers vectored exceptions on the faulting thread; its red-zone compatibility path is separate.

Reservations are committed explicitly. A fault in reserved-but-uncommitted memory is a real bug and falls through to the dump.

Branch 1 — emulating missing instructions

The PS5's Zen 2 supports instructions your CPU might not, notably the SHA extensions. When one executes, the CPU raises an illegal-instruction fault, and X64InstructionEmulator decodes the bytes at the fault address, performs the operation in software against the saved register context, and advances the instruction pointer past it.

static void Sha1Msg1(XmmWords& dest, const XmmWords& src2) {
    const uint32_t w0 = dest.w[3];  const uint32_t w1 = dest.w[2];
    const uint32_t w2 = dest.w[1];  const uint32_t w3 = dest.w[0];
    const uint32_t w4 = src2.w[3];  const uint32_t w5 = src2.w[2];
    dest.w[3] = w2 ^ w0;   dest.w[2] = w3 ^ w1;
    dest.w[1] = w4 ^ w2;   dest.w[0] = w5 ^ w3;
}

It is slow — a fault plus a software emulation per instruction — but these appear in checksum and hashing routines rather than in hot loops, and the alternative is not running at all.

Branch 2 — the GPU fault, and why it is the design

This is the most important branch in the emulator. Memory the GPU has cached as a Vulkan image or buffer is deliberately write-protected. When the game writes to it:

Page protection provides a notification when native guest code touches tracked memory. Guest GPU commands still carry ordering and cache operations; the emulator combines those commands with fault tracking to keep host CPU and Vulkan resources coherent.

Branch 3 — the crash dump

When nothing handles the fault, the handler prints a --- Guest fault context --- block — everything it can read without faulting again — and then exits through EXIT:

• The guest thread name from the pthread layer (or (host thread) if the fault was not on a guest thread) — which identifies the thread, not the executing module; use the faulting PC to locate the code.
• All sixteen general-purpose registers, four per line, as rax=….
• 96 bytes of code around the fault address — pc-48 .. pc+48, the faulting byte at offset 48 — probed with IsReadableRange first so the dump does not itself fault.
• 32 stack qwords read from rsp, again only if readable.
• The final Unhandled host exception: type=… code=… pc=… access=… address=… line: the exception type, the native code, the program counter, the access kind and the address that was touched.

This fault path prints guest context rather than a host stack walk. The guest frame-pointer walker is RuntimeLinker::StackTrace(), which prints Stack trace [thread = N]: followed by one [i] address, off=…, module line per frame, resolved against the loaded modules; the libc abort path calls it. Accepting a frame only when the return address lands inside a loaded module's segments is precisely what fixing the module layout in Mapping a program into memory buys.

void RuntimeLinker::StackTrace(uint64_t frame_ptr, uint64_t stack_ptr) {
    void* stack[20];
    int   depth = WalkGuestStack(frame_ptr, stack_ptr, stack, 20);
    LOGF("Stack trace [thread = %d]:\n", Common::Thread::GetThreadIdUnique());
    for (int i = 0; i < depth; i++) {
        auto  vaddr = reinterpret_cast<uint64_t>(stack[i]);
        auto* p     = FindProgramByAddr(vaddr);          // which module owns this return address?
        LOGF("[%d] %016" PRIx64 ", off=%016" PRIx64 ", %s\n", i, vaddr,
             (p == nullptr ? 0 : vaddr - p->base_vaddr), (p == nullptr ? "???" : name_of(p)));
    }
}

Sparse texture uploads

BufferCache::ObtainBufferForImage() reads streamed mip ranges through Memory::TryReadSparseBacking(). The guest address-space owner validates the complete range and rejects private mappings before the backing store copies resident pieces and zero-fills holes. A hole inside an owned sparse reservation is not the same as an arbitrary invalid guest address; do not replace failed reads with unconditional zeroes.