Concurrency & the memory model

The threads lesson identified execution roles, and coherency explained CPU/GPU visibility. Connect those ideas here: follow work handed between CPU threads, distinguish a lock from device completion, and identify which synchronization condition protects each resource. Bring that reasoning to the draw trace next.

CommandScheduler has a priority completion worker for guest-memory callbacks. DeferPriorityOperation() queues such callbacks after the submission tick; ordinary deferred callbacks manage GPU resources. Shutdown and dependent callbacks wait for the relevant priority work. Trace this separately from recording Vulkan commands on the GPU worker.

The main thread roles

Three of them are created before a single instruction of the game runs. The main thread drives the window and SDL; the guest thread runs the game itself; the GPU thread parses PM4 and drives Vulkan; and the audio thread mixes and plays back. This is the guest and GPU pair, both from emulator.cpp and graphicsRun.cpp:

static void Execute(const std::filesystem::path& game_patch) {
    auto patch_path = game_patch;
    Common::Thread guest_thread(
        [](void* param) {
            auto* rt = Common::Singleton<Loader::RuntimeLinker>::Instance();
            rt->Execute(*static_cast<const std::filesystem::path*>(param));
        },
        &patch_path);
    Libs::Graphics::WindowRun();   // main thread runs the window loop
    std::quick_exit(0);
}

GuestGpu::GuestGpu(RenderContext& renderer): m_renderer(renderer) {
    EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
    GraphicsInitJmpTables();
    m_gfx_cp = std::make_unique<CommandProcessor>(renderer, 0);
    m_gfx_cp->Reset();
    m_thread = std::jthread(ThreadRun, this);
}

Note the two thread wrappers across these files: the guest thread uses Common::Thread (the project's own wrapper, so it can keep a unique per-thread id for logs and TLS), while the GPU thread uses std::jthread (C++20, joins automatically on destruction). Both run on the same host process — a guest crash is a real crash in this process (see topic C3).

Handing work between threads

The guest never touches Vulkan directly. It fills PM4 buffers and calls GuestGpu::SendCommand, which has the classic "run inline if you're already on the right thread, otherwise queue it" shape:

void GuestGpu::SendCommand(Common::UniqueFunction<void>&& command) {
    EXIT_IF(!command);
    if (IsGpuThread()) {
        command();
        return;
    }
    Common::LockGuard lock(m_queue_mutex);
    EXIT_IF(!m_accepting);
    m_commands.push_back(std::move(command));
    m_pending_commands.fetch_add(1, std::memory_order_release);
    m_work_available.Signal();
}

Two ideas worth absorbing:

• The lock is held only to push onto the queue, never while running. m_commands is guarded by m_queue_mutex; the actual work happens outside the lock on the GPU thread. That is the difference between a queue and holding a lock across a long operation.
• A condition variable is the wake-up. m_work_available lets the GPU thread sleep when idle and wake exactly when work arrives, instead of busy-polling. The GPU thread's whole job is a loop: wait for signal, pop a submission, process it.

The GPU thread actually serves 57 queues — one graphics queue plus seven compute pipes × eight queues each:

static constexpr uint32_t ComputePipeCount     = 7;
static constexpr uint32_t QueuesPerComputePipe = 8;
static constexpr uint32_t QueueCount           = 1 + ComputePipeCount * QueuesPerComputePipe;
std::array<std::deque<Submission>, QueueCount> m_queues;

The guest submits to a specific queue the way it would on real hardware; Kyty keeps them separate and round-robins its single worker across them. That is a faithful model of a thing it is not physically reproducing.

Atomics and memory order

A useful counting example spans presentation/window/swapchain.cpp and presentation/window/window.cpp. Successful presentation increments a counter; the main window loop samples it to compute the title’s FPS:

// Presenter::Impl::Present()
window.loop.presented_frames.fetch_add(1, std::memory_order_relaxed);

// WindowContext::Run()
const auto frames = loop.presented_frames.load(std::memory_order_relaxed);

Relaxed operations remain atomic: concurrent increments are not lost, and each atomic object has a modification order. A sampled load need not reflect the latest increment. Relaxed ordering does not publish other memory, so observing this count says nothing about resource lifetime or GPU completion. Here the main thread only needs a statistics sample; it updates the SDL title itself rather than making the presentation thread wait.

For shared work, read the whole synchronization protocol. SendCommand() above protects the queue with a mutex and increments m_pending_commands with release ordering; completion checks use acquire ordering. Release/acquire can establish visibility when an acquire observes a relevant release. Sequentially consistent atomics additionally share a global order, but neither a stronger atomic order nor an atomic counter alone makes unrelated data accesses or object lifetimes safe.

Locks and condition variables

Where a structure is actually shared and mutated — an address-space free list, a submission queue — the tree locks. The project wraps std::mutex as Common::Mutex with a Common::LockGuard RAII guard, but also uses std::mutex directly:

auto [it, inserted] = m_free.emplace(vaddr, size);   // structured binding (topic C1)
...
std::mutex m_mutex;   // every allocation path takes this

And the quiet winner: thread_local. The PRINT_NAME flag from HLE libraries is per-thread, so one thread can enable tracing without a single atomic. Anything that is only ever touched by the thread that owns it needs no synchronisation ; shared guest memory still requires synchronization.

The coherency channel — the cross-thread handshake

CPU and GPU share one memory in the guest's model. On the host, the GPU thread must see what the guest thread wrote. The emulator's trick (covered properly in Page faults & CPU/GPU coherency) is that the GPU never polls — it faults. When the guest writes a page the GPU has been told about, a host page-fault fires on the faulting thread, and the exception handler routes it to the memory layer:

if (Libs::LibKernel::Memory::HandleGpuFault(access, info->access_violation_vaddr)) {
    return true;   // the fault was a coherency notification, not a bug
}

Page protection detects guest accesses; the fault handler coordinates with the memory and graphics layers to make those accesses safe. Protection is part of the tracking mechanism, not a replacement for locks or CPU/GPU completion ordering. On Linux and macOS the handler runs on a per-thread 64 KB alternate signal stack (InitializeThreadSignalStack) so the fault can still be delivered when the guest stack page is write-protected. It is the single most distinctive thing about this codebase, and it is why reinterpret_cast and page-granular thinking dominate.

The GPU's own clock: fences & timeline semaphores

Between the GPU thread and the Vulkan driver, ordering is expressed with semaphores. The renderer submits with wait ticks and signal ticks — a Vulkan timeline semaphore, a monotonically increasing counter the GPU advances as work completes:

n_info.waitSemaphoreValueCount   = submit.num_wait_semaphores;
n_info.pWaitSemaphoreValues      = submit.wait_ticks.data();
n_info.signalSemaphoreValueCount = submit.num_signal_semaphores;
n_info.pSignalSemaphoreValues    = submit.signal_ticks.data();

The reason the game and GPU can overlap is that several submissions are in flight at once. Each command buffer carries a monotonic timeline tick; MasterSemaphore reports the latest completed value, and deferred objects are released only after their tick passes. CommandPool starts in batches of four and grows by four when no completed buffer is reusable, instead of rotating through a fixed fence-backed pool.

The kernel's own synchronisation surface

Games use both native atomic instructions and console synchronization APIs, and those sit on top of FreeBSD-style primitives in src/kernel/: