Calling conventions & the ABI wall

The instruction sets match; the conventions do not. This chapter is the minimum x86-64 you need, and then the specific mismatch that shapes a surprising amount of KytyPS5.

The registers

x86-64 has sixteen 64-bit general-purpose registers. Their names are historical and inconsistent, which is confusing at first and then stops mattering:

rax rbx rcx rdx        the original four, extended to 64-bit
rsi rdi                "source index" / "destination index"
rbp rsp                frame pointer / stack pointer
r8 r9 r10 r11          added with x86-64
r12 r13 r14 r15

xmm0 … xmm15           128-bit vector registers, used for float arguments too
rip                    instruction pointer (not directly writable)
rflags                 condition flags

Plus segment registers, a leftover from 16-bit days. In 64-bit mode most are ignored, but two survive with a specific job: fs and gs provide a per-thread base address, used to reach thread-local storage. Remember fs — it comes back in Patching guest instructions and causes real trouble.

What a calling convention is

Machine code has no concept of a "function argument." When you write f(a, b), the compiler must decide: which register or stack slot does a go in? Who cleans up the stack? Which registers may f destroy? A calling convention (or ABI, application binary interface) is the agreed answer.

Two conventions matter here, and they disagree about almost everything:

They also differ on which registers a function must preserve. Under System V, rbx rbp r12–r15 are callee-saved. Under Microsoft, that list additionally includes rsi rdi and all 128 bits of xmm6–xmm15 (the upper YMM/ZMM portions remain volatile). A function compiled for one convention and called by the other will silently corrupt registers the caller expected to survive.

How KytyPS5 bridges it

Clang lets you set the convention per function with an attribute. Kyty defines two macros and uses them everywhere the boundary is crossed:

#define KYTY_MS_ABI   __attribute__((ms_abi))
#define KYTY_SYSV_ABI __attribute__((sysv_abi))

Any function the guest can call is marked KYTY_SYSV_ABI. The compiler then emits a System V prologue for that one function, while the rest of the emulator is compiled with the platform's native convention:

int KYTY_SYSV_ABI ContentExportInit2(const ContentExportInitParam2* init_param) {
    PRINT_NAME();
    if (init_param == nullptr) return CONTENT_EXPORT_ERROR_INVALID_PARAM;
    ...
}

The reverse direction exists too. When the emulator generates machine code that guest code will call — the TLS stubs of Patching guest instructions — that stub must switch back to the Microsoft convention before calling into C++. You can read the switch happening by hand in Jit::SafeCall:

sub    rsp,0x80            make room
pushfq                     save flags
push   rcx / rdx / r8 / r9 / r10 / r11 / rbx / rdi / rsi
mov    rbx,rsp
and    rsp,0xfffffffffffffff0   align stack to 16 bytes
sub    rsp,0x20                 reserve Win64 shadow space
movabs rcx,<handler>
call   rcx
mov    rsp,rbx             restore, pop everything back
...
ret

One more x86-64 detail: variadic functions

Under System V, a variadic function such as printf receives the vector-register argument count in al. This matters when forwarding a real variadic call. The current unresolved-import stub only logs and returns zero; it does not preserve arguments or forward a call.

It is also why Kyty cannot simply forward a guest printf to the host's vprintf: the argument list layout differs. src/libs/guestPrintf.cpp implements the guest format-string interpreter and argument decoding for exactly this reason.