There is quite a bit of variation here. Linux and, AIUI, quite a few microkernels, have a defined ABI at the user/kernel boundary. BIOS, a bunch of hypervisors, and some RISC-V M-mode interfaces are similar, albeit with “user” and “kernel” replaced with other execution modes.
OpenBSD and, very notably, Windows instead have a defined library interface. Windows has somewhat stable “system calls”, but only through NTDLL.dll. The real stable interface is kernel32, etc.
There are arguments in favor of both approaches. One significant, if rarely appreciated benefit of the user/kernel approach is that it enables user programs that don’t use the C ABI at all. Go on Linux is like this, and Go and the vDSO have interoperability issues because the vDSO is a bit FFI-ish in Go, and Go’s FFI is not very fast.