In general, wayland does not implicitly have a distinction between server and window manager. There is a server and there are clients. The server gets to decide where a particular client's windows are rendered. Wayland itself is just a spec of the protocol that the client and server use to talk to each other - the format of the messages, and a few documents that describe various such messages. What the server does is basically completely arbitrary.

I don't know about GNOME and KDE's wayland servers, but with sway and river and cage the server is a single process that does the job of both server and window manager. It doesn't mean everyone has to write the equivalent of an X server, ie code that interacts with OpenGL and the kernel interfaces, themselves - there is a popular library called wlroots that can be used for that.

I don't think it's impossible that someone wraps wlroots into a standalone "server" binary that then has its own protocol for talking to an arbitrary "window manager" binary at runtime. ie do the equivalent of what wlroots does with function calls and callbacks at compile time but at runtime via IPC. But I don't know anyone who's doing such a thing.

People have already mentioned wlroots as a starting point, but there is a less opinionated and more compatible (NVIDIA-ready) library that I’m really quite fond of called Mir: https://github.com/MirServer/mir

One thing to note, Wayland, unlike X, does not support server side decorations yet, so compositor’s responsibilities are mostly just placing windows.