I have been following the development of screen capturing under Wayland a little and what I can tell you is that at least for Wayland OBS directly interfaces with pipewire (which is very much platform specific and low level), see: https://github.com/obsproject/obs-studio/pull/4287/files