As I remember, generic DRM interface allows only writing to a framebuffer and not reading from someone's else framebuffer. Do they use some vendor-specific APIs?
The basic idea is to enumerate the CRTCs to get the relevant framebuffer plane resources and then use dma-buf exports to pass them to an encoder. Reality can be a bit more complex due to the existence of overlay planes but that's the idea.
https://github.com/LizardByte/Sunshine/blob/master/src/platf...