Xnest is probably enough. I've used it for similar purposes a few times. Don't know the equivalent for Wayland though.
That said, with both the X nesting approach and the Wayland nesting approach, you'd also need to run the screencasting application itself inside the nested server, not the just the application you want to cast. If the compositor supports a way to create headless outputs (as sway and hyprland do) that is much easier.