When recording OBS with OBS, you get the classic recursion of the preview window within itself.
How does OBS not crash when this happens? I can imagine two options:
1. Special or general-case recursion detection and recursion-limit in the code for screen-capture
2. Preview window is delayed by at least one frame so that the recursion happens across time
Is the solution one of these?
The recursion you're seeing is just old pixels from a previous capture that were copied into a new capture. It's the same thing that happens if you connect a video camera so its output is displayed on a TV/computer, then point the video camera at that output. To the video/camera computer system, it's no different than if there were no recursion "loop."
A special case is a microphone + speaker. If you feed the speaker output from the microphone back to the microphone, there's a feedback loop that amplifies the signal making that really loud humming sound.
This is what OBS does -- no recursion involved.