If you find this interesting, you might also be interested in this video of someone diving even deeper into how to make the dither surface stable: https://www.youtube.com/watch?v=HPqGaIMVuLs
1. Turn the previous dithered framebuffer into a texture
2. Set the UV coordinates for each vertex to their screenspace coordinates from the previous frame
3. Render the new frame using the previous-framebuffer texture and aforementioned UV coords, with nearest-neighbor sampling and no lighting etc. (this alone should produce an effect reminiscent of MPEG motion tracking gone wrong).
4. Render the new frame again using the "regular" textures+lighting to produce a greyscale "ground truth" frame.
5. Use some annealing-like iterative algorithm to tweak the dithered frame (moving pixels, flipping pixels) to minimize perceptual error between that and the ground truth frame. You could split this work into tiles to make it more GPU-friendly.
Steps 4+5 should hopefully turn it from "MPEG gone wrong" into something coherent.
The video ends in a place where I suspect even further advances could still be made.
But yes, there's still the issue of oblique angles looking different that still remains open AFAIK.