Reconstructing UI behavior from video instead of screenshots

I’ve been exploring a different approach to UI reconstruction: treating screen recordings as the source of truth, rather than screenshots, design files, or prompts.

Most tools that attempt to “generate UI” work from static inputs. A screenshot captures layout, but it misses what actually defines an interface: flow, state, and interaction over time.

A short screen recording, on the other hand, shows:

how navigation really works

which screens actually exist

what happens on hover, click, focus

which states are reachable vs. theoretical

I built a small prototype that watches UI video and reconstructs:

layout and component structure

navigation and page hierarchy

visible interaction states

a flow map of what was shown (and what wasn’t)

The key constraint is simple: if something wasn’t shown in the video, it isn’t generated. No guessing, no invented screens.

As a sanity check, I used a familiar public website as an example — not because it’s special, but because most people immediately recognize the structure and flow, which makes it easier to see what’s being reconstructed.

This isn’t about replacing design or development work. It’s more about reducing the “reverse-engineering” phase when all you have is a running product and no specs, no design files, and no original team.

I wrote a longer explanation here: https://www.replay.build/learn/behavior-driven-ui-reconstruc...

Curious if anyone here has tried similar approaches, or run into cases where static representations completely break down when you try to rebuild real interfaces.