So this is surprisingly bleeding edge, at least to me. I had to go learn about some hardware and physical imaging stuff I didn’t know to get my head around it.

Upshot: Event Cameras are a different sort of camera in that they have an array of sensor pixels, and sensors only fire when there is a brightness change for that sensor. This has a bunch of benefits, including very high dynamic range, reduced ghosting, and high frame rates, and has some downsides, like reconstructing video, and presumably others.

The paper seems to have started out with the idea that if you had event camera output, you’d be able to reconstruct more fine texture details. And, this works incredibly well, their baby model trained for 8 days significantly beats SOTA and looks a lot better in comparisons as well.

They then seem to have added a step where you simulate/infer event camera data from “normal” RGB video, using a different set of networks, and use that inferred event data to do the texture recovery, and … this also works.

Pretty surprising, and interesting. Their GitHub is full of people like “I want to try this” and then realizing it’s a fairly deep stack to deploy. Even as is, it seems worth someone building a GUI around this in an app, it’s quite remarkable.