Super Mario Bros Augmented Reality

awesome == spam

I'll take the downvote - but "because it's awesome" isn't why we are here. At least, its not why I'm here. There are plenty of places where links like this are great - but I'm probably not alone when I say it doesn't belong here.

I don't know the exact process or tools he used, but I can describe how I'd do this.

First, you need footage of the game being played through, obviously. Then, take the static elements (the playfield) and create a giant animated map, including coin animations and enemies walking. Then, splice in the capture of Mario running through the levels, removing any collected coins or defeated enemies from the "static" playfield.

Once your giant resolution animation is finished, you then need to shoot some video for the backdrop. In his video, he obviously timed his walking to where the playfield would be (possibly by playing back the video on a handheld device while filming?).

Now that you have the video game animation and backdrop, it's simply a matter of layering the animation onto the video using camera matching. Most video post editors have this capability.

Most of the work looks like it resides in the actual animation. If you notice in many underground levels, they aren't arranged the way you see in the video; he did alter some portions to form a continuous map.