How a new emulator generates 3D scenes from 2D NES games | Hacker News Reader

How a new emulator generates 3D scenes from 2D NES games

arstechnica.com

168 pointsbpierre22 comments

Wow, take this idea further with how google lets an AI "dream" with fish and pigs onto normal everyday scenes, you could allow the AI to visually interpret a game scene in real time but train it with domain-specific imagery so it's more fantastic. In other words, you get a scarier Dracula or cooler looking hero because its from the "imagination". We always say reading a book is more amazing because you imagine it to be so. Why not with AI doing the imagining for us?

Let's go even further beyond graphics, imagine a Visual Imagination API that could take in video as well as textual/voice input. This would enable a storyteller in front of a huge display, intensely narrating while a living canvas responds to the words spoken for everyone to see, now imagine this in VR, living inside your own AI's imagination of your own narration. Woooooooooooeeeeeeeeee -

OK because you keep up-voting... now imagine an augmented life filter that is an AI interpreting reality for you in a very rosy and agreeable way. It seems everyone and everything are pretty much awesome. Your AI output filter is particularly suave these days. After all, your personality is the best money can buy.

This isn't just a conversion of every pixel into a uniform voxel, either. In a game like Super Mario Bros., for instance, 3DNES converts pipes to into cylindrical 3D models

I actually think voxels might look way cooler than attempting to generate smooth 3D objects with the original sprites textured mapped on top. It would retain more of the original character of the game, while adding some depth.

> "I actually think voxels might look way cooler than attempting to generate smooth 3D objects with the original sprites textured mapped on top. It would retain more of the original character of the game, while adding some depth."

Do you mean like this mock up?

http://cdn.ubergizmo.com/photos/2008/7/3d-smb.jpg

Would like to see shadows, they are a great depth cue.

Yeah, that looks nice, though probably with a lot less depth, since the player can't move along the z-axis.

agreed, it would look much better with flatter lighting as well

Relevant discussion on the nesdev forum: http://forums.nesdev.com/viewtopic.php?f=3&t=13552

Some NES games used some crafty hacks to get more sprites on screen then allowed by the GPU and Video Ram, I am curious if they will every be able to render that without having to make many custom rules for each game.

I want to see Contra in this one. Particularly the scenes that try to approximate a FPS.

There's a video in the article that shows Contra in 3d!

Haha didn't even see it. Thanks. Went back and rewatched someone playing it without the code while I was at it.

https://www.youtube.com/watch?v=wcuAYSN-CvY

_and can load arbitrary (and definitely legally obtained) ROMs from the cloud._

I laughed.

Ack. That doesn't look very good. And it runs unplayably slow and glitchy / flashy.

"Your first assumption might be that geod has hard-coded specific algorithms for each supported game, training the emulator in how to detect specific objects and create good-looking 3D models for them."

I don't know why anyone would assume that, given how poorly it works. My current assumption would be that game-specific algorithms would be the only way to make this work given how ambiguous NES graphics are. I suppose a top-tier neural network expert could make a game-agnostic algorithm... but it would probably handle a frame per minute on a multi-thousand-dollar machine.

I wish someone would come up with better filters for emulating CRT glow so the game would look like a series of photos of the game being played on a top-of-the-line CRT (PVM / BVM). But that would probably be way too slow, too. Actually I have an idea of how to do it...

If I could get an array of pixels per frame into Unity or some other game engine and make each pixel a light... or even just make each pixel a partially transparent shaded sphere bitmap and multiply them together where they over lap... it would probably run too slow but I once made something like that and looked pretty good and ran plenty fast albeit at a sub-NES resolution.

You're way too pessimistic about doing image recognition on NES tiles. NES games don't update the tileset very often during gameplay. They can't--the NES has no framebuffer and a very underpowered CPU, so the VBLANK time is a mad dash to push all per-frame data to the GPU (PPU in NES-speak) by the time the video signal is going to begin. Most games barely slide under this window. Lots of games didn't even try to do dynamic tileset updates and stored tileset data in ROM.

Because of this, you can do a lot of graphics processing up front and cache it on a per-tile basis. The hit rates on that cache are going to be massive.

Agreed. The hardware itself does palette and tile lookups for each pixel, then shoves the data out the wire. Building an emulator that way is more literally how the hardware works, but it's slow. It's easier (and usually just as accurate) to cache the tiles for the current palette settings and blit them to the playfield as needed. There's no reason that the same wouldn't be true for 3D representations of those same sprites.

Updating the sprites in CRAM is fairly rare (and impossible in any game that uses CROM).

>I don't know why anyone would assume that, given how poorly it works.

You're tough. Every type of computer generated visuals have some cases that won't work well, given that there are so many semantic choices: anti-aliasing, dithering, textures, forms, compete for the information of each pixel. Each pixel can encode any subset of these features. We rely on the human brain to parse it and decode it meaningfully, and even then, it doesn't always work. If you've tried yourself at pixel art, you'll understand quickly (you can also look at the research on upscaling algorithms). So the result is very far from an actual product, but it's really impressive.

Alright, I put it far too rudely, but I do find this aesthetically displeasing compared to raw pixels and Contra looked completely unplayable, and it shows massive slowdown and audio distortion, so I don't get the point.

The article states that it's in beta still, so I assume the audio at least will get cleaned up.

>I wish someone would come up with better filters for emulating CRT glow

I'm a fan of Blargg's NTSC filter[a], included in snes9x-gtk.

[a]http://slack.net/~ant/libs/ntsc.html

I highly recommend reading Kyle Pittman's article on CRT emulation:

http://gamasutra.com/blogs/KylePittman/20150420/241442/CRT_S...

Reply on news.ycombinator.com