Hacker News Reader

Comment by ncmncm | Hacker News Reader

I wonder if anyone has tried rendering directly to AV1 (or x264, or what-have-you), e.g. in a video game. There is all this hardware devoted to turning a few bytes of input into lots of pixels out. Maybe you could do less work producing just the few bytes, and let dedicated hardware generate the pixels. Of course you don't need for the CPU to generate the bytes; code running on the GPU would still do that, but with less work to do, you might get higher frame rates, or better detail at the same rate.

You heard it here first. (That is, unless it doesn't work.)

View on HN

GPU pipelines deal with geometry and spatial vectors to compute final pixels, so arguing about video encoders at this stage misses the point - you can only start encoding after you have an image, not before. This could only be a post production step (basically what the Steam Link does, I guess). Generally it does not make sense, because you'd get a lower framerate that would have a higher latency. It would also be inferior in quality and consume more power.

Encoding in something that needs less bandwith does make sense if you try to send the signal through a network connection e.g. to your TV, but none at all if you have a screen next to your computer.

Also, where do you get the impression that the input is small? A typical game now-a-days comes in at multiple gigabytes, nothing small there; tons of high res textures that go into the rendering pipeline - if anything the combined output is much smaller than the input.

You're essentially talking about abusing a video compression format for procedural generation - similar to concept that you can make a chatbot by running a text compression algorithm "in reverse".

It's a fun idea. I don't think it makes sense for video games, because video compression formats are designed for, well, video (i.e. 2D moving pictures - the videos that compress best with current tech tend to be animation, with large areas of flat colour that move around). It would be a good fit for moving-pictures animation, something with the kind of motion that 2000s-era Flash animations had (or Inferno Cop for a recent example) - but to be successful with that style you want either beautiful hand-drawn art (like e.g. Child of Light) to make up for the lack of motion, or at a minimum you want clean vector lines. Video formats handle non-predicted parts more like jpeg, so you'd end up with a bunch of blocky and muddy shapes that moved around like a flash animation - not an aesthetic that I imagine appeals.

For a demoscene-style "minimal filesize" way of creating visuals it would make sense, except that the graphics card already has a lot of hardware for rendering 3D scenes from geometric primitives, and those tend to look better. I mean, ultimately the way you use video decompression hardware rather than generating pixels in general-purpose code is that you ship a video file rather than an executable that renders video - and video compression has a long way to go to catch up with procedural generation. I see this pretty directly as I sometimes render and encode videos (from MikuMikuDance), and a few hundred kilobytes of models, textures and motions will inevitably result in a hundreds-of-megabytes encoded video that still looks noticeably worse than the actual rendering.

> Maybe you could do less work producing just the few bytes

It takes more work to produce fewer bytes, because you've to pack the same information in fewer bytes. Entropy and all that.

Not if the information starts out in fewer bytes. Then, you're just rearranging the bits into renderable order. "Chicken" is a very small number of bits that would need to be expanded to more bits (leg down here, wing over there, beak up here) before handing them over to the decoder to make actual feathers. The representation in a game's scene list is overwhelmingly more compact than the input to a decoder would be, and most of it barely changes from one second to the next, as the pixels are all replaced dozens of times over.

The information does not start out in fewer bytes. It starts out in all the bytes that represent the 3D models of entities you're interacting with in your game and the textures to cover them, at least. That's usually more bytes than needed to show a 32 bit 4K picture. The rendering process reduces all the jungle you have in your VRAM to a few kB that are briefly shown on your screen.

It seems like you don't know the basics of 3D graphics nor the basics of video encoding. It is extra work and would save some bandwidth needed to pump the picture to the display, which is not an issue in 3D graphics (we have dedicated cables).

This is exactly what OnLive/Parsec/Playstation Now do. Keep the video card in the cloud, stream to a screen. Works pretty good if you have a fiber connection, pretty hit and miss otherwise.

I thought PlayStation Now streamed content from PS4 to PS Vita. I may be messing up the names tho.

You're thinking of Remote Play. That's what they call their streaming thing from PS4 to PS Vita, though it does support Windows nowadays.

games can be procedurely generated. Videos come from a source video that acts as a standard.

Reply on news.ycombinator.com