Comment by PaulDavisThe1st | Hacker News Reader

Comment by PaulDavisThe1st | Hacker News Reader

PaulDavisThe1st

It is fascinating to me, as someone who has been doing real time audio work for 20 years, to see how two fundamental processes have been at work during that time:

   * audio developers recognizing the parallels between the way audio and video hardware works, and taking as many lessons from the video world as possible. The result is that audio data flow (ignoring the monstrosity of USB audio) is now generally much better than it used to be, and in many cases is conceptually very clean. In addition, less and less processing is done by audio hardware, and more and more by the CPU.

   * video developers not understanding much about audio, and failing to notice the parallels between the two data handling processes. Rather than take any lessons from the audio side of things, stuff has just become more and more complex and more and more distant from what is actually happening in hardware. In addition, more and more processing is done by video hardware, and less and less by the CPU.

In both cases, there is hardware which requires data periodically (on the order of a few msec). There are similar requirements to allow multiple applications to contribute what is visible/audible to the user. There are similar needs for processing pipelines between hardware and user space code.

(one important difference: if you don't provide a new frame of video data, most humans will not notice the result; if you don't provide a new frame of audio fata, every human will hear the result).

I feel as if both these worlds would really benefit from somehow having a full exchange about the high and low level aspects of the problems they both face, how they have been solved to date, how might be solved in the future, and how the two are both very much alike, and really quite different.

I like your overall point a lot. I'm an ex-game developer who has also dabbled in audio programming and music so I see a little both sides.

I think the key difference, though, is that the consumer video needs are probably in the area of two orders of magniture more computationally complex for video than for audio. On consume machines, almost no interesting real-time audio happens. It's basically just playback with maybe a little mixing and EQ. Your average music listener is not running a real-time software synthesizer on their computer. Gamers are actually probably the consumers with the most complex audio pipelines because you're mixing a lot of sound sources in real-time with low latecy, reverb, and other spatial effects.

The only people doing real heavyweight real-time audio are music producers and for them it's a viable marketing strategy for audio programmers to expect them to upgrade to beefier hardware.

With video, almost every computer user is doing really complex real-time rendering and compositing. A graphics programmer can't easily ask their userbase to get better hardware when the userbase is millions and millions of people.

Also, of course, generating 1470 samples of audio per frame (44100 sample rate, stereo, 60 FPS) is a hell of a lot easier than 2,073,600 pixels of video per frame.

I agree that audio pipelines are a lot simpler, but I think's largely a luxury coming from having a much easier computational problem to solve and a target userbase more willing to put money into hardware to solve it.

> The only people doing real heavyweight real-time audio are music producers

I wouldn't discount the amount of people using voice assist and rely on noise reduction/echo cancellation in their meetings, or the near future where we have SMPTE 2098 renderers running on HTPCs.

I really think we're only a few years away from seeing a lot more realtime DSP in consumer applications. Conferencing, gaming, and content consumption all benefit immensely, and it would good for us all to start thinking about the latency penalty on audio-centric HCI like we do for video.

You don't generate 1470 samples of audio per frame.

You generate N channels worth of 1470 samples per frame, and mix (add) them together. Make N large enough, and make the computation processes associated with generating those samples complex enough, and the difference between audio and video is not so different.

Jacob Collier routinely uses 300-600 tracks in his mostly-vocal overdubs, and so for sections where there's something going on in all tracks (rare), it's more in the range of 400k-900k samples to be dealt with. This sort of track count is also typical in movie post-production scenarios. If you were actually synthesizing those samples rather than just reading them from disk, the workload could exceed the video workload.

And then there's the result of missing the audio buffer deadline (CLICK! on every speaker ever made) versus missing the video buffer deadline (some video nerds claiming they can spot a missing frame :)

You're restating the grandparents point: The only people doing real heavyweight real-time audio are music producers. Once Jacob Collier is done rendering his song, it's distributed as a YouTube video with single stereo audio track.

And, talking about learning lessons from other fields, there's no particular reason that Jacob has to to render his audio at full fidelity in real time. Video editors usually do their interactive work with a relatively low fi representation, and do the high fidelity rendering in a batch process that can take hours or days. As I'm sure you're aware.

I don't know much about video editors. Film sound editors in movie post-production do not work the way you describe.

> Film sound editors in movie post-production do not work the way you describe.

They don't generally lower fidelity. That is needed for video simply because the data sizes for video footage are so huge they are to work with.

But DAWs do let users "freeze", "consolidate" or otherwise pre-render effects so that everything does not need to be calculated in real-time on the fly.

I'm the original author of a DAW, so I'm more than familiar with what they do :)

Film sound editors do not do what you're describing. They work with typically 600-1000 tracks of audio. They do not lower fidelity, they do not pre-render. Ten years ago, one of the biggest post-production studios in Hollywood used TEN ProTools system to be able to function during this stage of the move production process.

Sure, but that's an extreme use case.

Typical DAW users are using a pro-sumer machine and bouncing tracks as necessary to fit within their CPU limits.

> You generate N channels worth of 1470 samples per frame, and mix (add) them together. Make N large enough, and make the computation processes associated with generating those samples complex enough, and the difference between audio and video is not so different.

Sure, but graphics pipelines don't only touch each pixel once either. :)

> Jacob Collier routinely uses 300-600 tracks in his mostly-vocal overdubs, and so for sections where there's something going on in all tracks (rare), it's more in the range of 400k-900k samples to be dealt with.

Sure, track counts in modern DAW productions are huge. But like you note in practice most tracks are empty most of the time and it's pretty easy to architect a mixer than can optimize for that. There's no reason to iterate over a list of 600 tracks and add 0.0 to the accumulated sample several hundred times.

> If you were actually synthesizing those samples rather than just reading them from disk, the workload could exceed the video workload.

Yes, but my point is that you aren't. Consumers do almost no real-time synthesis, just a little mixing. And producers are quite comfortable freezing tracks when the CPU load gets too high.

I guess the interesting point to focus on is that with music production, most of it is not real-time and interactive. At any point time, the producer's usually only tweaking, recording, or playing a single track or two and it's fairly natural to freeze the other things to lighten the CPU load.

This is somewhat analogous to how game engines bake lighting into static background geometry. They partition the world into things that can change and things that can't and use pipelines approach for each task.

> And then there's the result of missing the audio buffer deadline (CLICK! on every speaker ever made)

Agreed, the failure mode is catastrophic with audio. With video, renderers will simply use as much CPU and GPU as they can and players will max everything out. With audio, you set aside a certain amount of spare CPU as headroom so you never get too close to the wall.

I think the most concrete argument in support of your position is that typical high-performance audio processing doesn't take an external specialized supercomputer-for-your-computer with 12.3 gazillion parallel cores, a Brazil-like network of shiny ducts hotter than a boiler-room pipe and a memory bus with more bits than an Atari Jaguar ad.

> doesn't take an external specialized supercomputer-for-your-computer with 12.3 gazillion parallel cores,

Not anymore at least. :)

In the early days of computer audio production, it was very common to rely on external PCI cards to offload the DSP (digital signal processing) because CPUs at the time couldn't handle it.

> Gamers are actually probably the consumers with the most complex audio pipelines because you're mixing a lot of sound sources in real-time with low latecy, reverb, and other spatial effects.

There's a GDC talk out there about the complexities of audio in Overwatch, a competitive game. Not only do you have invaluable audio cues coming from 12 players at the same time, but also all of their combined abilities and general sound effects, like countdown timers and such.

One of the bigger problems talked about was that in a competitive game you can't just scale the loudness of an opponent's footsteps by the direct distance between them and you. You need to take into account level geometry, and so the game has to raycast in order to determine how loud any single sound effect should sound to every active player in the game.

Edit (link): https://www.youtube.com/watch?v=zF_jcrTCMsA

One more thing to add about Overwatch - with all the attention they give to audio, if your CPU is pegged and the game is trying to render 3d audio, nothing can save it and it drops audio.

The end effect is that on low-powered hardware your audio can end up buzzing. In addition the high costs of dolby atmos processing on the CPU affects the rest of the game, such that turning off dolby atmos is a viable way to get higher FPS at the cost of lower-fidelity audio.

One big difference though is jitter tolerance. To some extent, dropping visual frames will be seen as a reduction in average frame rate. For example, if scroll stutters, people may brush it off as "oh yeah the machine's loaded right now" and forgive and forget. However, the tolerance level of audio glitches is much lower (I think for most, but certainly for me). So the leeway for a computation to bleed over from one frame to the next is far less than for video.

Now 3D Audio is a thing on PS5.

> [video] stuff has just become more and more complex and more and more distant from what is actually happening in hardware

There's been a huge push recently to remove layers from graphics APIs and expose hardware more directly, in the form of Vulkan, DX12, and Metal. The problem that this has uncovered is that graphics hardware is not only very complex but quite varied, so exposing "what is actually happening in hardware" is incompatible with portability and simplicity. As a result you get APIs that are either not portable or hard to use.

> more and more processing is done by video hardware, and less and less by the CPU.

Unfortunately this will remain a reality because it comes with a 20-100x performance advantage.

The "lots of different hardware" side of it is where audio hardware was 20 years ago. Devices with multiple streams in hardware, and without. Devices with builtin processing effect X, and without. Devices that could use huge buffer sizes and ones that could only use small ones. Devices that could use just 2 buffers, and devices that could use many. Devices that supported all sample rates in hardware, devices that only supported one.

Things are not totally cleared up, but the variation has been much reduced, and as a result the design of the driver API and then user-space APIs above that have become more straightforward and more consistent even across platforms.

>20-100x performance advantage.

Yes, for sure this is one aspect of the comparison that really breaks down. Sure, there are audio interfaces (e.g. those from UAD) that have a boatload of audio processing capabilities on board, and offload that DSP to the hardware saves oodles of CPU cycles. But these are unusual, far from the norm, and not widely used. Audio interfaces have mostly converged on a massively less capable hardware design than they had 20 years ago.

Yeah, most audio processing is well within CPU capabilities even with a 100x performance disadvantage, so there's just no need to reach for hardware acceleration and all the complexity that comes with it. Rendering cutting edge graphics, especially on modern 4k 120Hz displays, is well outside CPU capabilities so that 100x boost is unfortunately essential.

Interestingly, for really advanced audio processing like sound propagation in 3D environments or physics-based sound synthesis [1], GPUs will probably be necessary.

[1] https://graphics.stanford.edu/projects/wavesolver/

physically modelled synthesis is no more advanced than anything going on in a modern game engine for video, and in some senses, significantly less.

so to restate something i said just below, the fact the "most audio processing is well within CPU capabilities" is only really saying "audio processing hasn't been pushed in the way that video has because there's been no motivating factor for it". if enough people were actually interested in large scale physically modelled synthesis, nobody would see "audio processing [as] well within CPU capabilities". why aren't there many people who want/do this? Well ... classic chicken and egg. The final market is no doubt smaller than the game market, so I'd probably blame lack of the hardware for preventing the growth of the user community.

ironically, sound propagation in 3D environments is more or less within current CPU capabilities.

I agree that physically modeled synthesis could easily push audio processing requirements far beyond CPU limits and require hardware acceleration similar to graphics. But I disagree that it's a chicken and egg problem. It's a cost/benefit tradeoff and the cost of realtime physically modelled synthesis is higher than its benefit for most applications. Mostly because the alternative of just mixing prerecorded samples works so well, psychoacoustically. (Though I think VR may be the point where this starts to break down; I'd love to try a physically modeled drum set in VR.)

I also doubt that specialized audio hardware could provide a large benefit over a GPU for physically modeled synthesis. Ultimately what you need is FLOPS and GPUs these days are pretty much FLOPS maximizing machines with a few fixed function bits bolted on the side. Swapping the graphics fixed function bits for audio fixed function bits is not going to make a huge difference. Certainly nothing like the difference between CPU and GPU.

I suppose that I broadly agree with most of this. Only caveat would be that you also a very fast roundtrip from where all those FLOPS happen to the audio interface buffer, and my understanding is even today in 2020, current GPUs do not provide that (data has to come back to the CPU, and for understandable reasons, this hasn't been optimized as a low latency path).

Re: physically modelled drums ... watch this (from 2008) !!!

https://vimeo.com/2433260

Consider the sound made right before the 3:00 mark when Randy just wipes his hand over the "surface". You cannot do this with samples :)

It says something about the distance between my vision of the world and reality that even though I thought this was the most exciting thing I saw in 2008, there's almost no trace of it having any effect on music technology down here in 2020.

Very cool video! I agree that it's a little disappointing that stuff like that hasn't taken off.

I would say that it's not impossible to have a low latency path from the GPU to the audio buffers on a modern system. It would take some attention to detail but especially with the new explicit graphics APIs you should be able to do it quite well. In some cases the audio output actually goes through the GPU so in theory you could skip readback, though you probably can't with current APIs.

Prioritization of time critical GPU work has also gotten some attention recently because VR compositors need it. In VR you do notice every missed frame, in the pit of your stomach...

Even with a fast path (one could perhaps the GPU being a DMA master with direct access to the same system memory that the audio driver allocated for audio output) ... there's a perhaps more fundamentak problem with using the GPU for audio "stuff". GPUs have been designed around the typical workloads (notably those associated with games).

The problem is that the workload for audio is significantly different. In a given signal flow, you may have processing stages implemented by the primary application, then the result of that is fed to a plugin which may run on the CPU even in GPU-for-audio world, then through more processing inside the app, then another plugin (which might even run on another device) and then ... and so on and so forth.

The GPU-centric workflow is completely focused on collapsing the entire rendering process onto the GPU, so you basically build up an intermediate representation of what is to be rendered, ship it to the GPU et voila! it appears on the monitor.

It is hard to imagine an audio processing workflow that would ever result in this sort of "ship it all to the GPU-for-audio and magic will happen" model. At least, not in a DAW that runs 3rd party plugins.

I agree it wouldn't make sense unless you could move all audio processing off of the CPU. You couldn't take an existing DAW's CPU plugins and run them on the GPU, but a GPU based DAW could totally support third party plugins with SPIR-V or other GPU IR. You could potentially even still write the plugins in C++. Modern GPUs can even build and dispatch their own command buffers, so you have the flexibility to do practically whatever work you want all on the GPU. The only thing that would absolutely require CPU coordination is communicating with other devices.

GPU development is harder and the debugging tools are atrocious, so it's not without downsides. But the performance would be unbeatable.

If I got my history right, one of the big motivations for physically modeled synthesis in video aka 3d graphics in games was cost. At least that's the argument The Digital Antiquarian makes[1]:

From the standpoint of the people making the games, 3D graphics had another massive advantage: they were also cheaper than the alternative. When DOOM first appeared in December of 1993, the industry was facing a budgetary catch-22 with no obvious solution. ... Even major publishers like Sierra were beginning to post ugly losses on their bottom lines despite their increasing gross revenues.

3D graphics had the potential to fix all that, practically at a stroke. A 3D world is, almost by definition, a collection of interchangeable parts. ... Small wonder that, when the established industry was done marveling at DOOM‘s achievements in terms of gameplay, the thing they kept coming back to over and over was its astronomical profit margins. 3D graphics provided a way to make games make money again.

Not sure if and how this translates to audio.

[1] https://www.filfre.net/2020/06/the-shareware-scene-part-5-na...

3d games were cheaper than Sierra style adventure games because those style games were moving towards lots of voice acting and full-motion-video. The 7th guest came out around the same time as doom, needed two cd-roms and cost $1 million to make (the biggest video game budget at that time). Phantasmagoria came out two years later, had cost $4.5 million and needed seven discs (8 for the Japanese Saturn release). Doom fit on four floppies and wasn't expensive to make (but I can't find a number).

Of course, today's game budgets are huge --- detailed 3d models are expensive, and FMV cutscenes are nearly mandatory. And games have expanded to 10s of gigabytes.

Hardware-wise both audio and video had converged on the exactly same thing: there is this buffer that should be scanned/played out. The fact that contents of this buffer is generated by bunch of specialized hardware for the video case and can be perfectly well computed in software for audio is just question of bandwidth and CPU performance.

That's only half true.

You still can't render an orchestral-size collection of physically modelled instruments on the CPU. A processor that could would be some of analogy for the modern GPU, but for various reasons, there has been little investment in chip design that would move further in this direction.

Games in particular created a large push for specialized GPU architectures, and there's really no equivalent motivation to do the same for audio, in part because there's no standard synthesis approach that things would converge on (the way they have in GPUs).

So, "perfectly well computed in software for audio" is more of a statement about the world as-it-is rather than one about any fundamental difference between the two media.

Is it possible to do audio processing using a GPU?

I don't have enough working knowledge or mathematics to fully comprehend the 3d pipeline but as I understand, it is a huge linear algebra/matrix manipulator and audio processing would fall into this. My reason for asking is that at least in the guitar world I've seen issues related to CPU processing and multiple VST plugins all wired together. Some guitarists use fairly complicated rigs to get their tone and when you have multiple amps, speaker cabinet simulation, and effects running you need a pretty powerful machine to do everything real-time without under-running buffers.

There are specific hardware solutions for this but I've always wondered if you could just create shaders or something and dump the audio stream to the GPU and get back the transformed audio stream with minimal CPU burden. Any insight here would be very helpful to me.

The last time I checked, the problem with GPUs and realtime audio is that although the GPU is extremely powerful and the bandwidth is fairly large, the latency for a roundtrip from/to the CPU is too large for what we typically consider "low latency audio".

Put differently, the GPU is plenty powerful to do the processing you're thinking of, but it can't get the results back to the CPU in the required time.

For video, there is no roundtrip: we send a "program" (and maybe some data) from the CPU to the GPU; the GPU delivers it into the video framebuffer.

For audio, the GPU has no access to the audio buffer used by the audio interface, so the data has to come back from the GPU before it can be delivered to the audio interface.

I would be happy to hear that this has changed.

None of this prevents incredibly powerful offline processing of audio on a GPU, or using the GPU in scenarios where low latency doesn't matter. Your friends with guitars and pedal boards are not one of those scenarios, however.

we have hdmi audio, is that a start? I assume that means the videocard is registering as a sound card as well, no idea if any processing is on the gpu...

100% unrelated. HDMI is an output from your video interface.

but that means the audio stream is at least passing through the video card, no? is it only passing through post mix?

the GPU is not involved in computing the audio stream. the audio side of the HDMI appears to the OS as an audio device, and the CPU generates and delivers the audio data to it. It's essentially a passthrough to the HDMI endpoint.

I would like to have the ability to use a GPU buffer for HDMI audio output. It would enable a couple of interesting applications.

Thank-you! That explains a lot!

I think the latency overhead can be low enough these days (kernel launch / copy data from CPU to GPU, read from global memory, do some compute, write to global memory, copy data back to CPU and wait for completion from the CPU side (which might copy the data to some other buffer) or non-blocking DMA write instead), say on the order of 10-100 us over PCIe, but there is a tradeoff between the units of work that one would give the GPU, the efficiency of the compute (and the working set size of data that you need to load in from global memory to do the compute), and the number of individual kernel launches that one would need to do to produce small pieces of the output. There are some tricks involving atomics (or in CUDA in particular, cooperative thread groups) that could allow for persistent kernels that are always producing data and periodically ingesting commands from the CPU to avoid the CPU constantly needing to tell the GPU to do things to make it easier.

As far as I understand, this information is out of date.

https://www.nvidia.com/en-us/geforce/guides/nvidia-rtx-voice...

https://gpuopen.com/true-audio-next/

Neither applications are designed to run while the GPU maxed out with graphics rendering, are they?

For the unpredictable part of the input, you can't have the best latency and throughput at the same time. If you want realtime in a variable load scenario, you have to cap the GPU usage.

That's how gamers do it when they want the lowest latency possible, anyway. Something like "find the lowest frame rate your game runs on, and cap it to 80% of that".

Both those links describe audio processing on a GPU. Neither of them address (from a quick skim) the roundtrip issue that occurs when doing low latency realtime audio.

> Is it possible to do audio processing using a GPU?

This thread fascinates me. A sibling comment addresses why it can't be done in realtime (barring some pedestrian-sounding hardware improvements), but the question of "how" fascinates me. I know a little bit about audio, I'm a classical musician with some experience with big halls, I know some physics, and I wrote a raytracer in JS back in the early aughts. Can we raytrace sound? It's a wave, it's got a spectrum, different materials reflect, refract, and absorb various parts of the spectrum differently... sounds pretty similar! The main differences I see is that we usually only have a handful of "single pixel cameras" (I expect to lose a bit of parallelism to that) and more problematic, we can't treat the speed of sound as infinite. I can imagine a cool hack to preprocess rays for a static scene, but my imagined approach breaks down entirely if the mic moves erratically.

I think it's too pessimistic to say it can't be done in real time on current GPUs. I think it should be possible to do reliable low latency readback with modern explicit graphics APIs. The harder problem is scheduling the audio work at high priority so it always finishes before the deadline, but this too has improved recently because VR compositors also have strict deadlines they must meet.

I remember hearing that Nvidia were working on an audio spatialiser using their RTX hardware/expertise, but I can't recall hearing about it after 2018 or so, which is where the project I was working on that I wanted audio spatialisation for ended.

There are a number of clever audio processing algorithms implemented on GPUs already. But that's not the issue at hand, which is doing so in a low-latency realtime context. GPUs are insanely powerful, and if you're just doing playback of material from disk, the tiny bit of extra latency they may cause isn't a problem, so go for whatever they can do. But if you're using the computer for synthesis or FX processing, you need better than GPUs can offer in terms of latency (for now).

https://on-demand.gputechconf.com/gtc/2014/presentations/S45...

We can already do this (without GPUs, and quite likely on them). It is done, for example, by a certain class of reverb algorithm which more or less literally "raytraces" the sound in room. It's not the most efficient way to implement a reverb (convolution algorithms are better for this). But it does allow for a large degree of control that convolution does not.

In addition, you may also consider physical modelling synthesis to be an example of this. Look up Pianoteq. It is a physical modelled piano that literally does all the math for the sound from a hammer of particular hardness striking a string of given size and tension at a particular speed, and then that sound generating resonances within the body of the piano and other strings, and then out into the room and interacting with microphone placement, lid opening etc. etc.

Most (not all) people agree that Pianoteq is the best available synthetic piano. It's all math, no samples.

There are similar physical modelling synth engines for many other kinds of sounds, notably drums. They are not particular popular because they use more CPU than samples and can be complicated to tweak. They also really benefit from a more sophisticated control surface/instrument than most people have access to (i.e. playing a physically modelled drum from a MIDI keyboard doesn't really allow you to get into the nuances of the drum).

Yes, sound can be raytraced - in realtime on current GPUs even. Using a ray model for sound waves is called Geometric Acoustics and works reasonably well in most cases, although it has a few shortcomings, mostly when dealing with low frequencies (room modes, diffraction).

The Oculus Audio SDK contains an acoustic ray tracer for that. See https://www.oculus.com/blog/simulating-dynamic-soundscapes-a... for a short writeup on what is acutually in there under the hood.

Disclaimer: I've worked on that ray tracing engine at FRL for a while.

I wonder if an FPGA board could be used for that.

Could you be a bit more tangible about how video developers have have just made things more complex? I don't necessarily disagree, since I can't wrap my mine around modern video codecs easily (in concept sure, but they're harder to use in practice), but that might be just because I'm ignorant?

I think Ralph (Levien) did a fine job of that in the original article, albeit primarily focused on the issues surrounding compositing.

I wasn't talking about codecs, but about how software interacts with the hardware. We're already dealing with the move away from OpenGL, which for a while had looked as if it might actually finally wrap most video in a consistent, portable model. Metal and Vulkan are moderately consistent with each other, but less so with the existing OpenGL-inspired codebase(s).

But there's even more simplistic levels than this: in audio, on every (desktop) platform, it is trivial for audio I/O to be tightly coupled to the hardware "refresh" cycle. WDM-WASAPI/CoreAudio/ALSA all make this easy, if you need it (and for low latency audio, you do). Doing this for graphics/video is an insanely complex task. I know of no major GUI toolkit that even exports the notion of sync-to-vblank let alone does so in any actually portable way (Qt and GTK have both made a few steps in this direction, but if you dive in deep, you find that its a fairly weak connection compared to the way things work in audio).

Reply on news.ycombinator.com