Four common mistakes in audio development
atastypixel.com
atastypixel.com
Realtime forces people write less, writing only what actually matters, and all this in my opinion helps people to be better engineers (in-a-way). I just wish it looked more attractable for people - hacking your way out in time/resources constrained systems (like MCU's for example) can be as fun as hacking html/js to make your site behave as you want.
I spent a few years working in the games industry, and let me tell you, enterprise-like tens of thousands of lines of code files with massive OOP hierarchies were all too common. That's why I think Unity3D is fantastic -- they do the OO (composition) in their engine libs, you write simple scripts.
In my experience, what really forces people to write more code "that matters" is programming languages and frameworks that constrain them.
Unity may have its detractors and genuine deficiencies but the Unity projects I worked on were much more maintainable due to the constraints of working inside Unity itself.
In what languages have you seen a big code base that you liked? Just curious.
As a game developer who mostly worked in Unity, this is horrifying. Unity-based projects are pile of horseshit in terms of architecture and organization, and I honestly hoped that AAA stuff is held to a higher standard.
It may have changed a bit now that hardware speed doesn't increase that much anymore, but historically, that even applied to game series. Even if you knew there would be a sequel, that sequel would target vastly improved hardware, which means new graphics effects, higher-resolution textures, etc.
That is not an environment that leads to high-quality code. The ability to update a game over the Internet hasn't improved that situation, to state it mildly.
I remember when we'd take UE3 drops, it's take one engineer a whole month of compiler errors + fixing stuff to get it back to where we started with.
https://cs.chromium.org/chromium/src/?sq=package:chromium&ty...
So, while these code bases are wonderfully informative and good to learn from in many ways, they won't teach you "modern" C++ styles. I put "modern" in quotes because there's nothing wrong with choosing a style that doesn't use the latest whiz-bang features.
It's a fine guide, but it was written for google and has some stuff in there as a result of that (no exceptions being on example).
While I can understand the ideas behind it, exceptions do cause head aches, I find it really weird. Enforcing this means using a very limited part of the C++ standard library, e.g. no std::vector, std::map, etc, because all those can throw exceptions. If you say no to exceptions but yes to std::vector then you either need to write code that works in the presence of exceptions (which means you could just say yes to them) or you have shoved your head in the sand like an ostrich thinking if you can't see them they can't happen.
https://github.com/zeta0134/pikmin-nds
The NDS is an odd little system. Despite supporting 3D graphics, it entirely lacks useful things like a floating point unit, or a hardware divide. All of your math ends up being done in 12.20bit fixed point. Square roots are already slow on normal systems, and they're just ridiculously slow on this thing, plus you've only got something like 550k hardware cycles per frame to get everything done.
Getting over 100 objects to draw, have basic physics applied to them, and perform AI in a reasonable amount of time has led to some eye opening revelations. Right at the start of the project I spent ages trying to come up with the fastest way to sort all of my objects back to front, to make the renderer work. I was just sure this was going to be the slowest part of the draw, so I spent probably two weeks on sorting implementations alone before finally giving up and choosing a std::priority_queue just to have something that worked. Then I actually tested the thing, and came to the (obvious?) conclusion that I had wasted a great deal of time. The std::priority_queue was handling 100 or so objects really really fast, and the system was lagging badly because the drawing itself was poorly optimized and needed to be revisited. I had to start actually profiling my code, and the recurring theme was that the bottlenecks were almost never where I thought they would be, and usually surprised me.
It's a whole ton of fun, I'll say that much. Not sure if I'll end up doing anything useful with the engine once it's done (obviously I cannot do anything meaningful with a Pikmin clone due to IP reasons) but I'm enjoying just being able to push the limits of the system.
A huge downside with this technique is that color changes during H-blank (waiting period between scanlines) and more critically V-blank (rather long waiting period after each full screen draw) cannot be seen. When I was using this technique, I had the engine actually wait until all of V-blank had passed before it started doing the timing colors. This still ended up being pretty hard to read of course, as the colors would change flicker rapidly at 60FPS, couldn't be easily saved to view later, and were complicated significantly by the multi-pass nature of the engine, where each visible frame is actually composed of several hardware frames, all with totally different timings. So I scrapped that method almost immediately and wrote text-based profiling code that used the built in hardware timers.
Emulators are generally pretty accurate at CPU timings, but other hardware timings are lacking, especially with the amount of time it takes the GPU to process certain operations. (DesMuMe is my usual emulator for quick testing, no$gba is the most hardware accurate and has an excellent debugger, but it still falls short of real hardware testing for timings.)
The NDS does have hardware timers, so I wrote a quick and dirty set of profiling functions. I can issue a start and stop keyed on a topic, and it will measure the number of cycles between the calls. So, any bit of the engine I wish to time, I just surround with matching start/stop calls using the same topic, and then print the results of all the topics out to a text console on the bottom screen. (If you're running the game, hit Select a few times to pull this up. Go withdraw a full squad of Pikmin and see the numbers change and the framerate drop.) It's crude but effective, and keeping in mind the 550k cycle limit per frame allows me to get a rough estimate of what percentage of my frame time is taken up by a particular routine.
I've generally found that my physics and AI calculations are proportionally slower on real hardware than the emulator reports, by a small margin. I suspect this is due to the ARM9's weird TCM cache implementation, but I really don't know what's going on. Graphics calculations are way different though, and are usually reported much faster by the emulators, so I always re-profile any graphics "optimizations" on real hardware to make sure I haven't run into something that the emulator just happens to run fast.
Some realtime is truly restrictive. Realtime audio on a GHz-class processor is not.
If you're not experiencing performance issues every day despite your smartphone being "ghz class", I want your phone!
> although there is a high horse present in this article, consider me standing beside it pointing at it, rather than sitting on top of it.
Something we can all aspire to!
"What," say you, "are you giving me advice? Indeed, have you already advised yourself, already corrected your own faults? Is this the reason why you have leisure to reform other men?" No, I am not so shameless as to undertake to cure my fellow-men when I am ill myself. I am, however, discussing with you troubles which concern us both, and sharing the remedy with you, just as if we were lying ill in the same hospital. Listen to me, therefore, as you would if I were talking to myself. I am admitting you to my inmost thoughts, and am having it out with myself, merely making use of you as my pretext.
Good point. A major misconception that a lot of people have is that lookup tables are always faster than actually calculating a value, but cache locality is usually not given as much attention.
It should still hold true today for small embedded processors with slow arithmetic and no cache.
In my experience, writing everything in C/C++ has its own host of problems. What's worse, a GC stutter or a segfault? :)
Besides, actually writing the DSP code makes up a comparatively tiny portion of the development time. Tuning and tweaking the DSP algorithms takes far more, and designing the user interface dwarfs both. I know multiple audio products which use Lua for their UI layer because the productivity increase is so significant. And there, you don't have to worry about performance to anywhere the degree you need to in the DSP.
What informs your guess that it's "around java speed" ?
As to slow scripting languages - TFA was talking about determinism. A slowish scripting language could give you deterministic runtime, at least as deterministic as C or assembly (interpreted Forth is a simple example), and on the other hand there's no shortage of ways to get your C or C++ code to stall for an unknown amount of cycles (TFA enumerated some.) Of course slower languages are slower, and in a faster one you're more likely to get away with stalls because you have more slack to begin with.
You can't control the determinism of the OS or user's environment, sure. But you can make sure you're not adding additional indeterminism. If the OS can't deliver stable timing, no audio app will under those circumstances. But for the times when it does, you don't want to be the only instrument popping and underrunning, while every other effect is glitch-free.
Raw speed isn't really that great, because you can't process ahead of the present. If you need to provide a 128 sample buffer every 2.9 ms, it doesn't matter if a slow language takes 2.8ms, and a fast one takes 0.1ms. However, if every 10 seconds the fast language takes 10ms, you've lost 4 buffers! Consistent speed is the aim.
It all depends on how much latency you can accept. With enough latency, you can bufferize over anything.
A large problem for deterministic timing are the cache hierarchies of modern CPUs.
Whoa, slow down. Testing will tell you which one is faster, but if you're applying a complicated function to every sample (eg. one that uses pow, log or trig especially more than once), lookup tables are definitely to be considered. The more complex the function, the more of a performance gain it will represent. An arithmetic approximation such as Chebyshev approximation can also work, but may or may not be more efficient depending on the function.
And yes, make sure you optimize the size of your table to be as small as possible while maintaining acceptable quality (which also depends on the interpolation you're using to access it). Everything is a tradeoff. New processors are efficient but not magical, and I've seen a marked performance improvement after applying a lookup table on a small plugin a few months ago, all of this on a very modern i7.
edit: it might be different for a single plugin, where there isn't much need for cache, but as the core loop grows bigger, you also need the cache more.
An unfortunate suggestion, given the context. :)
GP and others always mention using C/C++, but what about Rust? Is it viable [enough] for these kind of problems? For high-performance code in game engines etc.?
Edit: Ah I should've searched: https://news.ycombinator.com/item?id=11908849
Ok, everything else made sense, but why on earth would you use function pointers in performance critical code if you can avoid them at all. Same obviously goes for virtual methods.
edit: i've never used a virtual method, but my impression is that performance-wise it's quite different to pointers to simple functions. most of my audio code is pod-structs to layout the data in the order that it gets acessed and standalone functions that then modify stuff in the structs. it's pretty barebones data oriented programming compared to the other extreme, some baroque meta virtualised oop architecture.
Check JIT libraries (like GNU lightning) and LLVM. Also LuaJIT might be interesting.
"llvmpipe", software rasterizer that uses LLVM for runtime codegen for shaders: http://www.mesa3d.org/llvmpipe.html
Having "a mix of pod structs and function pointers" is analogous to having an interpreter for some scripting language. Do you consider interpreted scripting languages fast?
Interpreters are slow pretty much for that reason. Having to chase pointers.
Runtime code generating JITs wipe floor with interpreters, being usually 10-100x faster.
Now I'm not saying your application will be 10-100x faster, but done right, it'll absolutely be faster if your performance critical "inner loop" / filter chain needs to currently follow pointers. It's just a question of how much and whether it matters for the use case.
Yes, it might be similar at the core, but i would guess the usual audio pod-struct/function mix is many times less complex than a full jit language interpreter.
What does chasing pointer mean to slowness then? My understanding is that its similar to data pointers, that a certain region of memory needs to be looked up and be fed into the instruction cache. If the process is layed out so that instruction cache misses are minimised, whats so bad about it?
Other than executing a lot of useless instructions, causing pipeline stalls due to indirection, potentially mispredicted branches, etc. I guess nothing.
Just remember if the actual operation is just a few instructions, the price for all that extra work can be 1-2 orders of magnitude.
If you only need to chase pointers say a million times per second or less, go for it. It won't matter so much. In worst cases you might lose 10% (of total cpu core capacity) performance, if even that. If it's in tens of millions or more, it's pretty likely to be a significant bottleneck.
What does your profiler say? What kind of ILP you're getting?
edit: since you were mentioning jit optimisation and code generation - this is a field i do not have any experience, but my impression so far is that this runtime re-arrangement of code can yield quite a big performance increase for higher level languages (jvm does this afaik). Is the link you provided the same idea for statically compiled c/c++ code?
Sometimes you do need to use function pointers. It's just if it's in a performance critical path, you might want to consider alternatives.
There's no silver bullet how to structure such flexible, dynamically configurable high performance system.
I'm pretty surprised if your deadline is really that 23 microseconds (60 fps / 7000). You can't even reliably dispatch an IRQ request in that time!
> I wouldn't call that similar
If you have a long audio filter chain it's not all that different from a long fragment shader. You can think of a fragment shader as a filter chain.
> Since you can't provide any advise for an alternative to function pointers
Uh. That's pretty hard without seeing the source code and full problem description. Especially because you seem to be expecting some magical solution. Well, those don't exist, it's just hard work.
Besides, like I said earlier, sometimes function pointers are appropriate for the task. Just understand the cost. Do keep in mind audio filter chains can be pretty complicated. If your filter chain has many concurrent channels with hundreds of dynamic filter steps for each sample to pass through, yeah, you might have some performance problems.
> i must challenge your expertise in audio programming since you seem to project your view from a graphics programming point.
Be my guest. Just wanted to share something I know about and give new ideas. Btw, I'm not doing graphics, but mostly I/O with micro/millisecond level deadlines.
Your comparision to graphics seem to relate to long FIR filters, which i dont't use because of cache reasons (big kernel/lookup table). FIR filters can however sound pretty nice with static parameters but get extremely expensive for realtime modulation. For realtime modulation i suggest IIR filters (Finite Impulse Response vs Infinite Impulse Response filter architecture).
Of course it's hard to give advice into the blue without any source, but i atleast expected some general hint, like 'avoid branches, use function pointers' on a next-level, since you were so opposed to function pointers to begin with.
I am also not completely sure what's the best approach in this regard, and i surely did not enough profiling in this regard.
That said, i remain sceptical regarding your critique of function pointers in a single threaded audio application, because of a lack of an alternaltive.
I/O on a microsecond critical deadline certainly seems to be in a neigbourhood to these problems. Do you use some arcane javascript asm.js with some jit auto-reconfiguring optimising for this, or how would you describe your approach?
1 / (60 Hz * 700) = 23.8 microseconds. (Yeah, should have rounded up to 24).
What worked for me in the past was just concatenating (well, memcpying) executable filter blocks of x86 code while ensuring parameters were in the right registers.
Loop start block -> filter 1 -> filter 2 -> filter 3 ... -> loop end block.
I made these blocks by processing specifically compiled (not at runtime, but ahead of time) functions, stripping prologues and epilogues. There's more to it, but it's been a while.
Crude, but effective and fast even when each filter step was pretty small on its own.
About the 1/(60 * 700) what does that number even mean to you? 1/60 is the usual refresh rate in seconds, about 16 ms, why do you multiply this with 700, this makes no sense?
> Graphics in a minimum 60 fps context is about 700 times less time sensitive.
16.6 ms (60 fps) / 700 = ~24 microseconds.
> So you put in the parameters into registers 'per hand' in assembly and then memcopyed some function to the right adress (using a function pointer?) to process these registers? Wow. That sounds pretty complicated. For now i just use gcc and function pointers and let gcc do the rest.
I concatenated blocks of compiled code together (compiled as position independent), avoiding branches and indirect calls.
Loop setup block took care of putting right parameters to right registers.
Hardest part was getting the filter parameter passing right. Dynamic configuration was trivial after that.
There were no function pointers except one to trigger the top level loop.
For your concatenated blocks of precompiled code maybe you should have used simple function pointers, and one monolithic precompiled block to achieve the same result like i originally suggested.
That's how the original version was like, but it was way too slow. I achieved eventually about 10x faster performance for typical (dynamic) configurations.
That is, my laptop has ~43 cycles to get out all channels of each pixel, vs. more than 60,000 cycles to get out all channels of each 44.1kHz audio sample.
But the pixels only have to be delivered by screen refresh, and the samples only have to be delivered before the sound card tries to play them--probably 64-1024 samples (or further) in the future.
Here's a test I did on the Raspberry Pi 3 (with source so anyone can repeat it on more Intel-y hardware): https://gist.github.com/LnxPrgr3/31eaf5648a9956f5c576f70d876...
Calls through a function pointer are technically more expensive, but it's a factor of ~1.1, not 10-100.
Interpreters are slow because they chase pointers and otherwise branch in a manner the CPU can't predict. Calling the same function through the same pointer over and over again isn't that.
C/C++ compiler abstracts the difference between inlining and actual call. For best results, cross compilation unit optimizations should be enabled so that inlining can occur across object and library boundaries.
> Interpreters are slow because they chase pointers and otherwise branch in a manner the CPU can't predict. Calling the same function through the same pointer over and over again isn't that.
A lot of function pointer usage is also data dependent. Data dependency and too complicated dependency chains are what breaks prediction.
Breaking inlining is probably the real cost to this. Still, we're talking ~9 cycles per indirect call on this hardware. If you call each callback per buffer instead of per sample (which everyone seems to do), you're down to an amortized cost below 1 cycle per sample for a filter chain 14 deep with only 128 sample buffers. 14 filters probably swamp that cost.
Edit: A good compiler will completely remove my trivial function if it's inlined into call.cc's code. It has no side effects, and neither does the loop calling it 100,000,000 times. The resulting timings would be meaningless: doing nothing is cheap!
In 9 cycles, you could execute up to 18 SIMD instructions, each operating on 256 bits worth of data.
For scalar, 9 cycles can be up to 30 instructions or so. For good code, average about 15-20.
> If you call each callback per buffer instead of per sample (which everyone seems to do), you're down to an amortized cost below 1 cycle per sample
Function pointers are perfectly ok even in high performance programming, if it's not what burns the cycles. If you can batch the filter, all is good. If you can't, well, you might need to think up some other solution.
Profiling rocks. I did simply some experimental cases where I ran same test setup with dynamic function pointers and hardcoded code. I quickly noticed that hardcoded version was over an order of magnitude faster. I figured out a way to narrow the difference and ended up doing dynamic code generation.
> doing nothing is cheap!
Code you don't need to run is the fastest code you can have. Infinitely fast in fact.
https://github.com/jvoorhis/siren
I wouldn't expect this to run on the latest LLVM without a little hand holding (the most painful part was porting it to run on x86_64 after developing it on my white MacBook!) But it worked rather well and was a fun platform for experimenting with composition, synthesis and compilers!
Desktop CPUs can predict function pointer jumps too, you know.
This is true on the CPU side (for some CPUs) but what about compiler optimizations?
Using "volatile" should make stores and loads happen where they are written in the code, but can it be relied on in multi threaded use cases? It's generally frowned upon for a good reason, but perhaps it's acceptable if you're targetting a limited range of CPUs (the article seems to be focused on iOS and ARM only).
A safer bet would be to use __atomic_load and __atomic_store (from [0] or [1]) or C11 atomics if you have the appropriate headers for your platform. They provide safe loads and stores for all CPU architectures, and provide the appropriate memory barriers for cache coherency (for architectures that do care).
[0] https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins... [1] http://clang.llvm.org/docs/LanguageExtensions.html#langext-c...
To give a concrete example (but artificial) what might happen, consider this case:
global_flag = 0;
for(this loop takes a long time) { ... }
global_flag = 1;
for(this loop takes a long time) { ... }
global_flag = 0;
As far as I know, using atomic load and store is the only way (apart from locks) to guarantee correctness and portability.
my_buffer->something = 42;
global_pointer = my_buffer;
global_flag = 1;
my_buffer->something = 42;
write_barrier(); // Not the actual function; will vary depending on your environment.
global_pointer = my_buffer;
write_barrier(); // Not the actual function; will vary depending on your environment.
global_flag = 1;
It's simpler to use atomic loads and stores or locks, since their implementation already has the required barriers with all the details (quick: what's the difference between an acquire and a release atomic access?).
(Note that this is different in Java, where `volatile` always implies a memory barrier.)
ACQUIRE prevents reordering any loads and stores from after the barrier to before. RELEASE is the opposite, no load or store before it may happen after the barrier.
ACQUIRE is what you use when locking a spinlock, RELEASE is when you unlock.
Put these incorrectly and you're not guaranteed to have all loads and stores happening when the spinlock is locked, and your program is no longer guaranteed to behave as expected.
The memory model implemented by c compilers absolutely do not allow this. The problem is not cache coherency but optimization. The compiler assumes it understands the visibility of the variables and reorders, combines, and eliminates reads and writes. C is not a high-level assembler.
> A safer bet would be to use __atomic_load ... C11 atomics
These should work but be careful. Use the semantics defined by these constructs not the semantics of any assembly you might imagine the compiler generating.
Safer and an order of magnitude slower.
It's also not very easy to predict the relative difference because it depends on which core(s) the threads get scheduled on. If both threads happen to get on the same core and the data stays in caches, we're talking about 1ns (L1) to 5ns (L2). An atomic load/store would be about 20ns (4..20x slower).
If they happen to get scheduled on different cores, the CPU interconnect will have to deal with its cache coherency protocol (similar to atomic load/store) and the result will be around the same.
We're talking about pretty miniscule amount of time here. It's a choice between fast if you're lucky but possibly incorrect anyway or marginally slower but guaranteed correctness.
In other words, isn't this a non issue when comparing the performance of atomic types and "regular" types, as they both would require this?
Also, do the atomic types handle ordering?
I could be going crazy, but I remember seeing some of those atomic types simply being typedef'd to "int" in some OS code before.
EDIT: I was thinking of the "atomic_t, atomic_inc, atomic_set, etc..." in the Linux Kernel. It guarantees atomic behavior, but not ordering. Memory barriers are still required when using it between different threads. The C11 types appear to be different, as they accept an ordering argument in their load/store functions.
No-one inspects their binaries after every recompilation or compiler (flag) change.
Really? I was under the impression that "volatile" basically did nothing other than prevent the compiler from optimizing out a variable that appears to not ever be written to, but is actually written to via an ISR or something.
Also, how does the compiler usually handle global variables? Since they can be modified by some code that was linked in, does it assume them to be volatile?
Of course, this issue is slightly different than the atomic-ness of certain data types. We're talking more about situations where you have a series of atomic operations which you expect to happen in the order of which they're written.
I think it also stops loads and stores to be reordered with other volatile loads and stores, and yes, it indeed is typically useful for memory mapped i/o, interrupt handlers and such. But most of the time it's not the right thing to do.
> Also, how does the compiler usually handle global variables? Since they can be modified by some code that was linked in, does it assume them to be volatile?
No, they're not considered volatile but may have stricter guarantees with reordering than local variables (esp. when combined with calls to foreign functions).
If multithreaded code needs to be correct and portable, using locks or atomics is the way to go.
It only prevents reordering by the compiler. CPU can still reorder them as much as it pleases, volatile doesn't affect it at all.
Of course you're pretty safe on x86. Not so on other platforms. Typically you find this kind of issues when porting x86 code to some other platform. X86 writes are always in order with other writes. Loads are always in order to other loads. Just need to remember loads are sometimes ordered before stores.
Again, volatile does not prevent CPU from reordering stores and loads.
Those architectures that do this typically provide specialized always serialized instructions for communicating with memory mapped devices. If not, you need to have proper fences to ensure right order.
X86 stores are always in order (except for some specific instructions, but they're no concern in this context), so it doesn't need this.
Don't. Use. Volatile.
It's meant for reading from hardware registers and nothing else.
There's nothing to prevent the re-ordering of reads/code around your volatile block. This is doubly nasty in that Windows will insert a memory fence which makes everything look like you'd want it. Soon as you port to another platform, boom all sorts of concurrency issues.
Seriously, use atomics. That's what they're built for.
Windows is not inserting any memory fences anywhere in your code. MSVC cl compiler will also only do so when you specifically ask it to do so by using an intrinsic.
> /volatile:ms
> Selects Microsoft extended volatile semantics, which add memory ordering guarantees beyond the ISO-standard C++ language. Acquire/release semantics are guaranteed on volatile accesses. However, this option also forces the compiler to generate hardware memory barriers, which might add significant overhead on ARM and other weak memory-ordering architectures. If the compiler targets any platform except ARM, this is default interpretation of volatile.
From https://msdn.microsoft.com/en-us/library/12a04hfd.aspx
Really, just don't use volatile.
I haven't seen it doing that on any of my code using volatile, mostly for CAS and fetch-and-add (lock xadd).
I was an early adopter of digital vinyl (if you haven't seen it before, the records have a screechy signal that software can use to determine the position of the needle on the record, which it then maps to an audio file).
A friend of mine purchased the most popular unit at the time but it was really unreliable. It crashed once while a club full of people were dancing and that was the end of it for me. I switched to a unit by a small company (Serato) that had just been released (2004) and never looked back. The unit itself still works perfectly, there was a bug back in about 2008 that they tracked down and patched for me.
Apparently the original brand has now caught up technology-wise and they're a big player, but I will never, ever, ever buy their kit. Reliability issues with audio gear can completely destroy your trust.
I guess one advantage is that it looks like they're playing the actual records. It had me fooled for a minute.
At the time neither the tech nor acceptance were there for the modern tools. While increasingly there were CDJs around, they didn't feel the same for people who did any scratching.
Also, turntables were everywhere you went, so you could take everything you needed in your bag (laptop, 2 x records, little conversion box) and you could just plug into an existing setup, anywhere, without issue.
Digital vinyl bridged the gap as people started moving over. Also, at the time the software only worked with records and cds. Now you have a huge selection of controllers to choose from.
I'm actually about to sell all my stuff to buy a "controller" (something like https://www.google.co.uk/search?q=reloop+terminal+mix+8&tbm=...) so that I have something convenient and always ready to go.
For a low-budget hacker solution, you can just obtain the time coded vinyls, use any reasonable USB audio interface with a RIAA preamp and use xwax (open source) or Mixxx (a GUI frontend to xwax, etc) that runs fine on Linux.
You use the digital vinyl because it feels identical to real vinyl. Those dinky little jog wheels on DJ controllers are for the pretenders, they lack the precision and weight to be much use for scratching, and are considerably more finicky for beatmatching; Serato is/was a digital vinyl system first. Those that learned to spin on turntables tend to swear by them; there is a certain tactile aspect that is ahrd to get elsewhere. (Serato has since pivoted to controllers, since CDJs seem to have taken over the club scene)
I publish an open source DVS called http://xwax.org/. I had the same goals as you describe; it had to be 100% stable before it did anything else. Hopefully it serves as a reasonable example of some of the advice in the article.
Seriously, check it out, it's awesome. Write your audio DSP in a language suited to it, compile to efficient C++ (among other languages) and optionally embed it in a (huge) variety of environments. (even Javascript for webaudio)
It's a functional description language that describes a DSP graph, and can generate block diagrams too. Not to mention, it has libraries containing any filter you can imagine. Highly recommended.
Another key point is that FAUST doesn't let you manipulate individual samples or time, but instead, focuses on the dataflow, which means you don't have to start all your programs by a time loop ; for example, a simple 2-input mixer is just "process = +;". It's a huge improvement over "for(t=0;t < n;t++) out[t] = in1[t] + in1[t];".
Still, nice idea.
Even though it's a concept from realtime computing, I though it would be widespread in general-purpose OS'es as well. Really, it seems like a useful feature for any OS that implements priority at all.
What would be the downsides of having it in a general-purpose OS?
The ones I can think of are development cost and processing overhead.
Rust, on the other hand, I am very keen to explore, especially with some of the SIMD work that's been brewing. Somebody made bindings for the VST2.4 API/SDK already, and I've been mulling over putting together a brief proof-of-concept with it: https://github.com/overdrivenpotato/rust-vst2
Rust at a high level could be suitable, when well-tested libraries exist for it.
I think people end up overselling the difficulty here, because people often strike me as speaking as if the 10ms maximum GC delay is actually the minimum or something and as if the GC is running uncontrollably keyed by a random number generator or something, rather than the amount of garbage generated. It's not that hard to imagine an audio application that runs a 50us GC every several minutes or even less often if you preallocate everything. In practice it's not entirely dissimilar to writing in C and avoiding malloc.
But the real level of difficulty is still something that should make you think, especially as you move from "a single filter" to "a full synthesizer suite".
Rust would generally be a better language, as long as you can step up to the somewhat-harder language, and can find the libraries you need. The payoff for the somewhat-harder language is that you'll have much better assurances about many of the relevant issues. It's what I would generally choose for this task, again, given libraries. But if someone did want to use Go because it's a bit simpler of a language, I wouldn't immediately breath flames on them, unless they were being obviously too ambitious.
I am very interested in this. Do you have any links with more info?
In many ways, Go is like a scripting language, but it does let you do some systems-like things in a way that Python or Perl or the other similar languages completely don't.
I haven't had the time to work on it in a while but it works: https://github.com/pac-dev/protoplug
DISCLAIMER - I work for the company that makes Audio Weaver.
All you would need here is 2 (or more) copies of the shared data structure and a few pointers to the data structure. You fill in a draft version of the data structure and only change the "read" pointer to point to it when it's ready. Changing that pointer is, I would hope, atomic. You can then change the "write" pointer to point to a different copy of the data structure to work on.
To make sure the client only reads consistent data, it can make a copy of that pointer before it starts processing, because the pointer itself might change.
Using 3 buffers instead of 2, and if you're sure the processing of a buffer takes less time than the switching cycle time, you can be sure your buffer data will not ever change in the meantime.
In graphics programming, if you are not done with a frame when it would be the time to show it, you can just delay it and it won't cause much problems.
In audio, if you are not done processing the data when it should be playing, you WILL hear a glitch. Is double buffering really that useful then? Can't you just maintain a write pointer into a circular buffer that is always ahead of the read pointer?
So what happened was that latency was utterly unpredictable, and we had buffers that went dry (starving the output DAC) or filled up anyway.
I'd walked into the project a couple years earlier, and wound up spending six cumulative months getting a handle on all the problems, which were distributed amongst several different code bases, and ultimately involved fixing bugs in every single component, including a really nasty one in the OS hypervisor. It was quite a ride.
Isochronous audio is hell. Sure, y'all are smarter than the average bear, but . . . isochronous audio is hell, and you're gonna need rollerskates. :-)
What does that even mean?
And it's already assumed but he article; it mentions it in terms of "the system has to deliver n seconds of audio data every n seconds to the audio hardware. If it doesn’t, the buffer runs dry". The article is focused on generating the content that has to go in these buffers.
Because the problem with just increasing buffering is that it adds more latency. In your triple-buffered video game example, the action is now an extra frame behind the double-buffered case.
Hence the focus of audio folks reducing buffers to the minimum; for any audio application that's acting on data from the real world.
For example a simple live audio processor, or musical instrument.
Small delays of milliseconds seem insignificant but they create various artefacts and effects when that software is used alongside others or in a 'live' scenario.
But you are rightly getting at something that is a problem -- that the knowledge of using large buffers for audio seems to have got forgotten in some software; there seems to be a modern assumption that in order to get stable audio without glitches, that realtime threading is necessary and tiny buffers for trivial playback tools. Forgetting that systems did this a long time ago with nice large ring buffers, and the ability to flush or re-fill the buffer.
The operation is still atomic. You will not get a partially written word size value.
C11 and C++11 atomics also guarantee ordering depending on the memory model parameter.
Look up Sutter's talk titled "Atomic Weapons" for more detail.
Nothing wrong with using Swift (the language) to render audio in realtime on the audio thread. I'd change the advice above to: don't send messages to Objective-C objects during your audio thread.
You don't want any allocation or deallocation when you need realtime performance.
If you don't create objects or cause objects to become unreachable, you also won't be creating garbage. So you won't have GC pauses. Unless the GC system is braindead and runs even when there's no heap pressure.
As a side note, I sure wish browsers would hurry up and implement web audio workers so that this could be true for me!
Why do we have a web browser at all and not just a way of running arbitrary code from a network in a sandbox? We'll end up re-imagining the browser as an OS with workers for anything an OS could do anyway in the end if we continue to need to have the browser act as an application runner.
Well, if you try, you wind up with something that looks an awful lot like a web browser - especially once WebAssembly is a thing. The API that many desktop operating systems provide is not designed for the security model that you're looking for, so you wind up building a new one on top of it - see WinRT when Microsoft needed a sandbox.
We used to have things like Java Web Start (still do in some enterprise systems), and, well, it's not exactly any better than a web browser except that it has a better view layer for applications than HTML/CSS. It's also not supported on mobile platforms.
We do, that is how mainframes work.
It is also available in Mac OS X, Windows and mobile OSes, one just needs to make use of the respective APIs.
As for the GNU/Linux and *BSD I am out of touch how good the current sandbox support for cnames, Lx and others actually is.
Having `AudioContext` in a worker does not get you a lot. It's planned, but not prioritized very high. What happens when you're calling methods in JavaScript on an `AudioContext` is that we simply validate the input and queue messages to the audio thread, so this is effectively exactly what the article is talking about. We use some locking, but we've carefully wrote the code so that it's not really an issue. We lock just to swap two pointers in a message queue. This could certainly use atomics, but we have more important things to do first.
That said, the interesting thing developers need and want is `AudioWorklets` [0], and that's what we at the W3C Audio WG is working on more or less full-time, these days. It used to be called `AudioWorker`, but got renamed for a lot of reasons [1].
[0]: https://github.com/WebAudio/web-audio-api/wiki/AudioWorklet-... [1]: https://github.com/WebAudio/web-audio-api/issues/532
1. Does what you said include ScriptProcessorNodes?
2. Is this true of all browsers per specification, or true of a particular browser's implementation? My experience with web audio has been that in Chrome things work as you say, but in FF I could never find a webaudio library that played back without heavy clicking, so I supposed audio workers might help.
3. Given what you said, what will be different about Worklets? I scanned the links but I guess I don't follow the bigger picture.
2. It's true for all browsers. ScriptProcessor runs on the main thread and can be glitchier in Firefox (but depending on the use case it can be the opposite). See https://padenot.github.io/web-audio-perf/#scriptprocessornod... for more details.
3. Worklets are bits or javascript that run on the audio thread directly, where you're supposed to only implement the DSP bits, and only communicate with the main thread using message passing.
It would be awesome if we could prevent this crackle somehow on a lower level of abstraction. What I mean by that is that if the buffer runs dry, the hardware (or the OS/audio driver) could do some prediction in order to bridge any gaps in the audio more nicely.
(I say "fading in", but it might have been that I had the reverb applied but dry, and transitioned to wet just before the signal ended.)
By the way, it wasn't done in code; I did it manually in Ardour [0].
[0] <https://ardour.org/>
What do you mean by latency issues? Why would there be any?
The brightness of the click depends on the speed of the level change. The faster the level changes, the brighter is the click. So, the level change speed can be compared with the cutoff of a lowpass filter. There is an easy formula for it:
Let's consider a level change from full to zero (or from zero to full) output from one sample to another on a machine that uses 44.1kHz sample rate. So, we first transfer the sample to milli seconds:
1 sample equals 1/44100 second, which is = 0.02267573696ms.
To calculate the cutoff frequency of the click, just use this formula:
Cutoff (Hz) = 1000 / Level Change Time (ms)
which in the example results in:
44100Hz = 1000 / 0.02267573696ms
Whoops? This the sampling frequency and, err, very bright.
However, what it will successfully avoid is the loudspeaker driver attempting to instantaneously snap from some nonzero x-position back to it's origin, which is what causes the really nasty clicks.
Smooth interpolation will avoid a really nasty pop, but in the real world, musical waveforms are highly complex, so any interpolation algorithm, however smooth, will produce some kind of artefact if you chop the wave in the middle of a cycle and smooth it to zero.
This can be observed when setting loop points in a sampler - you are usually provided with tools to help you match the loop points to the zero crossings. This is not enough however to remove all artefacts. Only some zero crossings will do: one has to match the higher-order cycles in the waveform as well. I don't really have the mathematical vocabulary to really describe what I mean here, but hopefully it's clear.
(BTW when I say driver in these posts I mean the magnet-and-cardboard-cone assembly in the speaker, not any kind of software.)
Filling the audio buffers with some predicted blocks of audio to avoid an ugly sounding gap.
See for instance in this Thesis: "Low-Latency Audio over IP on embedded IP systems" http://www.ti5.tu-harburg.de/staff/meier/master/meier_audio_...
Sect 4.1.2 Packet Loss Handling
This probably works but the proper way to do it back then (and the way drivers work today) was to hook an interrupt that gets fired when the audio buffer needs to be filled.
So there's an element of concurrency in it since you don't know when the IRQ is going to fire. Synchronization was easier with single cores, though. Just disable interrupts and you're done.
No probably about it - it worked like a charm.
> but the proper way to do it back then (and the way drivers work today) was to hook an interrupt that gets fired when the audio buffer needs to be filled.
Which was much more trouble than it was worth when writing a 64k intro on a deadline.