Declarative, non-intrusive, compile-time C++ reflection for audio plug-ins
github.com
github.com
I've spent some (the happiest part!) of my career (and ~2/3rd of my life, hobby wise) involved with sound/music technology, designing and writing audio (and some video) plugins (VST, AU, PD, and others), audio middle-ware, plugin host frameworks, low level DSP algorithms optimized for performance, authoring music/sound protocols (MIDI-over-BLE, and tons of unfinished network sync things...), translation layers for seamless porting of effects between APIs (ie: auto-generation of an iOS app, or embedded microcontroller code, from VST source code), algorithms/processes for "JITish" C/ASM code generation from a concise higher level mathematical definitions (think: something you'd type in a CAS), so you can edit the math at runtime, and see the result within seconds.
8 years ago I made a PoC of something similar to this, using dark C++ magic (policy based design, lots of TMP) to enable compile-time optimization of filter chains, where each filter is encapsulated at a source level then connected in a dataflow graph, to produce an optimized binary of the whole chain, suitable for running as an application or on bare metal. Apparently, much of what I was going for eons ago (with c++11) is possible now with c++20!
I've just begun to skim the code, with an arrogant "This is in my domain of expertise!" hat on. So far, I really, really like what I see, an uncommon experience when I look at Other People's Audio Code. All sorts of little clues that point towards "good code smell". ("Oh good, they used <such and such> library..", "Hah! Correct usage of memory fences!", &c)
I tip my hat to you, author! I am embarrassed I hadn't heard of this project before!
As soon as I can afford to focus on anything non-paying, I look forward to contributing, competing, or both, with this project and author. :)
Dear lord yes. YES. They're all signals (audio, images, video, radio waves, EEG readings, &c). Just in different dimensions and timebases. The math used to tease meaning from the signals is pretty universal.
Understanding fundamentals like convolutions, filters, frequency domain, bandwidth, sampling limits, &c are just as critical if you want to do video processing as audio. A Wiener Filter will clean up audio, as well as an image. To a very crude approximation, the math that lets us compress audio to MP3s is the same that lets us compress images to JPEGs.
That first question though, is hard. I've been doing it for too long to have any useful advice.. (sorry) I would say, the best way to get into it is to... well... just start.
Don't worry about the programming language, try things out in a high level tool, like MATLAB. With a foundation of simple diffeq and some stats, you'll be able to do stuff :)
https://theaudioprogrammer.com/discord/
There are some devs from some well-known companies, as well as maintainers of a few popular frameworks there.
They're (probably) going to tell you to use JUCE, use "Projucer" as a build tool, and C++ as a language. That's what most people do. I don't particularly love C++, and I prefer the CMake build, instead of their custom build tool, but potato-puhtahto.
There's also a Rust Audio development community:
And it may be a bit niche, but there is an (in my opinion) fantastic framework for writing audio plugins in D:
https://github.com/AuburnSounds/Dplug
It uses Physically-Based Rendering for the UI (PBR), which is generally used for 3D game stuff I think? It's a neat approach and the UI's it produces can look very realistic -- only framework that does this AFAIK.
The author Guillaume is a nice person and will answer questions in the #audio-dev channel of the D-lang Discord that is linked in the Dplug repo.
> I look forward to contributing, competing, or both, with this project and author. :)
yes ! please do :)
This is mindblowing work. I'm having some trouble wrapping my head around all of it. Will lead with the context that:
A) I am not very familiar with audio programming specifics, or DSP at all. I have done some things with VST2 + VST3, and looked into LV2.
B) I have very little background with C++
1. Is this targeted more towards the DSP and algorithm side of audio plugin development, or is there interest/use for the plugin side too (IE, generating VST/LV2 compatible interfaces)?
2. Would a byproduct of this be that you would be able to implement functionality + interact with C++ classes from other languages? Or generate C ABI's? I see the note: "Automatically generate Python bindings for a C++ class"
-----
Edit: I also wanted to say that when reading your blogpost, the way you write C++ is very easy to understand. Not sure if it's a stylistic thing or a purposeful choice. I usually have to spend a second reading C++ code to comprehend it, but in your snippets, I was able to (mostly) grok it first pass.
Same for the example code in the repo.
> 1. Is this targeted more towards the DSP and algorithm side of audio plugin development, or is there interest/use for the plugin side too (IE, generating VST/LV2 compatible interfaces)?
it does generate something compatible with VST2.x and I've been working on VST3 this week-end (and pulling a few hairs, that API is horrendous, the smallest example is a few thousand lines of codes).
For LV2 I don't think it will be directly possible as LV2 requires separate .ttl files which describe the plug-in. Technically one could make a small program which from the C++ reflection, outputs the .ttl on stdout and kindly ask CMake to call it and generate the .ttl file from it.
> 2. Would a byproduct of this be that you would be able to implement functionality + interact with C++ classes from other languages?
yes, that's a goal, make your DSP processor in C++ and then write a small UI for it in python or QML without having to write binding code.
I've tried to find ways to auto-generate a C ABI but sadly:
- asm("...") only wants string literals, strings generated through constexpr don't cut it (otherwise it'd be possible and even relatively easy to generate e.g.
asm(R"(
.global foo_set_x
.text
foo_set_x:
jmp _ZN3foo5set_xEi
)");
- I tried writing directly the binary code and appending it to the ELF sections but it seems that there's no easy way to do append to .symtab / .stringtab where the name of the symbols are without using a linker script.
> "it does generate something compatible with VST2.x and I've been working on VST3 this week-end (and pulling a few hairs, that API is horrendous, the smallest example is a few thousand lines of codes)."
If you want a tip about the smallest possible VST3 implementation, (and maybe you already know this) there is a class called "SingleComponentEffect", and an example called "AGainSimple" that uses it and is a fully self-contained single file full VST3 plugin:
https://github.com/steinbergmedia/vst3_public_sdk/search?q=s...
Also, I spent some time trying to get VST3 SDK usable with vcpkg so you could just do:
// vcpkg.json
{
"name": "my-audio-lib",
"dependencies": [
{
"name": "vst3sdk",
"version>=": "3.7.3"
}
]
}
https://github.com/microsoft/vcpkg/issues/5660#issuecomment-...
It ended up requiring more changes than this, but in the end I wound up fixing the CMake build so it spit out both the VST3 SDK (helper classes) shared lib and the almost-header-only lib ("pluginterfaces" directory).
I ought to upload the patches somewhere in case it's useful for you/anyone else.
> "yes, that's a goal, make your DSP processor in C++ and then write a small UI for it in python or QML without having to write binding code."
> "- asm("...") only wants string literals, strings generated through constexpr don't cut it (otherwise it'd be possible and even relatively easy to generate e.g."
What I did instead was do:
git diff > vst3sdk-no-vstgui-cmake-compatible.patch
https://github.com/GavinRay97/vst3sdk/blob/master/vst3sdk-no...
I also just said fuck it and zipped up the whole project, you can download it here:
https://github.com/GavinRay97/vst3sdk/blob/master/vst3sdk.zi...
With this fixed version, using the VST3 SDK becomes much much easier. All you have to do is add it as a vendored subproject in CMake:
project(VST3Example CXX)
add_subdirectory(vendor/vst3sdk)
add_executable(VST3Example src/main.cpp)
target_compile_features(VST3Example PRIVATE cxx_std_20)
target_include_directories(VST3Example PRIVATE vendor/vst3sdk)
# This links SDK if you want to use the "SingleComponentEffect"
target_link_libraries(VST3Example PUBLIC sdk)
$ tree -L 2
.
├── CMakeLists.txt
├── src
│ └── main.cpp
└── vendor
└── vst3sdk
#include "public.sdk/source/main/pluginfactory.h"
#include "public.sdk/source/vst/vstsinglecomponenteffect.h"
using namespace Steinberg;
using namespace Steinberg::Vst;
class MyVST : Steinberg::Vst::SingleComponentEffect {};
__declspec(dllexport) IPluginFactory* GetPluginFactory() {}
Take a look a this[1] for example. See how trampoline, the destructor and the size are passed in with the 'i' constraint and are referred to their value with the %cX constraint (yes, the code is write only and even with a lot of comments I have only the most vague idea of what I was trying to do here).
Probably more work is require for PIC though.
[1] https://github.com/gpderetta/delimited/blob/7e755d643ee45897...
https://github.com/steinbergmedia/vst3_pluginterfaces/blob/m...
# pluginterfaces should actually be a header-only library,
# but it has some sources as well which need compilation.
> "I almost have it working, will push tomorrow after some sleep and then will add vcpkg as you showed :)"
Here the idea is to write the algorithms in a way that is more future-proof, by not having them to depend on any run-time API, just a generic specification (given as a set of C++20 concepts). This way the algorithms will still be useful in 10 years when everyone has moved to API N+1, unlike a metric ton of existing audio software which depends on a specific audio / media-object API for no good reason (today ! When they were written C++ wasn't advanced enough to allow this at all)
- all the objects in https://github.com/pcastine-lp/LitterPower for instance
- all the algorithms in https://github.com/MTG/essentia/tree/master/src/algorithms
- ditto for https://github.com/cycfi/q/tree/master/q_lib/include/q
- ditto for all the VCV algorithms: https://github.com/VCVRack/Fundamental/tree/v1/src
- ditto for all the Bespoke objects: https://github.com/BespokeSynth/BespokeSynth/tree/main/Sourc...
- ditto for all the LMMS plugins: https://github.com/LMMS/lmms/tree/master/plugins
etc etc, there's a couple hundred of those, which always depend on some API and are thus not easily portable across environments: if tomorrow you want to make an audio software and want to use one of the, say, VCVRack plug-ins you're going to have to bring the whole VCV run-time API along.
Related projects are Faust (https://faust.grame.fr/) and SOUL (https://juce.com/discover/stories/soul-first-public-beta-rel...) but they both are domain-specific languages with their own compilers. I wanted a pure-C++ thing instead, which allows to call directly native code, and enables more than just generic audio processing: unlike Faust and SOUL (last time I checked) it's possible to make a message-based object for Pd or Max, not just an audio filter or synthesizer.
My end goal for this is that when I make an object for the main software I'm developing, ossia score (https://ossia.io) then the whole media arts community can benefit :-)
More baroque, for sure. More future-proof? Only if this is the one true form of reflected code that gets adopted.
> This way the algorithms will still be useful in 10 years when everyone has moved to API N+1
Actually, this way algorithms are not useful today since nobody writes them this way, and in 10 years, whoever writes this way right now will probably have moved on.
Now, if some small community (like C++-reflection-and-audio-buffs) commits to this, then maybe it makes sense for them. But not generally. General reflection (not to mention runtime reflection) is probably the way to go.
ah, actually not :) the library tries to do things in two steps in many places:
1/ map the user's code to a proxy depending on which concepts it conforms to: for instance whether your audio processing function is written per-sample (that's still in progress tho):
float operator()(float input) { return input * 0.5; }
void operator()(float** inputs, float** outputs, int frames) { for(each channel, each sample) { ... } }
this means than if tomorrow we get reflection and metaclasses (I hint to this in the readme), and that we can write the "ideal" struct which (to me) would look like
struct my_effect {
[[name: "Main inputs"]]
audio_input main_input;
[[name: "Main outputs"]]
audio_output main_input;
[[range: {20, 20000}]]
[[unit: hertz]]
slider frequency;
// will create an appropriate message in max / pd with the name some_function
void some_function(int, std::ranges::view<std::variant<float, int>> list_of_stuff);
auto operator()(std::floating_point auto in) { return in * gain; }
};
of course the existing algorithms will stay uglier, but they'll also keep working with new environments (and, as far as possible, without a runtime perf. hit)
> General reflection (not to mention runtime reflection) is probably the way to go.
yes, we all want that !
In the audio space I can imagine Concepts' Duck-typing is probably fairly effective for this work, you're rarely going to have something that looks like this flavour of duck but is not, in fact, a duck. It has been many years since I wrote LADSPA plugins, but there I'd guess the only confusion might be that there's deliberately no API difference between PCM data and CV, a reasonable strategy if you're building a modular synth but probably not what you want to listen to as output. Such differences are expressed only in metadata.
On the other hand, the same thing that presumably made this an attractively simple project for C++ 20 features also makes its practical utility very limited. Over the years what seems to matter to musicians is mostly the packaging, not the algorithms. The same exact algorithm exists for twenty years and then somebody famous calls it "Wonky-smog-burble", paints it fluorescent purple and presents the control parameter as an integer between 3 and 17 - suddenly it is this week's hot new sound. Which is fine of course, but the API doesn't matter at all to such fashions.
It would be nice if there was an example of that too.
I'm not sure I understand: the whole idea is that since the objects are just C++ structs, to "consume" them from some C++ program would just mean including the file and using it.
Otherwise, if you want to know how you can do similar reflection on it (for instance to generate an UI from any of those processor) the simplest example is the python binding, the meat of it fits in ~100 lines of code:
https://github.com/celtera/avendish/blob/main/src/python/pro...
if that is what you mean, then yeah what I plan to do is to write some documentation to explain the techniques and reflection API