trypwire

Hey all, Nick from Elementary Audio here. I spent some time in the discussion below yesterday and coming back to it again now I can see this continued to get way more attention than I expected! Thanks for all of your thoughts.

There's a lot of discussion here, and although I'd love to visit every comment in specific I think it makes sense to speak in summary to a few of the major themes/underlying questions here. In general, the discussion here surrounds totally valid concerns that I'll try to cover, and I understand after this thread that my messaging needs some serious work to speak to those concerns.

Before I get to those questions, I want to start with some of the things that Elementary is _not_ well suited for, or some areas where Elementary currently has some limitations.

* It's a young project; there are plenty of standard library features that are not implemented yet (oversampling, several more filter topologies, iFFT, etc). That might mean Elementary is currently ill-suited for your project

* It uses a generalized, block-based graph rendering algorithm. If you have hard performance requirements or need highly specific and hand tuned DSP optimizations, Elementary is currently not ready for you.

* Because of the current rendering algorithm, you cannot specify signal flow graphs that have 1-sample feedback loops.

With that out of the way, let's talk through some of the questions here that seem to underly many of the comments.

1. Why JavaScript?

* It's inarguably one of the most widely used languages in the world

* It's more than fast enough for the role that it plays in Elementary

* It's the language of the web; which isn't going anywhere any time soon, and at the same time the industry is showing us demand for audio software that runs in the web.

* JavaScript/web tech has pioneered the way that we write app UIs over the last decade, and continues to do so

* JavaScript/web tech has pioneered oen of the best developer experience/workflows in software development

2. How can JavaScript possibly be fast enough for this?

* Elementary uses JavaScript only to _describe_ audio processes, not execute them

* All of the actual audio processing is handled in the underlying C++ audio engine

* This engine is compiled to native, platform-specific targets in the appropriate cases (i.e. plugins, cli), and only compiled to WASM when appropriate for running in the browser

* The description of the audio process is done with an extremely lightweight virtual representation for which JavaScript is more than fast enough, and for which garbage collection and seamless integration with application logic and user interface programming is desirable

3. What about latency?

* The core audio engine adds no audio latency overhead whatsoever; if you're running a platform/native target, you're running with the same latency that your drivers/plugin host are delivering to Elementary. Until, of course, you describe an audio graph which introduces new latency; that's on you.

* When compiled to WASM to run in the browser, of course there's additional implications here, but that's a reality of targeting the web in the first place. Elementary doesn't force you to the web, or force you to wasm, but if you want to run in the web, accepting the latency implications yourself for your own apps, Elementary can go there with you

4. What about bottom line performance? Professional, commercial software??

* Elementary will not make the claim that for executing a static graph it will outperform native C/C++ hand-tailored to the application demands. That would be crazy; it's probably fair to say that highly specialized code can always outperform generalized algorithms.

* Elementary will make the claim that for plenty of applications, generalized graph processing is surely fast enough.

* The generalized graph processing algorithm is fast enough because the audio processing is purely native (or compiled to wasm); we do not execute any JavaScript itself on the audio processing thread

5. What about Faust/SuperCollider/PureData/ChucK/Max...?

* I have direct experience with many of these languages, and have reviewed almost all of the others in detail. I think they're excellent pieces of software for prototyping and exploring. I also think they're generally weaker for shipping complete applications.

* Of course you can use Faust/Max Gen or similar to generate C++ and then bolt on your own application harness, and I'd encourage you to try. For non-trivial applications, interfacing between the UI and the DSP becomes cumbersome. Elementary aims to provide a better experience here.

6. Why do any of this at all?

The current dev process and workflow for writing audio software has stagnated; it takes so much time to turn a prototype into a product and we regularly accept complexities and limitations that for so many types of apps we no longer need to accept. Elementary offers a new way of prototyping and shipping those apps more easily and more quickly.

Thanks again for all the discussion, I hope this clears up some of the details.

Cheers, Nick

100%. I'm actively learning exactly this, haha :). It's helpful feedback, thank you!

I couldn't agree with this more. Even beyond the web audio API, we have similar graph APIs in the low level/native C++ world that become unmanageable in their complexity simply due to an inability to compose and manage transitions with ease.

This is exactly one of the difficulties that Elementary aims to address, whether you're running in the browser (where indeed, Elementary uses almost none of the actual web audio api), or natively.

That is indeed on offer :)

The new docs structure broke the website links, I'll get that fixed asap. See here in the mean time: https://www.elementary.audio/docs/packages/plugin-renderer

Also, as another user here has already pointed out, the "plugin development kit" is still macos only, though I anticipate releasing the windows version within hopefully a week or two.

Nope! You can run the same Elementary code in the web browser, at the command line, offline (for file->file processing), or in a native audio plugin

Thanks! The docs just underwent a significant change for v1.0; the new plugins page is here: https://www.elementary.audio/docs/packages/plugin-renderer

Hey, Nick from Elementary Audio here.

You're totally right that in this domain you have to be extremely careful with performance and latency. Elementary takes great care to deliver that; it's a JavaScript "frontend" API but all of the actual handling of audio is done natively with high quality realtime constraints. The core engine is all native C/C++, and we run in the web by compiling to wasm and running inside a web worker. For delivering a desktop app or audio plugin, the native engine is compiled directly into the target binary.

So, while JavaScript and garbage collectors can bring a performance concern, we're only using JavaScript there for handling the lightweight virtual representation of the underlying engine state, and for that role JavaScript is plenty fast enough and the garbage collector quite helpful!

I’m a big fan of Faust, have used it before in many of my own audio prototypes. Its a great learning resource if you’re new to dsp.

It’s been an inspiration for me too in building Elementary Audio [1], which you might find interesting if you’re a JavaScript developer

[1] https://www.elementary.audio

I have indeed! I've used Faust quite a bit over the past few years, and it was very much an inspiration for Elementary. In general I wanted a tool that meant I could think and work on my dsp in the same way that I think and work on my UI, and with the same language even.

It's coming ;)

Awesome! Thanks for sharing

Thanks! Oh yea, such a rabbit hole.

I started trying to synthesize a snare as part of this library and blog post and got lost in the weeds so quickly. I ended up skipping it for now but will surely come back to it eventually

Thanks, and yes I totally agree (OP here). I looked into it for a brief minute: I'm using Ghost Blog to host my articles and I didn't see an easy way to attach the media there directly. The available options seemed to involve embedding a Souncloud link or so, which was enough to deter me at the time.

If anyone has ideas for simple audio clip sharing in a Ghost blog, I'd love to hear

Great recommendations in here, and I'm happy to see this thread getting such attention!

This is totally a shameless self-plug, but I think it could be interesting for you:

I'm working on a project called Elementary Audio [1] which is a javascript runtime + framework for writing native audio software. It's like the Web Audio API in that it's javascript+audio, but unlike Web Audio in that it aims to target true native audio apps, like plugins for your DAW or hardware projects.

The API that it offers feels to me very much like thinking and working in analog synths, which is why I think you might find it interesting. You can describe and wire up signals and just see what they sound like without having to worry about what needs to happen under the hood for you to hear it.

I put together a guide for dipping your toes into making sound [2] and you'll find there a bunch of other resources that I recommend for getting into the topic.

I should note too that it's currently in beta and only supports macos and linux (windows coming soon!)

[1]: https://www.elementary.audio/ [2]: https://docs.elementary.audio/guides/making_sound

Thank you!

I've used Faust quite a bit myself, and it has been an inspiration of mine for this project. But indeed, it has limitations that were prohibitive for me, and I'd rather not have to context switch from a DSL for my DSP into some completely different language for the rest of my app if I can avoid it.

> I couldn't tell from your post and linked material how the runtime works — I see that the high-level graph is handed over to the runtime, but is it interpreted? Compiled? Does it require platform-specific binaries?

Yea, check out this page of the docs: https://docs.elementary.audio/guides/native_rendering It is interpreted dynamically by the runtime, and currently does require platform-specific binaries but I'm also building the runtime such that end users can embed the functionality into any app they need

> Does this support writing audio sample processing code in JavaScript?

Currently, no. My intention is that the library of pre-existing nodes is both complete enough and low-level enough that 95% of things end users may wish to write can be done through this composition. That said, the embedded runtime engine is designed to be extended with your own C++ processors if you have an application which particularly needs that level of customization.

And why pure functions? Functions with side effects don't compose the way that pure functions do, because you can't rely on deterministic output given a consistent input. If you choose, you can still write your own non-pure functions to assemble your signal graphs, though I personally wouldn't recommend it :) The Elementary library will always aim to be strictly pure functions.

Thanks, fixed :)

I'd say there are two primary differences between this and any current web audio implementation that I'm aware of:

First is the model/API for actually writing your app. Web Audio is very imperative and, in my experience, doesn't handle dynamic behavior terribly well. Elementary is designed both for functional/declarative expression and for change over time.

The second piece is that Elementary is built around a core runtime which can be embedded anywhere. Web Audio is, as its name might suggest, quite coupled to the browser environment. My goal is that if you've written an audio app that runs at the command line with Elementary, you could package that same thing up and ship it as a VST plugin, on embedded linux hardware, etc.

Thank you!

As shown in the introduction video, with the `core.on('midi')` callback, MIDI voice triggering is not sample accurate. It's quite hard to invoke arbitrary user code in js and still hit a sample accurate deadline!

That said, the embedded runtime itself supports sample accurate interaction in this way if you're willing to get into a bit of C++.

Currently, building a VST plugin requires just about as much C++ glue as it takes to write the interface. The next step for Elementary though is a solution which addresses that bit too. My goal in the near term is a platform to write VSTs with no C++ at all.

I agree very much with your last comment there, and that is a major impetus for building this tool. I'd love to see what kind of plugins come out when it's that much quicker and easier!

Why do you say that? I've built the core of the engine entirely from scratch, entirely indepenent of juce. The CLI tool does use juce for opening the audio device, but that's about it.

Funny you should say that :) I'm currently building a small drum synth, will share it as soon as its ready.

In the mean time, check out https://github.com/nick-thompson/elementary for some examples that you can `npm install && npm start` to hear

OP here– good point, that's not well explained in the article! And thanks to the others who have answered already, you're pretty much spot on.

The core library, referenced here as `el`, contains a set of factory functions for describing various underlying processing nodes. So when you write `el.lowshelf()` you're just using one of the builtins (like how you might use a div, span, or button in a web app with React)

Awesome discussion in here, I'm happy to see it!

I thought I might throw my hat in the ring of shameless self promotion since you might be interested:

I'm working on a project called Elementary Audio, which is a brand new javascript runtime and framework for writing native audio applications– meaning desktop apps, like perhaps the mini-synth you're hoping to make, as well as audio plugins (VST, AU, AAX, etc) and even embedded hardware, all in JavaScript.

The project is still in an early stage (and macOS only right now). I'm planning to share it more widely here on HN once it's ready for beta, but I thought this thread might be interested in it before then.

Website: https://www.elementary.audio/

And here's an example MIDI Synth: https://github.com/nick-thompson/elementary/blob/master/exam...

You could even spin up an Electron frontend to take input from your computer keyboard and send those over to Elementary to render the synth sounds. It's actually an example project I was considering doing myself but you might beat me to it at this point!

Lastly, I'll say briefly that there are a lot of tools out there (as we're seeing in these comments), different things work for different people, and there's no single "right" way to start audio programming– find the path that makes it fun for you, because it's such an awesome field to get into.

Yea there's a link to the github with a brief blurb about open source on that page. I'm not planning on supporting Csound or LADSPA at this time, but I'm definitely open to pull requests!

Definitely, and I think your concerns are valid, but if you found that loading each file on demand yielded the optimal performance for your app, wouldn't you go with it? Additionally, I think the method the author is presenting can be very well suited to both partial bundles and individual files. For what it's worth, you and I are in similar positions and it sounds like we have very similar approaches with our own apps. As long as readers understand that this is one possible solution to a nontrivial problem, I'm happy with it.

It's important to understand that there are tradeoffs with lazy loading. There's a spectrum, right?

On one end, you could serve all of your js bundled with browserify on the first page load, which, for SPA's will frequently block rendering until the entire bundle is downloaded, parsed, and executed. If your bundle is small, who cares? Once your app grows to a large enough size, that initial delay might no longer be worth it.

On the other end, you could serve almost no assets with the initial page load, and request what you need when you need it. Your initial render will be blazingly fast but any first interaction might be slowed by the latency involved in going to get the relevant assets needed to execute the interaction.

Somewhere in the middle you may find a solution involving sending half of your assets initially and requesting the other half when you need them.

It's not helpful to write off various methods of asset loading because of assumptions about latency, bundle size, or DOMContentLoaded event delay. The important part of this article is that it speaks to the various approaches to this problem. I'm guessing the author took the time to weigh the pros and cons of these approaches, chose this one which particularly works for him, and wrote about it to share his findings.

This isn't a trivial problem with one solution. Dig into the performance of your apps while wearing multiple hats and take what works best for your situation ;)