Can anyone give a rough idea of the actual limitations? I would guess that there is a limit to how non-local the effects it can manage are.
extern char* turing_machine;
int main() {
process_turing_machine(turing_machine);
beep();
}If you aren't familiar with RNNs, think about it like a NN that instead of learning a input -> output function, learns a (input, state) -> (output, newState) function
A call to `y, hidden = layer.forward(x)` (where x has a batch size of 1, and an arbitrary length) produces two hidden states of dimensions `(1, 1, hidden_size)`, where hidden_size is the exact number you passed to the LSTM constructor. Those two states represent the long term and short term memory features.
You would need to have an LSTM with hidden_size large enough to store the samples (or a compressed representation) of your entire loop. Not to mention you'd run into other issues with handling the logic around variable length loops based on a pedal toggle.
But, yeah, at some point your signal has such a complex behavior on long time scales that there isn't a good way to predict it based on a limited state size (or at least gradient descent can't find a function to predict it for you).
If the future input samples have a meaningful impact during loop playback, then it hasn't learned the correct behavior of the original loop pedal.
Note that the linked project appears to use a hidden size of 20. Twenty floats. With that much space we're very much back to "sure, you might theoretically be able to loop if the information fits in the hidden size".
Increasing the hidden size beyond 20 still won't solve learning the complex state machine behavior of an original loop pedal, which can loop variable length audio. You'd need to provide the pedal state to the network in addition to the audio, and probably train need to train it on a bunch of different loop lengths (>thousands?).
This would mostly be an academic pursuit, as it's extremely impractical compared to the other uses of the device.
The hard part is that there are some fundamental limitations to deal with. The biggest is aliasing - distortion effects in particular deal with enormous amounts of distortion (> 100% THD) which creates spectrums far outside the range of hearing. Digital audio systems need to have high orders of oversampling to prevent audible aliasing (8-16x is not unheard of!).
After aliasing is memory. It's too early in the morning for me to do math but I'm almost certain you can't model a looper with a causal NN that has less internal state memory than the length of your loop. Doing so is dumb anyway, since loopers are pretty trivial and their biggest cost is memory. Same goes for digital delay and modulation effects, the algorithms are not expensive.
A 3rd or 4th order polynomial interpolator is pretty darn good and doesn't need a NN to find the coefficients.
Oversampled DSP algorithms work by oversampling, performing the nonlinear processing, then filtering to remove the higher harmonics, and finally downsampling again. We do it this way because it's convenient and easy to understand and based on proven mathematics. But nothing says these steps have to be distinct.
An oversampled DSP algorithm looks like a regular DSP algorithm from the outside, perhaps with some more state and latency required for it to perform the internal oversampling. You can also imolement such an oversampled algorithm entirely at the original sample rate clock; it just means the processing needs to internally process several samples per outer loop sample.
Since neural networks excel at modeling "black boxes" as one amorphous blob that we don't understand, I wonder if a NN could learn to model such an internally oversampled algorithm fairly accurately, and what the computational complexity would be.
Since you can model the oversampling/filtering/etc steps as linear convolutions with wider internal state at the original sample rate, I'm almost certain this will work with the right NN topology. It's obvious an NN can implement oversampling.
And so my question is: could treating the combined oversampled processing as one step, and training a NN on that, potentially result in a more efficient implementation than doing it naively? Especially for heavy distortion that needs high oversampling ratios.
It can’t do time based effects such as delay, reverb, flange, chorus, etc. The LSTM can model distortion, overdrive, compression to an extent, and amp circuits (including vacuum tubes).