Hacker News Reader

Comment by pontus | Hacker News Reader

Is there anything deep here? There's a parametrized representation of a function and it's being interpreted as a neural network, why is this surprising? It's like saying that Newton's second law is a neural network: F=ma can be written log(F) = log(m) + log(a). Aha, this is a neural network! The inputs are m and a, the first layer is sparsely connected with log activation functions, the second layer is fully connected with an exponential activation function:

F = exp(c1 * o1 + c2 * o2) o1 = log(c3 * a + c4 * m) o2 = log(c5 * a + c6 * m)

If you feed it enough data you'll find c1=c2=c3=c6=1 and c4=c5=0.

But saying that Newton's second law is a Neural Network, while correct, seems a bit deceptive in that it's not a deep idea at all.

View on HN

Op here, IMO the "deepest" bit is [1] - the network learns the DFT in order to reconstruct the signal, and is not explicitly trained on the DFT values from the FFT.

Admittedly, I should have mentioned that any linear transform can be considered to be a single layer neural network (if you want to see the world through a neural network lens), and will add this to the post at some point.

In fact, I have a series of posts planned, which will reveal that well known algorithms/models are actually neural networks...

[1] https://sidsite.com/posts/fourier-nets/#learning-the-fourier...

You might be interested in this line of work: https://eng.uber.com/neural-networks-jpeg/

Training straight from DCT coefficients to avoid spending time learning a similar representation in the bottom layers of the net. I've personally toyed with something similar on GANs to gauge the computational benefits of not doing convolutions in the bottom layers of a net but learning directly in a FFT-like compressed space instead.

Isn't there something like Fouriernets and spdnets already, and stuff on Riemannian manifolds and my head is spinning? Followed Daniel Brook's work for some time, e.g. https://hal.archives-ouvertes.fr/hal-02290838

As far as I can tell, you're still using a fixed inverse DFT as the reconstruction layer, so it's not just rediscovering the DFT on its own. Instead of learning a linear transformation from input-output pairs, it's learning the inverse of a linear transformation when that transformation is given as an oracle. It's not terribly surprising that this works, although there are probably some interesting issues of numerical conditioning in the general case.

Actually, "learns" here means to fit reverse linear transformation a.k.a. inverse matrix. He defines reverse FT matrix and using gradient descent numerically converges to inverse matrix. Nothing more than inefficient way to solve system of linear equations.

> I should have mentioned that any linear transform can be considered to be a single layer neural network

I think this should be turned around. A single layer neural network can be considered a linear mapping (but not necessarily an orthogonal transform or change of basis, like the DFT).

A more clear example of this are adaptive filters, which are trained in real time using gradient descent.

This is an important distinction because thinking of "X is a Neural Net" doesn't provide meaningful insight, whereas "Neural Nets with X properties are a case of linear dynamic systems, here's an example of how we can equate one linear transformation to a neural net" leads you to deeper conclusions on the analysis and synthesis of ANNs in the context of dynamics - which encompasses a much larger surface area than the DFT.

I've sometimes wondered how many signal processing kernels or filter architectures you could learn via SGD given the right data and search strategy. Maybe using something like Neural Architecture Search, so not just the weights / coefficients but the architecture too.

Could it discover an interpolation filter like upfirdn? An IIR Butterworth filter (it would have to be recurrent)? A frequency transfer function derived from two signals?

I imagined that the solutions it found wouldn't be "clean" but have other non-essential operators bloating them. Could it find new architectures that haven't been created from first principles?

I think a big question here is, "what is the goal?" A key distinction is whether the NN is intended to handle the signal directly (the NN is the filter) or if it's just a technique for finding filter coefficients (the NN designs the filter for the signal).

For some tasks, like system identification (used in echo cancellation), the NN is the filter - aformentioned adaptive filters are used for this case right now. It can also be used for black box modeling, which has numerous application in real time or otherwise.

For others like Butterworth (and other classic designs) there's not really a good reason to use a NN. Butterworth (and Chebychev I/II, elliptical, optimum-L, and others) are filter design formulae with a closed form (for a given filter order) that yield roots of transfer functions that have desirable properties - I'm not sure how a learning approach can beat a formulae that are derived from the properties of the filter they design.

There are iterative design algorithms that do not have a closed form, like Parks-McClellan. It is however quite good at what it does - it would be interesting to compare these methods against some design tricks to reduce filter order.

There are some applications of filter design that NNs can do that we don't have good solutions for with traditional algorithms, like system identification, another is in optimization (in terms of filter order) and iteratively designing stable IIR filters to fit arbitrary frequency curves (for FIR it's a solved problem, at significant extra cost compared to IIR).

As for topologies of the filter itself that is an interesting angle. Topologies affect quantization effects (if you wanted an FPGA to realize the filter with the fewest number of bits, topologies matter) and for time variant filters (like an adaptive IIR) there are drastic differences between the behavior of various topologies. I'm not sure how it would manifest with a NN designing the filter.

I meant learning the topology (I called it architecture) from scratch, or minimally from heuristics. What would I discover? Something that we recognize as an IIR filter? Probably not, but if it did, what would it discover that we didn't recognize over years of academic thought but could have from first principles? To me that's the intriguing angle.

I don't think I follow. These designs do come from first principles. In fact the reason we've had so much academic thought put into it is because 100 years ago we designed systems empirically and a few smart people went back to first principles to find the fundamental constraints on filter networks.

That is how we did create such topologies. But if we had not, could they have been discovered through neural architecture search [1]? NAS has been used to find architectures that outperform those that were hand designed. And in RL , Google used a method inspired by NAS [2] to learn methods built by hand (i.e. TD learning, DQN). I know it's not the same but if you use a bit of speculative imagination you can translate what has been done to signal processing applications.

[1] https://en.m.wikipedia.org/wiki/Neural_architecture_search

[2] https://ai.googleblog.com/2021/04/evolving-reinforcement-lea...

> But saying that Newton's second law is a Neural Network, while correct, seems a bit deceptive in that it's not a deep idea at all.

I guess the point is that neither is the idea of a Neural Network.

A neural network can also be set to learn unconditionally return a fixed value with no learning feedback. I don't think lower bounds on capabilities are very informative. So could many arbitrarily complex arrangements that do massive amounts of work only to discard it and return a constant. An upper bound of what an approach is capable of is more useful. Say no matter how vast a look up table is it will never return a different value for the same input regardless of prior sequence.

Perhaps it is the compsci glasses talking, but this is just one very specific instance where someone figured out a way to map the DFT problem to a neural net. I agree that it is unfortunate that it is being presented as some kind of big discovery, and that the fundamental lesson is either still undiscovered to the author or just unclearly communicated, but there is still good intention in there (sharing something you've learned and eliciting feedback).

It's a stretch to call it a neural network. It was already well-known that the Fourier transform can be seen as a matrix multiply which minimizes some least squares problem.

This can be found somewhere in S.M. Kay's Fundamentals of Statistical Signal Processing: Estimation Theory (Vol 1), Detection Theory (Vol 2).

I think this is the sort of thing that is very obvious if you are already comfortable with neural networks and the FFT. But if you're only comfortable with neural networks, and the FFT feels like arcane magic, this exercise might be very instructive.

Well the point is that it does not make sense to DFT data before feeding it into a multilayered neural network (or calculating the force generated by mass and acceleration as you point out). Those formulas make no sense: the network can just learn them on the fly.

In fact you'll find that this does not just work for the Fourier transform, but for any FIR filter (and some other classes), and therefore neural networks can deal with signals and construct low-pass filters, high-pass filters, bandgap filters, ... as required for the task at hand without the network (or it's designer) having any idea at all what is happening.

I mean there's some basic assumptions these reasonings make (main one is that you need to feed many discretized values from a time window).

Of course, a problem remains: local optima. Just because a neural network can construct a filterbank or do a DFT, doesn't mean that it will actually do it when the situation warrants it. If there's a local optimum without filters ... well, you may get unlucky. It there's many local optima without filters ... sucks to be you.

> Those formulas make no sense: the network can just learn them on the fly.

> Just because [...] can construct a filterbank or do a DFT, doesn't mean that it will actually do it when the situation warrants it.

These statements seem in conflict, no?

I don't think I can agree with that. You do a Fourier transform when the data you're working with doesn't admit easily or at all certain operations in the time domain but it does in frequency domain. If you already know that to be the case, preprocessing with a FFT is a better idea than hoping a neural network with enough layers uses a few of those to much less efficiently perform a DFT. Always take advantage of pre-existing knowledge of structure in your data. With the FFT especially, depending on how your data is being ingested, you might be able to use specialized DSPs that implement the FFT directly in hardware. These are cheap and easy to find since they're used in frequency-division multiplexing.

Practically, yes, absolutely! Theoretically, we want the "black box" learning system to do such feature extraction itself. At least, that is the goal.

> does not make sense to DFT data before feeding it into a multilayered neural network

is false.

"It would be nice if we didn't have to DFT data before feeding it into a multilayered neural network" is true, but a completely different statement.

Which is weird, because biological neural nets seem to have some evolutionary pressure to do hardware fourier transforms before neurons even get involved.

You can see this most clearly in the auditory system where the incoming signal is transformed into the frequency domain by the cochlea before the signal is received by the epithelial cells.

Neurons absolutely love working in the frequency domain, but they seem to prefer to not be the ones to do the binning in the first place.

This is probably because the neurons involved can't switch fast enough. Our neural hardware is simply too slow to work in the time domain directly.

They do just fine with acoustic frequency ranges.

Firing rates for neurons are in the order of 10hz - under the minimum acoustic frequencies of about 20hz.

Peak neuron firing rates basically only touch the lower bounds of frequencies that need to be processed.

Without the "hardware acceleration" of the cochlea that preprocesses the time domain signals to frequency domain first, basically our whole audio sensorium is out of processing range.

Normal nervous signal transmission speeds are in the order of the speed of sound, switching rates are in Hz range. Voltages are in the 10s of millivolt range.

Neuron spiking isn't rate encoded, it's temporal encoded. Ie. the distance from the next expected regular interval is the signal, not the spiking interval itself.

Additionally there's large variations in neuron spiking rates.

Agreed.

It's encoded in an elegant way.

It has to be, due to the signalling constraints.

It's not like a microphone-based electronic system that's fast enough to directly transmit an analog sample of air pressure changes.

Are you suggesting that anything over 10Hz is perceived with the representation of a rate, rather than direct stimulus?

It has to be encoded somehow.

Maybe not a rate.

It can't be "amplitude modulated" because in general signals are pulses.

So it's kind-of digital.

It sometimes can't be a pulse rate encoded signal due to frequency cutoffs.

As far as I understand they've identified several different encoding schemes and will probably find several more.

Yes it is encoded. Essentially the rate of firing for a neuron encodes an exponential value to be represented. This is called "spike trains".

You can see this clearly if you do an extreme slowdown of a human movement. Then, suddenly, what looks like a smooth movement, like raising an arm (and because of inertia it is smoothed of course), isn't really smooth. A pulse arrives in the muscle, and there is 20ms where the muscle is tensioned, and then it's back to neutral for 100ms. Then another spike arrives, another 20ms where a lot of tension is put on the muscle, the movement accelerates, and the muscle goes back to neutral. It's not a continuous movement at all.

https://en.wikipedia.org/wiki/Biological_neuron_model

But odds are good that it's not just the value that's encoded. Many experiments have shown that it matters a lot if the signals are in phase (ie. they encode the same or some multiple of a value, but that the signals started at the exact same time matters, maybe more than the value itself. Or in the encoding: while for the value only the distance between 2 spikes matters, if 2 spikes on 2 different neurons occur at the exact same time, this will be interpreted as very relevant, even those both spikes may have very different firing rates)

I think I understand your sentiment, but consider our tactile sense of vibrations. We are able to distinguish different frequencies with the nerves in the skin, without cochlea doing the frequency domain transform. I think there are very different spectral limits and aliasing artifacts, but it's not like we can only sense a 5 Hz vibration either...

Again, I think that the sensor itself is detecting the frequency.

The actual signal isn't "direct".

IIRC from reading up on haptics - different tactile sensors are sensitive to different ranges of stimulus. Different cell bodies sense different effects and frequency ranges and send an encoded signal.

We tend to forget that the underlying technology is very different and needs different encoding, signalling and computation approaches.

The main sentiment is that our nervous system is quite slow compared to our electronics and needs to use different design approaches and novel encodings, to get things done.

In much the same way our electronics can't engage with visible optic phenomena "directly". I mean visible light frequencies are too high to directly sample and transmit electronically the sensor used has to deal with that.

Right, I know it isn't direct. The ability to introduce aliasing and fool this perception system is the basis for haptic feedback systems. I was responding more narrowly to the topic of a frequency domain transform as you get from the cochlea. I didn't mean to suggest that there are not other "hardware" signal transforms or feature extraction happening prior to the nervous system signal processing.

As I understand it, the touch sensing structures are more like a zero, first, and second derivatives (deformation/tension, velocity, acceleration) and these coincidentally have different frequency sensitivity curves. My reading is that in some ways this might be more similar to how our visual system integrates different photon detection signals to interpret color. Rather than the clear frequency domain transform of the cochlea, there are broadly overlapping sensitivity curves with only 2-3 peaks.

The "Deep" part ia to realize what this really means in terms of limitations of ML/NN!

Can be represented as a neural network is not the same as is a neural network??

I think the issue here is that almost anything can be represented as a neural network. You could create a neural network that does a xor operation, for example. There is nothing new about this.

Reply on news.ycombinator.com