Show HN: Nevermind XOR – Deep Learning Has an Issue with Sin | Hacker News Reader

Show HN: Nevermind XOR – Deep Learning Has an Issue with Sin

lostmsu.github.io

34 pointslostmsu10 comments

You're using a pointwise loss, which means you are treating each timepoint as conditionally independent. That's a deeply flawed assumption, probabilistically.

Try an autoregressive model of the joint probability distribution of the sine wave timepoints -- like WaveNet [1]. It will nail your sine wave -- just as it nails mixtures of sine waves (speech and music). :)

[1] https://arxiv.org/abs/1609.03499

I am sure a LSTM or even a small 2 layer network can learn to predict the next numbers on the clock from the numbers on the clock exactly one second ago, but that is not the point. Given such a network it will take you forever to make a prediction for second number N=10^315, yet it takes rather short time to divide with modulo.

I guess a broader point is that while the universal approximation theorem is a get-out-of-jail-free card for Neural Nets, nobody said "how quickly". A bit like trying to approximate sin(x) with a Taylor series for large x is suboptimal – Chebyshev polynomials are better (and Padé approximants probably better still), but both "work". Unless you have data points on the boundary of the domain, or some analytic knowledge that the solution space is bounded somehow, expect weird shit...

The remark about the Chebyshev polynomials just sent me down another rabbit hole.

>Their x + sin(x) * 2 activation only really worked when the number of full cycles in the dataset was less than the number of paramters of the network, which completely misses the point. interesting, how did you come to this conclusion?

It is just a guess, that tries to explain the observed behavior.

> def forward(self, x): return torch.sin(self.freq * x)

I don't think this qualifies as deep learning anyway

Well, the point is that neither large models could be trained, nor could a trivial model which is known to have a set of parameter values, that satisfy the given task.

yeah, and the way the problem is being treated is flawed, but that's another thing

“stohastic” should that be stochastic?

Reply on news.ycombinator.com