Hacker News Reader

Comment by SemanticStrengh | Hacker News Reader

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the semantic Causal reasoning abilities of human NLU that is involved in argumentation, inferences and understanding. But people are pathetically fooled by the fallacy of it being a uNiVeRsAl aPpRoXiMaTor.

View on HN

> And then your NN can't represent anything else than periodic functions

Significant parts of x+sin are close to linear. You don't need to use this activation as the result layer either. Why would we lose anything?

> If we had to build separate programs for each product requirements variations

We pretty much do? We use both extremely generic frameworks both in technical sense (.net) and organisational (sap). But we also have software written to specific requirements where needed (there's millions of very specific ways to invoice someone, companies get invoicing platforms written just for them from scratch). There's space for both approaches.

And yet neural networks can solve symbolic integral and derivative problems and differential equations better than other computer algebra programs. Sure one network might fail to compute sin(x) numerically, but another could easily tell you its derivative is cos(x). Turns out they are pretty flexible. Do they need to do everything?

Source? I would be very surprised if there was a neural symbolic PDE solver better than what’s in wolfram mathematica.

At the cost of how many parameters?

You are wrong.

https://www.vincentsitzmann.com/siren

? I did not say that NNs can't represent periodic functions. Show me a NN calculator with >99% accuracy if you wanna refute me.

> and then your NN can't represent anything else than periodic functions..

This is a quote from the parent comment where you state that NNs with periodic activations can't represent non periodic functions.

Siren (what I linked) uses periodic activations and is able to represent non periodic functions.

Oh OK, thanks yes that's what I was looking for! However if this NN can represent non Periodic functions, what cost does the Periodic function support incur on non Periodic function accuracy, if any?

It actually behaves a lot better than non periodic functions in some scenarios. It seems that period functions enable a network to encode positional information within [-1,1]. This is supported by the fact that positional encoding is needed in transformers to make them work.

Reply on news.ycombinator.com