Hacker News Reader

Comment by srcreigh | Hacker News Reader

It already is.. can’t find the paper, but some image related deep learning model used sin(x) type of inputs to their model. This was a core improvement.

Not unlike how you can linear regress on x1,x1^2,x1^3,x2,x2^2. Encoding prior knowledge about input data as additional inputs helps with machine learning. 1980 called it wants its accomplishments back.

View on HN

I think most transformer models use sin and cos for positional encoding. They were recommended in the original transformer paper, Attention is All You Need.

But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding).

Was there a model that applied sin and cos to the input?

In a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier transformations of the inputs:

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739

Sin(x)-style encoding is called positional encoding and it’s used in neural radiance fields (NeRFs) and also in transformer-based models.

Could also be Sirens[1] which was all about sin as activation, while sinusoidal positional encoding are (iirc) just from the original transformer paper.

[1]: https://arxiv.org/abs/2006.09661

Gpt does this iirc

I doubt you’ll find any decent audio or image related models which don’t use sine inputs. It’s ubiquitous. It’s why jpgs and mp3s work.

Reply on news.ycombinator.com