Hacker News Reader

Comment by AbrahamParangi | Hacker News Reader

Candidly, what the hell am I reading? The fundamental thing a network operates on is a (often massive) multidimensional tensor. GPT3+ has per-token dimensionality in the thousands. Not only do we already operate on incredibly high dimensional objects, it’s the only thing modern neural networks do.

View on HN

I had the same reaction. The whole time I was expecting a paragraph to explain why this is fundamentally different from binary classification techniques like SVM, which explicitly uses a hyperplane in high-dimensional space to divide the classes.

I actually had to check the publish date on this. The first few paragraphs are absurd.

If we are being pedantic, transformers operate on integer vectors, and in the process turn them into high dimensional matrices.

Hah, same here. How is [colour, shape] or [blue, red, square, circle] not a vector? How does a NN work if not by operating on vectors/matrices/tensors?

This Quanta Magazine is absolute horseshit. Might as well just remove all that text and link to the author's paper. But hey, it makes for a nice article with a progress bar that animates as you scroll up and down.

The papers:

https://neuroscience.berkeley.edu/wp-content/uploads/2016/05...

https://redwood.berkeley.edu/wp-content/uploads/2022/05/kane...

If they linked the papers, it would be too obvious how poorly they described what they were about.

But they are not transparently symbolic.

They are not but there’s a lot of lifting being done by assuming a “hypervector” (an embedding, to literally everyone else) for a concept. Does such a separable embedding exist and do you have a good way to find it? Seems like one of those things where, oops, it’s just worse than gradient descent on massive functions.

Reply on news.ycombinator.com