Hacker News Reader

Google built transformers

talk all you want about it's an ad company and da did da, in the end the whole LLM revolution is built on their tech

The LLM revolution can be traced back to a 1996 paper from AT&T

you always build on top of something else and I'm not saying otherwise

fact still remains that Google came up with an impressive and transformative piece of technology

The groundbreaking paper from 96 did not lead to immediate advances because of the absence of computational power. This was then used by Hunton for the speech recognition paper which created the deep neural network revolution providing funding to create the transformers.

attention layer: core of transformer, was invented outside of Google.

I thought that was pioneered in the paper that came out of Google?

Attention was first proposed in this paper: https://arxiv.org/pdf/1409.0473.pdf and it is referenced from transformer paper.

at this rate no one ever invents anything cause everything is built on math

that some of the elements came out of somewhere else doesn't change the fact that Google came up with an impressive and game-changing piece of tech

It was not impressive at that time, because didn't achieve any quality breakthroughs. Later adapters of transformers like GPT and Bert achieved it with impressive results.

Reply on news.ycombinator.com