Hacker News Reader

> the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage

The search space could easily be so large that a less disciplined approach might yield fewer useful outcomes even with the compute advantage. Being forced to be focused and disciplined might actually be a big advantage to you.

View on HN

It’s hard to be disciplined about a black box though. That’s one reason why we’re all speeding off at a thousand miles per hour on transformers - the architecture works, why try other things?

whoever comes up with the next trick could win big.

Could, or meta makes you irrelevant with by reproducing your trick and giving it away for free

Yann LeCun certainly seems to be one of the more interesting people in the space. He’s a notable skeptic of Llm intelligence and incredibly smart.

To be fair I don't think anybody saw the boom in LLMs coming from the initial Attention paper. At the time it was one of many ideas that sounded like they had potential.

Whoever chooses the next winning lottery numbers could win big...

that's kinda my point, we don't know what the next winning move will be.

Attention was invented because Bengio lab had to be disciplined about a black box (google had more compute)

Reply on news.ycombinator.com