Hacker News Reader

Comment by leetharris | Hacker News Reader

> I also wonder - is the compute the main limiting factor today ?

So much of this is a blackbox that you have to wing it for lots of things and try stuff. The more compute you have, the more YOLO runs you can do.

For example, the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage because they can afford to train lots of random ideas and see what kind of advantage they get. We have to be much, much more measured and predict our outcomes better.

View on HN

> the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage

The search space could easily be so large that a less disciplined approach might yield fewer useful outcomes even with the compute advantage. Being forced to be focused and disciplined might actually be a big advantage to you.

It’s hard to be disciplined about a black box though. That’s one reason why we’re all speeding off at a thousand miles per hour on transformers - the architecture works, why try other things?

whoever comes up with the next trick could win big.

Could, or meta makes you irrelevant with by reproducing your trick and giving it away for free

Yann LeCun certainly seems to be one of the more interesting people in the space. He’s a notable skeptic of Llm intelligence and incredibly smart.

To be fair I don't think anybody saw the boom in LLMs coming from the initial Attention paper. At the time it was one of many ideas that sounded like they had potential.

Whoever chooses the next winning lottery numbers could win big...

that's kinda my point, we don't know what the next winning move will be.

Attention was invented because Bengio lab had to be disciplined about a black box (google had more compute)

What's the process of deciding what stuff you will try ?

> So much of this is a blackbox that you have to wing it for lots of things and try stuff

Isn't OP talking about this _blackbox_ of gains?

But this is highly disappointing too - there are speed gains and tradeoffs to be had at every blackbox layer, but instead of doing that and actually gaining these improvements across all future experiments, we do YOLO runs and don't garner them at all.

Reply on news.ycombinator.com