Hacker News Reader

Maybe. The problem is that you need billions to train new models.[1] At least with how things are now.

[1] https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-...

With 6-7 more doublings in compute power per watt, consumers will have the power of 1000 A100 GPU's in their iPhone. "Eventually" will come before I die, at least. That would happen outside of GP's time estimate but a moderately funded university consortium could probably afford it just a few doublings from now.

I think we'll reach architectural changes that make this moot before we reach hardware for it. The way we train these models is constantly in flux, and we just need someone to crack continuous learning so we can pass models around and train them en-masse, using the collective unused compute that is literally sitting on mine and everyone else's desk right now.

When considering future of tech, its valuable to consider there are at least two well-trodden paths:

1. semi-linear extrapolation of existing tech and progression (maturing tech)

2. new paradigms approaching the problem from a new angle or with new insight that invalidates or levels up past 1.

Since we're in the midst of a cambrian explosion for both 1. and 2. IMO I dont expect limitations as we've been seeing them will hold up even under the medium term.

Reply on news.ycombinator.com