Ask HN: Is it hopeless to try to build new foundational models now | Hacker News Reader

Ask HN: Is it hopeless to try to build new foundational models now

1 pointnutanc6 comments

Sama says in an answer to a question(What should Indian startups do to build foundational models?) that it is hopeless to try to build foundational models that can compete with OpenAI. What does the community think? Is OpenAI too far ahead? Is there a possibility of some new tech coming which can displace OpenAI?

We are betting on the fact that such new tech is possible. Just wanted to get the feel of the community in this regard.

View on HN

I think it is very much possible, especially in economies like India and China. If some business cases for Chat GPT are realized, I expect competitors to surface.

The a difficult part is probably getting high quality training data. And there may very well be a barrier where improvement of the model stops and completely new strategies must be devised.

Sure, you would need significant investments to start such a venture and success is not guaranteed. It is nothing a garage shop could do.

It's not hopeless, but it requires so much money and time that you might as well surrender all hope. Competing with OpenAI is tough, finetuning llama to a point that it competed with GPT3.5 required several finetuning attempts and many months of research.

There's still room for more AI models, but competitive ones are remarkably difficult to build. Competing with OpenAI's thousands of GPUs is not an easy feat.

What if we have an alternative to deep learning that does not need those thousands of GPUs?

Then it would have to compete with OpenAI's solution, which is no easy feat. I'd be interested in hearing more about this alternative, though.

Have written a little about it here, https://gpt3experiments.substack.com/p/building-a-new-embedd...

Will do a bigger blog post soon on the approach. Can mail you some details if you are interested.

https://twitter.com/RajanAnandan/status/1666641010284449792?...

Reply on news.ycombinator.com