Hacker News Reader

Is there a fundamental difference?

I mean, the only thing GPT does is predict the next word, which makes it not so different from a compression algorithm. And diffusion models (the image generating stuff) are essentially fancy denoisers.

Depending on how you assemble the big building blocks, you get generation or you get prediction.

View on HN

GPT-3.5 is not a Markov chain, this is trivially true. While ‘predicts the next word’ is true, the mechanism of it is of interest and that is most certainly not trivial.

Depends how far you take the word 'fundamental', on the one hand yeah most DL systems are trying to predict something, and they generally have some concept of compression built in. But in terms of the steps to curate a dataset, train, test, iterate and actually use the model for a given end goal - they are pretty fundamentally different.

I think the thing is though in Large multi models you give it all the data and test it against everything. And it generally does better across most of the benchmarks.

That depends entirely on the use-case - for example if you wanted to build an AI to operate a self-driving car, just training on unlabelled data scraped from the internet is only going to get you so far. It doesn't learn how to do EVERYTHING (not yet at least).

Reply on news.ycombinator.com