I think text prediction only gets you so far. But I guess you could use the same principles to predict the next symbol in a binary string. If this binary data represents something like videos of physical phenomena, you might get the AI to profound, novel insights about the Universe just with next-bit prediction.
Hmmm, maybe even I could code something like that.
It would probably be insanely expensive. But I feel like it would be almost guaranteed to acquire a world model far richer and more robust than ChatGPT’s.
Human babies learn by watching the world around them. Video frame prediction feels much closer to that than text prediction, and given the wildly impressive results we are seeing with large text prediction models alone, it seems like an obvious next step.
While I understood what you meant, I'd just add that babies learn by a combination of multisensory triggers (so not only _watching_)
There are three objections to this.
The first is the astonishingly large amount of CPU power that this would take, given how high bandwidth video is. The second is that it is hard to believe that some thing really coherent could emerge from this, and certainly, it has never been shown. The third is that the world might be "richer" in terms of information-rich but seeing the world through YouTube's eyes would likely be a degrading and incoherent experience.
If you have the money, you can apply the transformer architecture to many different tasks and people are experimenting all the time. I think one of the big challenges is always to come up with methods for training such enormous models pragmatically without cost exploding.
[1] https://huggingface.co/docs/transformers/model_doc/canine
Sure, you encode bit-by-bit. But it is a fixed-length code, which is even worse than character-by-character.
Maybe you only get worse training and inference time. But I wouldn't be surprised if the encoding also serves as a Bayesian prior, and with a different encoding, you get worse results (for given data).
Language also follows very common and repeatable patterns. "Hello" is often followed by "How are you?", etc. Just like Zipf's Law dictates that some words are used exponentially more than others, there are linguistic and conceptual patterns that appear with predictable frequency. If your bits don't follow similar rules, the results might not be as clean.
I'm pretty sure you could code a transformer to work on binary or video data. Sounds like a great github project. But it's unlikely you'll have the scale of data to do anything close to ChatGPT.
Are you sure? See "the Pile":
The paper cited there by contrast, argues for select training sets:
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations.
Word sounds. I can not read without hearing the word. (Now I wonder about those born deaf.) Based on that subjective experience which I presume is rather universal among the hearing, tokenized phones and phonemes seem promising.
I do not hear words as I read. (As I write I do)
I'm also a whole word reader, it is IMHO generally faster than needing to mentally hear words.
Makes reading sci-fi with funky alien names much easier! :D
Also, non phonetic writing systems will push you towards not hearing while reading. For example, when I read chinese there are plenty of characters (hanzi) whose pronunciation I don't know (was it hai? Pai?), but I remember the meaning just fine.
Even with English (I'm not a native speaker) sometimes I won't remember how to pronounce a certain word, but I'll read it just fine.
Makes sense. I also tried an earlier suggestion to (effectively speed) read and that seems to send constant interrupts to the speech center -- which is how I am beginning to imagine how it works -- and it struggles to keep up and very soon becomes silent. atm still prefer my apparently pedestrian reading mode as it has a strong 'pleasure of reading' aspect that as of now I do not get when attempting whole word parsing.
How do you experience the pleasure of reading? For me, the reconstructed voice of the author and in general the (for the lack of a better word) musical aesthetics of the 'text' are significant.
Curious: What's your favorite piece of poetry?
I very much like Emily Dickinson, why? (What's your favorite piece of poetry?)
How much of this file's raw data would be encoding and metadata, vs the content of the movie?
It's a seductive thought to be able to just throw raw bits at a model, regardless of what those bits represent, and have it just magically attain LLM qualities in reproducing the data you would want it to.
Something to think about: GPT3/ChatGPT tokenize at the byte level. If they tokenized at the bit level the model would learn Utf8 encoding over time. Unicode characters that require more than one byte to represent, such as emojis, are not learned directly but the model can still reproduce them.
https://www.deepmind.com/blog/building-architectures-that-ca...
It wouldn't be "video model", it would be an "anything that can be expressed in binary" model.
> Perceiver: General Perception with Iterative Attention
Biological systems perceive the world by simultaneously processing high dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. Perceiver is a deep learning model that can process multiple modalities, such as images, point clouds, audio, and video, simultaneously. It is based on the transformer architecture and uses an asymmetric attention mechanism to distill a large number of inputs into a smaller latent bottleneck. This allows it to scale to handle very large inputs and outperform specialized models on classification tasks across various modalities.