Special Projects (2016)
openai.com
openai.com
The next big thing is Robots and some stealth company building today is going to be a trillion-dollar giant in few years time.
Don't confuse AI with LLMs. "We" know about AI for a long time. We even have a term for when AI fails expectations, AI winters.
The point is, typically when something seems like it came out of nowhere, it just means you didn't dig deep enough. Ideas don't come at an instant, fully formed like Athene from Zeus' forehead. It's brick by brick, one twist on an existing idea and zeitgeist at a time.
Artificial Intelligence
|
+-- Symbolic / rule-based AI
| +-- expert systems
| +-- search / planning
| +-- logic / knowledge representation
|
+-- Machine Learning
|
+-- classical statistical ML
| +-- regression
| +-- decision trees
| +-- SVMs
| +-- Bayesian methods
|
+-- Neural Networks / Deep Learning
|
+-- computer vision
+-- speech
+-- Natural Language Processing
|
+-- Transformers
|
+-- Large Language Models
|
+-- chat systems
+-- multimodal models
+-- tool-using systems
+-- agents
But in the end, a lot of it was proven right, even if it wasn't quite in the way we imagined.. A lot of pre-LLM interpretations of AI imagine it as some sudden 0 to 100 breakthrough, like one day someone writes an AGI program in their basement and takes over the world with it. There's shades of that in here too, talking about worries of organizations secretly developing AI capabilities and needing to track public data for patterns to discover it. In the end there was a 'magic program' in the transformer, but it doesn't seem like they really foresaw how the program would be useless on its own, and the 'AI' would come from ingesting as much data as possible, a process that has built incrementally over years and been very much exposed to the public.
That's basically the bitter lesson. Academics only reluctantly swallowed that pill and still aren't satisfied with this answer. It's ugly and feels like it shouldn't work because intuition would say there are too many combinations, curse of dimensionality, etc. But it turns out it's just line go up, extrapolate Moore's law and don't worry too much about philosophical-level breakthroughs just count the flops and bits. Ray Kurzweil's scifi extrapolations turned out closer to the truth, whether deservedly or by luck.
I've really started changing my mind on this. "There is no AI, only I" is the position that I've really come to adopt.
Part of it is from Michael Levin's quote "Humans only can really see intelligence at human scales an immediately discount anything that doesn't exactly match their experience". The other part is most peoples immediate assumption that for something to be intelligent it has to be alive. Lastly is there may be platonic intelligence, that some kinds of intelligence may arise from the very structure of our universe when accessed.
We have really entered an age where thinking that human intelligence = intelligence is an anti-pattern that tells you less about the world and blinds you to what is actually occurring.
Biological intelligence, human intelligence, electronic intelligence, algorithmic intelligence are all subsets of intelligence set theory and even where humans like to call themselves a general intelligence it's distinctly likely that we're less generalized than we expect.
Also a lot of the vision and speech ideas cross pollinated with the NLP field. One big trend that enabled faster progress is bringing all this onto a common platform. First via Deep Learning and backprop, formulating everything as some vector input, some model architecture, some vector output, some loss, and then gradient descent optimization. This replaced the specialized optimization tricks people used to develop for their own little niche tasks. Before DL, papers usually derived their own math for how to solve their own specific formulation of a task, so it was hard to reuse ideas.
(Reuse was also hard because platforms like GitHub didn't exist, the Python ecosystem wasn't nearly close to what we have, code sharing wasn't as common, and anyway the code was some mess in MATLAB, not in a sane language.)
The second thing that allowed converging these fields was the transformer architecture that allowed turning everything into tokens and throwing it all into the same transformer architecture, making multimodal models that can learn from everything and do everything, instead of having to make specialized models for each little task.
IMHO, robots won't become a thing until they're running local AI, at which point they will become the thing that crushes humanity.
Even for domestic robots people don't care, see just about every cloud connected smart vacuum robot and the numerous data leaks
And for companies, most of them are fine using SaaS and cloud based hosting platforms, so why would it be any different for robotics?
Sure there is the latency argument, so at least something doing very short term decision making needs to run on device or at the edge, but for most task the end to end time is in the single didit seconds to minuites, so the extra 100ms or whatever that you get going out to some model in the cloud isn't a deal breaker.
That thread is a goldmine. Very humbling.
- superswordfish on July 28, 2016
This sounds rather like the plot to Greg Egan's "Crystal Nights" (2008)
Interesting that Anthropic led the way here.
Only one remains at OpenAI.
If you ask me, it'd be better if none of them remained at OpenAI.
Keep watching, then. I'm not so sure history will be quite as grateful as you seem to think. The future has a funny way of looking very different from what people expected when they were busy building it.
Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
https://web.archive.org/web/20260000000000*/https://openai.c...
I wouldn't view them as consistent at anything other then myopicly following the more or less obvious trends required to sustain LLM architectures over the years