What if Generative AI turned out to be a Dud?
garymarcus.substack.com
garymarcus.substack.com
20 years ago, I was doing computer engineering in Canada and neural networks were not sold as a dead end. It was made clear they were a very powerful tool.
This will out perform "a steak" for sale 365 days a year even if everyone else has the same kind of marketing drivel. in fact someone will probably buy both you and your competitors.
The Marvelous Ribeye: Where Flavor Meets Elegance
Exhibit A-->Prepare to be seduced by the enchanting symphony of textures and tastes that only a meticulously selected Ribeye can offer. Our expertly curated steak boasts marbling that weaves intricate pathways of rich, buttery fat throughout the meat, ensuring unparalleled succulence. Each mouthwatering slice unveils an orchestra of flavors, from the savory umami notes to the gentle whispers of natural sweetness.<--
adjectives sell...nouns tell...
And am being very charitable that that kind of writing works
That aside, we're obviously in a bubble, that's the easiest claim in the world to make. We're also in a real estate bubble, etc. The only claim that matters is predicting when it will pop, otherwise it's just hot air.
If you compare with smartphones then we are pretty much in the late 1980s brick phone era where the technology is just starting to be adopted by society.
The key technology, AGI, has not even been invented yet. It’s possible we may see a mini bubble if AGI takes a while to be developed, but I think it’s more likely that we won’t because even the current generation of AI has a lot further to run.
To clarify, I assume you are talking about Gen AI when you say "AI"?
In fact, it wouldn’t surprise me that Wall Street is already using it to trade some non-insignificant fraction of the US economy. Similar to how Linux is probably holding modern civilization together behind the scenes but most people are completely oblivious.
Every smartphone runs multiple AI systems in it. Some are used multiple times a day, most just working and keeping out of your way.
Unless you exclude from AI the stuff we used to call AI until it started working really well.
I understand generative AI as a million monkeys on a million typewriters, but we figured out how to rig the typewriters to generate convincing and sometimes correct output.
Neither monkeys individually or collectively understand anything about how they responded. They just get a treat for doing well.
AGI is something very different IMO. Language and speech will be emergent in AGI without requiring massive inputs, like how a child learns.
LLMs seem like sort of a (very impressive) parlor trick. Neat, useful and terrifying, but not a pathway to AGI.
ChatGPT was the largest growing site in the history of the Internet, reaching 100 millions users in two months. This seems more than 1980 brick phones
It’s disingenuous comparing the way Facebook scaled to ChatGPT which scales primarily along social media vectors with massive distribution.
For example I am currently doing self-directed study in maths. It turns out chatgpt is very good at explaining lots of things if you prompt correctly. You need to check things yourself and be on the lookout for plausible but wrong explanations, but it's like having a patient pure maths friend to work on things with - your friend won't get everything right either after all.
Something with real commercial potential is what microsoft released (I think today) [1] which is a sort of wizard thing that accellerates various azure gpt-powered standard deployments that do things like set up private ephemeral instances that connect to your enterprise data sources behind your firewall. This has enormous potential given how difficult it is in most large enterprises to actually join data sources together, find basic information etc. Think of an automated helpdesk that can draw on all of the stuff in all of the various disjointed systems in most big companies. That has the potential to be quite transformative to the workplace for a lot of people.
In general I think this bar of "transforming humanity" is just arbitrary and way too high for any technology. You could look at the map on your phone and say "that's all very well but will it transform humanity?" well no. It won't make toast either, but it's quite useful sometimes. Just as overinflated expectations and hype are unhelpful I think it's easy to be cynical and miss important tech developments just because we impose arbitrary standards on them that other technologies didn't have to meet on their way to adoption.
It's interesting that in the final matrix example it somehow messed up the Latex by forgetting to put \begin{bmatrix} at the front which is why the formatting goes bonkers. It's also kind of cheesy that it chose a diagonal matrix rather than having to find the roots of the characteristic polynomial but any case you get the idea hopefully and can see how this would be helpful when you're trying to learn this stuff yourself? I'm certainly finding it great as I work through textbooks etc to chat through concepts in this way to check my understanding and drill into things.
I get the impression that at one time he was trying to do AI (or similar) research, but it didn't involve neural networks. And ever since neural networks turned out to be a useful approach, because it wasn't the path he chose, he switched his career to putting down any AI with a neural network.
The most annoying thing is that they aren't very well written arguments and he doesn't come up with new ones, much less actual competitive alternative approaches to AI.
Also I think that we really do need alternatives to the giant black-box neural networks that are more predictable and auditable but also perform. Yet you never hear him talking about doing any such research.
It would be less disappointing if his background was as the owner of a 1990s-style furniture store in Queens. But supposedly he is a researcher.
However, I think he ignores some practical use cases. I am not a good example since I am retired (I limit myself to about 6 to 8 hours a week consulting, and I am always working on writing a book) but I feel that I am at least 25% more efficient when using tools like GitHub Copilot integrated into Emacs and using Bing+ChatGPT or Bard for bouncing ideas off of, and sometimes for search. I find that if I carefully express an idea that I am thinking about that I usually get something at least interesting to think about and sometime very useful.
I am excited by research on hybrid approaches to AGI, but also excited by what we have right now with OpenAI and Hugging Face APIs and models that can be self hosted.
I really find it to be more annoying than helpful. The mental burden of figuring out what the code is actually doing when a large chunk appears is exhausting, and it leaves me with a false level of understanding similar to the feeling of competency you get when you type out a tutorial's examples.
Underwhelming is how I'd describe my experience with ChatGPT too. There are so many people making claims of ridiculously improved efficiency, that I'm really unsure if it's me or not.
For instance, I’m often fighting incorrect array indices and slices, pulling out magic numbers into variables, decoding some one-liner completion, or figuring out what’s usable and what I need to cut out and replace in a larger block.
What’s really struck me is how much I really want Agda style hole-filling completion. I want to just be able to type a variable name and have it destructure cases for me. That happened a few times, and it was really nice. I’d forgotten how nice it was to be able to do that in Agda. I want that everywhere.
Also, just noticed who you were, I really enjoyed some of your Lisp books.
The issue I've seen with our latest ai is how we've barely been shown real world use cases as we used to see (ai playing games, ai iterate on design, ai learning music) I mean hell legally speaking we don't even know the ownership of ai's creation (as in the court hasnt set a precedent).
Until we get more of those kind of ais/robots, in a diverse set of fields, I think it'll just be powered by novelty.
if signal(ml_model) > noise_floor:
return "not dud"
elif signal(ml_model) <= noise_floor - epsilon and signal(ml_model) - signal(previous_ml_model) >= expected_improvement slope:
return "might show potential promise"
else:
return "not ready for years to come, or likely dud. unknown 'true' status from this point of measurement."