AI Is Smoke and Mirrors

That's a lot of words from someone who doesn't seem to have tried using these tools yet?

The Hype is smoke and mirrors, but there are tons of real use cases where things were MUCH easier than before. Things like text to speech or image recognition previously required a team of 10 ML engineers a couple of years to build now take a weekend.

It's smoke and mirrors to the "business world" because none of these translate into bottom line numbers though, and I think that's all this guy is able to see

No matter how much the chattering class wants to portray this as a "fad" and not an inevitability, those of us who can type and read can see the reality for ourselves.

The usual pattern is to overestimate short-term change and underestimate long-term change.

We're a year since this stuff really got popular, and so I think a thing that's happening right now is many people noticing the short-term changes were overestimated.

There is the old quote “we overestimate the impact of technology in the short-term and underestimate the effect in the long run”. I watched this with the internet. First it was a toy (before maybe 1991}, then it was massively overhyped (1992-2001), then disillusionment kicked, in the end it transformed a lot of business, maybe not in the way predicted but the impact is undeniable. Same with AI, right now we are probably at the 1992 internet stage. There will be a lot of wrong paths but it’s pretty clear this will have a massive impact.

People were right to be skeptical about blockchain and cryptic, but AI is on a totally different level of usefulness.

IMO it's not an entirely unreasonable take.

It seems some companies adopting GenAI at this stage are doing so, at least in part, because they want to believe (that they can replace workers), rather than out of any sober analysis of what it can actually do. Maybe there's an element of FOMO too, and companies wanting to use GenAI because they hear everyone talking about it.

No doubt LLMs will continue to improve, but what remains to be seen is if there will be a direct path from LLMs to AGI (which is where the real value gets unlocked), or if we'll just continue to see quantitative improvement in benchmark scores, hallucination reduction, etc, but not much qualitative change in the types of task they are capable of.

I do think that AGI is inevitable, maybe not even that far off (but certainly not next 5 years, probably not next 10 years *), but there's a lot missing from LLMs to get there, and it seems that at least one critical piece, online learning, may require a different approach.

* Note that it's already been 7 years since the transformer paper came out, and all we've really seen since then is a bunch of engineering work in making them more efficient and how best to train them. We haven't yet seen any advances in "cognitive architecture", or even any widespread recognition that there is a need to do so. If all people are doing for the foreseeable future is building pre-trained LLMs, then that is all we will get.

> a bombshell scientific paper recently cast serious doubt on AI developers’ ability to continue to dramatically improve their models’ performance.

This is the problem with this type of (AI isn't real, I promise!) writing. Each and every single time. This bombshell scientific result that he links to? Is Gary Marcus lamenting about a single paper, specific to a single architecture, which primarily shows how quickly algorithmic improvements are bringing down energy costs[0].

We get it. You don't like AI. But you don't have to lie about it, unless of course you are Gary Marcus and this is how you talk your book and earn your speaking fees.

[0]: Algorithmic progress in language models. https://arxiv.org/abs/2403.05812

This article, like so many AI hit pieces, fails to make any credible claim against AI. This one spent far too long propping up a tenuous analogy to smoke and mirrors.

Is current-gen AI valuable? Obviously. Look at how many people say as much and willingly pay for the services.

Is current-gen AI worth the hype? No, not really. Current models are impressive but highly limited, as any user can attest (and as any hater will focus on).

What the naysayers always miss is exponential growth. They don’t look towards the future. Nvidia’s stock is high because of hype, yes, but also because the market is anticipating huge advancements in model ability. Current models are limited, but future models are world-changing, and we have every reason to believe models will continue to scale in this way.

Just you wait for GPT-5.

I think the AI itself is working extremely well. The state-of-the-art LLMs are exceptional at many real-world tasks and are deployed in many production code bases successfully.

The more recent AI failures (like the humane pin) are at least partially a hardware problem. These LLMs require huge amounts of compute to do nontrivial tasks, and a wearable just can't run locally with the current architectures.

Machine learning however works, and generates hundreds of billions of dollars a year.

Both can be true!

The Problem is that it isn't smoke and mirrors.

People see legitimately novel and exiting new things computers can do. The Problems is that those things are extrapolated into the future based on insanely optimistic predictions.

Up until now every single innovation has had times of rapid advances and adoption followed by a slowing down and stagnation. Believing that artificial neural networks aren't subject to that trend is, again, extremely optimistic.

The greatest folly, I believe, is people who look at arbitrary problems and imagine AI solving them. This will not happen. Yet I see that thinking again and again. Artificial neural networks are good at certain things and bad at others.

OpenAI has succeeded massive on making people believe that it can continually improve on basically every problem at an accelerating rate. If that is true, it is obviously the most valuable company in the world. But if artificial neural networks behave like any other innovation there will be a point at which implementation is impractical and improvements will be limited.

Clearly AI (re-skinned ML) is following the standard Gartner hype cycle graph.

IMO, the struggles with AI in robotics exposes where the current AI bottlenecks are more clearly.

Specifically, we don't have a training data corpus for any of our AI robots, enough to help them perform generalized task behaviors. So we have to go out and create them / use non training techniques, which crucially cannot leverage the existing learning / movements of humans accomplishing things.

ChatAI's training data of "generalized text on the internet" mirrors exactly what it can do right now — know a bit about everything. But there lacks computer-readable corpuses for the things that are truly useful — 1) efforts that lead to a task accomplishment and 2) better-than human efforts that lead to better-than-human output.

The key question is this: can AI reach beyond the human dataset that is currently powering it, crucially, at low(ish) cost? Or will AI always be fed training data? If we think the former, then we have to admit our algos aren't quite there yet and this iteration of AI is selling that dream.

But even with our current data-fed AI algos, we have not yet put a lot of effort into the root of the challenge - how to quickly build large sets of high quality training data. I think there can be optimization here (over a period of 10-20 years) on a similar scale of going from million dollar computer mainframes to hundred dollar laptops. If high quality training data for specialized efforts were indeed available at a low cost, then the AI revolution will effectively deliver the dream it promises - just wait for Moore's law.

The first step to solving that is an extremely polished LLM. You can see the proof of concept with SORA and other image-to-text efforts on how quality training data is created with the help of LLMs.

But of course, "LLM generated great training data" doesn't draw headlines, so it may just look like nothing is happening for a long, long time, until you see AI task bots pop up everywhere.

Look, when I can dump a function into GPT4 and tell it to "refactor the following function with an emphasis on maintainability and testing" and it does a pretty damn good job of refactoring it, I have a hard time believing that this is "smoke and mirrors".

>AI’s smoke and mirrors moment - Because of course, AI is not inevitable.

Long term it seems pretty inevitable to me. Maybe the current LLMs are overhyped and there will be an investment slump before the next improved algorithm but long term it's here to stay.

It's probably a bit like the early days of electical power. Some of the early gadgets were no doubt rubbish but electrical stuff was there to stay and become a normal part of the world.

I can't agree anymore. After trying out udio.com my mind is completely blown at what I thought was possible.

I do think they are right about AI corrupting infrastructure of the internet, Google is pretty much useless if you aren't trying to buy a product and is full of SEO'd to hell AI generated content that is just terrible or completely lacking information.

It has made job seeking a pain in the ass and is leading to tools of manipulation on every side, nothing is real, everything is a grift now seeking short term gains while playing scorched earth with everything long term.

It seems hard for people to take a nuanced approach that GPT-4 level models have in the present the potential to improve many people's lives and corporations' bottom lines while still being cautiously pessimists on the next generation of models.

GPT-4, Claude 3 & Co. are simply too useful for certain coding tasks or to review a contract. Obviously, you need to understand you're dealing with a probabilistic being, and for many tasks it isn't the correct tool, but I use ChatGPT ~5 a day and the $20 are super well spent. Now, there's an ocean apart of me liking to use ChatGPT and the promises from Silicon Valley and Microsoft.

"Please write the first couple sections for an article based exclusively on the content below. Be direct and concise, but dive into details. Use a personal tone with “you” language. Write in the style of blogger Brian Dean. Use short paragraphs for easy readability. Change the wording so we don’t plagiarize the source material. Make the headlines somewhat catchy and intriguing while still concisely communicating what each section is about.

- AI, also known as artificial intelligence, is overhyped - AI will never be able to replicate what a human can do - AI will never replace humans - AI is a trend that will die out - AI is not really artificial intelligence, just a trick"