Problems the AI industry is not addressing adequately

Of course, but that's part of my whole point.

Such statements and targets about how close we are to "AGI" has only become nothing but false promises and using AGI as the prime excuse to continue raising more money.

Another way to say it is that people think it’s much more likely for each decent LLM startup grow really strongly first several years then plateau vs. then for their current established player to hit hyper growth because of AGI.

Microsoft itself hasn't said they're doing this because of oversupply in infrastructure for it's AI offerings, but they very likely wouldn't say that publicly even if that's the reason.

The other reason is because the primary focus of the last 3 years has been scaling the data and hardware up, with a bunch of (much needed) engineering around it. This has produced better results, but it can't sustain the AGI promises for much longer. The industry can only survive on shiny value added services and smoke and mirrors for so long.

Last week I had Claude and ChatGPT both tell me different non-existent options to migrate a virtual machine from vmware to hyperv.

Week before that one of them (don't remember which, honestly) gave me non existent options for fio.

Both of these are things that the first party documentation or man page has correct but i was being lazy and was trying to save time or be more efficient like these things are supposed to do for us. Not so much.

Hallucinations are still a problem.

Nonsense, there is a TON of discussion around how the standard workflow is "have Cursor-or-whatever check the linter and try to run the tests and keep iterating until it gets it right" that is nothing but "work around hallucinations." Functions that don't exist. Lines that don't do what the code would've required them to do. Etc. And yet I still hit cases weekly-at-least, when trying to use these "agents" to do more complex things, where it talks itself into a circle and can't figure it out.

What are you trying to get these things to do, and how are you validating that there are no hallucinations? You hardly ever "hear about it" but ... do you see it? How deeply are you checking for it?

(It's also just old news - a new hallucination is less newsworthy now, we are all so used to it.)

Of course, the internet is full of people claiming that they are using the same tools I am but with multiple factors higher output. Yet I wonder... if this is the case, where is the acceleration in improvement in quality in any of the open source software I use daily? Or where are the new 10x-AI-agent-produced replacements? (Or the closed-source products, for that matter - but there it's harder to track the actual code.) Or is everyone who's doing less-technical, less-intricate work just getting themselves hyped into a tizzy about getting faster generation of basic boilerplate for languages they hadn't personally mastered before?

A few days ago, I asked free ChatGPT to tell me the head brewer of a small brewery in Corpus Christi. It told me that the brewery didn't exist, which it did, because we were going there in a few minutes, but after re-prompting it, it gave me some phone number that it found in a business filing. (ChatGPT has been using web search for RAG for some time now.)

Hallucinations are still a massive problem IMO.

It'll be Vaswani and the others for the transformer, then maybe Zelikman and those on that paper for thought tokens, then maybe some of the RNN people and word embedding people will be cited as pioneers. Sutskever will definitely be remembered for GPT-1 though, being first to really scale up transformers. But it'll actually be like with flight and a whole mass of people will be remembered, just as we now remember everyone from the Wrights to Bleriot and to Busemann, Prandtl, even Whitcomb.

I know people who've taking quite good comp from startups to do things that would require fundamental laws of physics to be invalidated; they took the money and devised experiments that would show the law to be wrong.

Hanging a queen is not evidence of a lack of intelligence - even the very best human grandmasters will occasionally do that. But in pretty much every single video, the LLM loses the plot entirely after barely a couple dozen moves and starts to resurrect already-captured pieces, move pieces to squares they can't get to, etc - all while keeping the same confident "expert" tone.

At that point, the question of whether the model really does understand is pointless. We might as well argue about whether humans understand.

From a purely practical standpoint, we don't know of any non-computable physical systems and it's just so painfully god-of-the-gaps to say "The brain could contain new physics that transcends everything we know even though this has never proved true with any other complex system we ever gained knowledge about. It's all proved computable".

We're also pretty good at working around human 'hallucinations' and other inaccuracies. Whether it be someone having a bad day, a brain fart, or individual clumsiness. eg in a (bad) organisation, sometimes we do it with layers of reviews and committees, much like layers of LLMs judging each other.

I think too much is attached to the notion of "we don't understand how the LLM works". We don't understand how any complicated intelligence works, and potentially won't for the forseeable future.

More generally, a lot of society is built up from empirical understanding of black box systems. I'd claim the field of physics is a prime example. And we've built reliable systems from unreliable components (see the field of distributed systems).

many things are unpredictable on the real world. Most of the machines we make are built upon layers of redundancies to make imperfect systems stable and predictable. this is no different.

You need to send data, but only 50% at random comes through. You could move things closer together, send at higher power, improve the signal modulation to give you higher snr.

All of these things are done, this is improving the base system. But at the end of the day rf is in the real world, and the real world sucks. Random shit happens to make your signal not come through.

So what do you do? You design fault tolerant systems.

You add error correction to detect and automatically fix errors on receive.

You add tcp, to automatically retransmit missed packets.

You add validation to make sure that the received data is sane.

You use ecc ram to validate that an ion from the sun has not bit flipped your data.

These are extremely complex hierarchies of systems, and fundamentally they are unreliable. So you design the whole thing with error margins and fall back handling.

Yes a better understanding of the underlying system in question does allow you to make more efficient error correction/validation, but it does not change the fact that you need error correction.

And in the case of RF signals, for example, the most optimal design is not zero errors. In fact they way you design it is so that for a given snr, you expect a given probability of error. To get the maximum throughput, you design your error correction coding to handle that. And because even with that, there can be edge cases, you design higher level Mac layer logic to resend.

Yes, a better understanding of the error cases will make the error correction (agent loops) in llms better over time, but it will not remove the error correction.

there may be situations similar to RF, where a less accurate, but faster model, with more validation is preferred for a variety of engineering reasons, either throughput/cost/creativity/etc.

You need to look at llms as black boxes of the physical world, that we question, and get a answer. At the level of complexity it is more akin to physics, than software, there is no reliability inherent to them, only what we design around.

And there are tons of people doing the fundamental physical research into how they work, and how to make them work better.

But that is a completely different avenue of research from how to make useful systems out of unreliable components.

You do not need to fully model the path propagation of em waves in order to make a reliable communication system, it might help, but it will be fragile if you solely rely on it.

And if you engineer a reliable system architecture, it does not get fully invalidated when the scientists build better models of the underlying systems. You may modify the error correction codes but they do not go away.

For example the original Morse code, had hand handmade validation checks, handshakes etc. the architecture did not get wholly replaced with the advent of Shannon's information theory, even if some of the specific methods did.

And WW1/WW2 telegraphs, and communications were still useful despite not understanding the underlying information theory and being unreliable in many situations.

The numbers would bankrupt them within weeks.

If that’s the case, then the gulf between current techniques and what’s needed seems knowable. A means of approximating continuous time between neuron firing, time-series recognition in inputs, learning behavior on inputs prior to actual neuron firing (akin to behavior of dendrites), etc. are all missing functionalities in current techniques. Some or all of these missing parts of biological neuron behavior might be needed to approximate animal intelligence, but I think it’s a good guess that these are the parts that are missing.

AI currently has enormous amounts of money being dumped into it on techniques that are lacking for what we want to achieve with it. As they falter more and more, there will be an enormous financial interest in creating new, more effective techniques, and the most obvious place to look for inspiration will be biology. That’s why I think it’s likely to happen in the next few decades; the hardware should be there in terms of raw compute, there’s an obvious place to look for new ideas, and there’s a ton of financial interest in it.

Things I think will be hard for LLMs to do, which some humans can: you get handed 500 pages of Geheimschreiber encrypted telegraph traffic and infinite paper, and you have to figure out how the cryptosystem works and how to decrypt the traffic. I don't think that can happen. I think it requires a highly developed pattern recognition ability together with an ability to not get lost, which LLM-type things will probably continue to for a long time.

But if they could maths more fully, then pretty much all carefully defined tasks would be in reach if they weren't too long.

With regard to what Touche brings up in the other response to your comment, I think that it might be possible to get them to read up on things though-- go through something, invent problems, try to solve those. I think this is something that could be done today with today's models with no real special innovation, but which just hasn't been made into a service yet. But this of course doesn't address that criticism, since it assumes the availability of data.

The reason I believe it can be achieved in this time frame is that I believe that you can do much more with non-output tokens than is currently being done.

so these arguments by fundamental distinctions I believe all cannot work--the question is how new are the AI contributions. Nowadays there's of course still no theoretical breakthroughs in mathematics from AI (though biology could be close!). Also I think the AIs have understanding--but tbf the only thing we can test is through testing on tricky questions which I think support my side. Though of course some of these questions have interpretations which are not testable--so I don't want to argue about those.

In addition, with your argument, should you not legalize all drugs in the quest for maximising profits to a select few shareholders?

AFAIK, the workings of addiction is not fully known, I.e. it’s not only those with dopaminergetic dispositions that get ”caught”. Upbringing, socioeconomic factors and mental health are also variables. Reducing it down to genes I fear is reductionist.

You are saying suffering is allowable/good because eventually different people won't be able to suffer that way. That is an unethical position to hold.