Comment by zer00eyz | Hacker News Reader

Comment by zer00eyz | Hacker News Reader

You sir get an F in history and the industry does too.

Does no one remember why expert systems fell apart? Because you have to keep paying experts to feed the beast. Because they are bound to the whims and limitations of experts. Making up data isnt going to get us there, we already failed with this method ONCE.

Open AI's bet with MS and the resignation of all the safety people says everything you need to know. MS gets everthin up to AGI... IF you thought you were close, if you thought that you were going to get there with a bigger model and more data then you MIGHT want MS's money. And MS had its own ML folks publish papers with "hints of AGI", The google engineer saying "it's AGI" before getting laughed at...

I suspect that everyone at OpenAI was high on their own supply. That they thought AGI would emerge, or sapience, or sentience if they shoved enough data at it. I think the safety minding folks leaving points to the fact that they found the practical limitations.

Show me the paper that has progress on hallucination. Show me the paper that doubles effectiveness and halves the size. These are where we need progress for this to become more than grift, than NFT's.

> Does no one remember why expert systems fell apart?

Many of the current generation of AI experts mostly either did not pay great attention to the history of AI or they believe this time is completely different. They would do well to spend more time learning about history.

However, your view doesn't strike me as correct either. Expert system fell apart because the world was more complex than researchers realized and enumeration was essentially discovered to be infeasible (more or less as you say). But the impossibility of enumerating the world isn't news, everyone knows "the bitter lesson". And this isn't the past - now everyone on earth carries around a computer, a video camera and a microphone. They talk to each other through the internet. Remote workers screens' are recorded. Billions of vehicles with absurd numbers of sensors are roaming around the world. More of the arenas that matter to humanity are digital and thus effective domains for automated exploration and data generation.

The information about how the world operates exists or can be generated, the only real question is how to get your hands on it.

> The information about how the world operates exists or can be generated, the only real question is how to get your hands on it.

I'm sure I could read all the information for an astrophysics course in a relatively short time. Understanding it is a different matter.

Understanding is a loaded term. But large transformers seem pretty good at learning from datasets (of more or less any modality) to they extent that they can create useful new datapoints and allow you to work with existing datapoints in useful, structured ways.

Transformers main usefulness is translation, including translations from technical lingo to natural languages and vice versa, or code to natural language etc. That was their targeted usecase when originally made and that is what most people use them for. This is what you use them for that doesn't require hallucinating. They don't understand so they can't replace technical experts, they can just translate a bit back and forth but you still need an expert who understands the domain since translations isn't enough for a layperson to replace an expert.

The other use case is to make them hallucinate tropey stories and concepts for brainstorming, this isn't nearly as useful though due to it adding so much low quality stuff here when it hallucinates.

> The information about how the world operates exists or can be generated

The hubris of mathematics. At what scale does whether prediction become 100 percent accurate? How large of a model do you need, and how big of a computer to run it?

Do we thing that reducing the world to a model and feeding it through (what isnt even close to a model) of "thought" or "interaction" or ... what ever you want to bill and LLM as is going to be any more accurate than weather prediction?

100 percent accurate will never happen, nor does it need to. But think about the intelligence of an average human. Can we beat that? At least along some collection of concrete axes, enough to create a form of intelligence that can rightfully be called general?

It remains to be seen, but the days where I scoffed at that idea are firmly in the past where they belong. Today we are building machines with intelligence high enough that it is forcing us to reconsider and redefine what intelligence is. And there is a huge amount of progress just sitting in front of us, waiting to be fed into models.

Even AGI as a whole is overhyped. It's a valuable goal, but AI that beats humans on narrow metrics is still economically valuable because of scale.

Are you arguing that weather prediction isn't very good right now? Within a day or two it's very, very good, no?

>> Does no one remember why expert systems fell apart?

There were many reasons. One of them was the "Knowledge Acquisition Bottleneck", but that was not about the cost of paying experts, rather the cost of creating and maintaining a potentially very large knowledge base (i.e. one big mother of a database of production rules). Also, the fact that many experts' knowledge is tacit and not easily formalisable.

Modern machine learning began in the 1980's as an effort to overcome the Knowledge Acquisition Bottleneck. Accordingly many early machine learning approaches were designed to learn production rules for expert systems. Decision trees, one of the staple classifiers in data science, come from that era; you can tell, because decision trees are a symbolic, logic-based "model".

There were other problems with expert systems, e.g. their infamous "brittleness". But modern, statistical machine learning systems, are also criticised for "brittleness" too.

There were also purely political reasons and nothing to do with science or technology considerations. Then there was the 5th Generation Computer Project, and the AI winter, and then there were no more expert systems.

The journal of Expert Systems with Applications is still alive and well, on the other hand, although it mostly publishes on machine learning and neural nets these days. With an occasionally cool article, like one about Wolf Colony Optimisation I spotted recently. Too tired to look for links now, sorry.

They aren't going to show you any papers at all, they like money.

Experts weren’t the bottleneck on expert systems it was the systems weren’t particularly adaptive, were too rigid, weren’t able to make abductive conclusions, and the user interfaces were way too difficult in situ. LLMs actually tackle quite a lot of these issues FWIW but I wouldn’t look at them as a replacement for expert systems. Instead they’re probably what will make them useful by providing a natural human interface and a way of providing an abductive “reasoning” ability ontop of traditional expert systems.

I've seen some infographics that shows LLMs practically need to see same data 4 times or less, and once is fine too(trained for one epoch).

And I was like, y'saying, it's a zipped list of edge cases...

Its a zip with lossy compression for text. First useful lossy text compression algorithm we have made.

It was trained that way, so it would be weird if it wasn't.

>Show me the paper that doubles effectiveness and halves the size.

LLM's have pretty clearly been the most rapidly advancing technology in the history of humankind. Are you not entertained?!

Reply on news.ycombinator.com