Hacker News Reader

Comment by Planktonne | Hacker News Reader

Everyone involved in AI should have read The Library of Babel [1].

It's about many things, but perhaps the most relevant idea here is that no information matters without understanding. We could generate all possible knowledge, but unless someone--a human--can verify and understand it, it doesn't count. The cure for mortality could be written on the moon, but if no one reads it, it hasn't really been discovered.

[1] https://maskofreason.wordpress.com/wp-content/uploads/2011/0...

View on HN

> unless someone--a human--can verify and understand it, it doesn't count.

One counter-point: Humans have not figured out how general anesthesia works, but we use it every day to great effect.

https://en.wikipedia.org/wiki/Theories_of_general_anaestheti...

Seems a bit nitpicky, we certainly understand how to use it with great effect, we just don't understand the biochemical mechanism.

Only because we understand everything _except_ how it works.

how can we understand absolute everything and still don't know how it work ?

Trial and error :)

I wouldn't say that is "understanding". We must fully be able to explain to say that we understand it.

There are two types of understanding: knowing how a candle works and knowing how a candle works.

I find it hard to distinguish between knowing how something works, and being able to predict its behavior (including ways to create and destroy it) with probability approaching 1

The part that does count is that we can verify that we does what we want it to do. We just don't know how.

Same with the black box part of AI. What the weights represent? Arcane dark magic if you ask me. What do they do? Well with LLMs we're all experiencing it.

> The part that does count is that we can verify that we does what we want it to do. We just don't know how.

How do we know the person getting anesthesia doesn't die and a new soul/consciousness replaces them?

(We can ask the same question about going to sleep, er even walking through a door, but it's still something to think about).

This was my existential horror for most of my life as I almost never remembered dreaming or going to sleep - until I got a CPAP. Once I started using it, there was continuity to my life, I vividly dream and remember those dreams now. I never scored too badly on the apnea tests but just enough to qualify for one so I said what the hell and got it. Even though I don't use it regularly, something about my sleep has changed for the better.

Well generally we monitor their heart and lungs to see whether they died

you can verify it does what you want it to? How? Everything is a probability and so is your verification.

You're correct that verification *reduces* risk. It does not eliminate risk. Humans are imperfect. That does not mean that we should stop seeking understanding on principle though, because the journey towards that understanding improves outcomes.

Recall also that LLMs are not actually entities. In their current form, there is no sentience, there is no agency. They are tools. Therefore, their output must benefit the user that requested it. Right now, that's humans (and, ideally, the planet at large; we don't exist in a vacuum) and so it makes sense that humans should verify that output and try to ensure that it aligns with their goals.

That's not to say that the output of an LLM is useless, far from it. But we should still *try* to understand its output. It gives us at least some chance to notice flaws, and an even greater chance to appreciate the implications and tradeoffs of the solution it picked.

I don't disagree with anything you said. But the thread is about... LLM outputs, even if it cures cancer and solves world hunger, it is useless (it does not count for anything) if we don't understand it.

Interesting, I really don’t see your point. It would definitely be a lot more useful if we also understand it.

But extra lives saved definitely counts for a whole lot and is definitely useful.

One comparison I see: there are so many religious people that simply wish they are saved from their cancer (or whatever else). They pray for it. They don’t care how it happens. They simply want to live longer, etc.

It’s a sketch of an argument, I hope you know what I am getting at.

We have just convinced ourselves it is statistically safe, much like most of medicine. Most drugs are not made from molecular simulation of an entire human body.

Of course we have also convinced ourselves before that cocaine in drinks, lead in petrol, asbestos in walls… were all safe…

This reminds me of the Feynman interview where the interviewer asks "how do magnets work" and he goes on this rant of how that's unanswerable and you have to decide on what is it you really want to ask. You can't expect to know the full chain of knowledge because at some point you will be asking about quarks and gluons and then hit a wall where "nobody knows". Similarly you cannot simply just give up any investigation at all because then you'll end up recommending people to put lead in their cars.

My favorite part of Feynman's magnetism rant about how the question is terrible and can't be answered is that he seemingly accidentally gives a very good explanation: that magnetic repulsion is a highly concentrated version of what repels his hand from the arm of his chair.

While I get the point and it's true.

I'm a little more flexible, if the new knowledge (that human's don't understand) can be put into a mechanism and have an observable effect, I'd be happy enough. e.g. a new type of rocket fuel that burns 1000x more efficiently.

We already don't understand LLMs and they are having oberservable effects

If some LLM somewhere managed to produce a coherent and _correct_ explanation of how it worked inside, but no human ever saw it, would it matter?

We know how to apply LLms to problems, which is a subtly different thing to understanding how they do what they do. It's similar to fire: I can cook using fire, but I don't really understand how fire _works_. Heat+oxygen+fuel, sure, but what goes on chemically? I dunno. Doesn't stop me using it. (Pretty sure _humanity_ knows how fire works, though).

Doubt.

If I vibecode a video game and manage to sell it on Steam, the information definitely mattered even though I didn't understand any of it.

There are drugs that nobody truly understands how they work and they are being used by professionals in actual treatments literally right now. We know purely statistical facts like "if drug X is used for condition Y it will help Z% of patients" and can only speculate as to their mechanism of action. They are used anyway and still benefit a lot of people.

Yeah that information mattered when someone wrote it down, not when the LLM regurgitated an approximation.

The regurgitated approximation can matter just as much depending on what results it accomplishes. If I somehow make money out of this, then it's mission accomplished. Arguably, they might even matter more than my artisanal hand crafted software projects which never paid me a living wage.

Making money matters? Weird.

> but unless someone--a human--can verify and understand it

Why though a human? Mathematical proofs are very ivory-towery, but if OpenAI would solve a subkind of Cancer, without mortal humans understanding, cancer is still be healed.

Not unless humans were able to produce the medicine or perform the surgery or whatever is necessary to actually heal someone. In order for that to happen it has to be verified and understood by humans, like the other commenter said.

Does it, though? What if future AI can do everything from designing the cure to building a self-contained universal surgical medcapsule? "Just lay down and I'll fix your cancer real good, trust me, bruh." If it was shown to work, would you use it?

Well sure, if it's already proven why wouldn't I use it?

The AI we currently have can't do that though. The AI we currently have is a glorified chatbot. It can't fabricate anything and I doubt it could even reliably design simple real world devices.

It's a really cool and useful technology, and maybe one day it will be as good as you describe, but right now it just isn't and there's no guarantee it ever will be.

I don't think this necessarily follows. The point of the Library of Babel is more one of permutations/combinatorics than of knowledge accumulation itself. In 'our' Library of Babel, each book would be informative, if not perhaps flawed in some ways. That's quite different from one in which in which the number of books that contain anything coherent whatsoever cleanly rounds to zero.

Update:

"I own nothing, have no privacy, [never have to think, and am not required to solve any problems,] and life has never been better."

https://en.wikipedia.org/wiki/You'll_own_nothing_and_be_happ...

Some things can simply be beyond humans, but what if an AI could understand it? Just because we don't understand something, it does not mean that it does not matter.

Ironies of automation might be better suited to explain the problem.

> but unless someone--a human--can verify and understand it, it doesn't count

Does it need to be a human or can it be some other form of life?

It can be other form of life, but that other form of life need to be extremely aligned and keep humanity interest as top priority.

But we have pretty good understanding of the logical foundations and the truth of computer generated math. So having a big library which humans could never produce or fully understand has still a value on its own. It is just another level of abstraction.

Reply on news.ycombinator.com