Hacker News Reader

>weird - even if AI was literally omnipotent and omniscient, you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do.

one would hope that one ability of an 'omniscient and omnipotent' AI would be greater understanding.

When speaking of the divine (the only typical example of the omniscient and omnipotent that comes to mind) we never consider what happens when God (or whoever) misunderstands our intent -- we just rely on the fact that an All-Being type thing would just know.

I think the understanding of minute intent is one such trait an omniscient and omnipotent system must have.

p.s. what a bar raise -- we used to just be happy with AGI!

That’s because gods are a mythical/supernatural invention. No technology can ever really be omniscient or omnipotent. It will always have limitations.

In reality, even an ASI won’t know your intent unless you communicate it clearly and unambiguously.

> In reality, even an ASI won’t know your intent unless you communicate it clearly and unambiguously.

I recently came to this realization as well, and it now seems so obvious. I feel dumb for not realizing it sooner. Is there any good writing or podcast on this topic?

The communication I get from customers is seldom clear and never unambiguous but I’ve managed since the 90’s

Right, but you have to do a lot of work, and really most of your work is in this area. Less on the actual building stuff.

Figuring out what to build is 80% of the work, building it is maybe 20%. The 20% has never been the bottleneck. We make a lot of software, and most of it is not optimal and requires years if not decades of tweaking to meet the true requirements.

Not really a bar raise - many people have assumed that "AGI" would mean essentially omnipotent/omniscient AI since the concept of the technological singularity came into being. Read Kurzweil or Rudy Rucker, there's a reason this sort of thing used to be called the "rapture for nerds."

If anything I've noticed the bar being lowered by the pro-AI set, except for humans, because the prevailing belief is that LLMs must already be AGI but any limitations are dismissed as also being human limitations, and therefore evidence that LLMs are already human equivalent in any way that matters.

And instead of the singularity we have Roko's Basilisk.

Genies, maybe? They are omnipotent and (generally) sufficiently aware of your desires that they shouldn't actually get "confused". Genies are tricksters that will do their absolute best to fulfill the letter of your wish but not the meaning.

The problem goes deeper: verification is harder than generation. When writing an answer yourself, you build the logic chain from scratch. When verifying AI, you have to deconstruct its logic, cross-reference facts, spot hidden hallucinations, and only then approve. For complex cases (which are exactly what the humans were left with), the time for quality verification approaches the time to write from scratch. If the time becomes roughly equal, the AI stops being an accelerator and becomes just a source of noise that yields no productivity gains

Move fast and break things. When a black box can be blamed, why care about quality? What we need is EXTREMELY strict liability on harms done by AIs and other black box processes. If a company adopts a black box, that should be considered reckless behavior until proven otherwise. Taking humans out of the loop is a conscious decision they make therefore they should be fully responsible for any mistakes or harms that result.

Shhhhh that’s a primary unspoken feature - lack of responsibility

>you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do.

this sort of assumes that most humans actually know what they want to do.

It is very untrue in my experience.

Its like most complaints I hear about AI art. yes, it is generic and bland. just like 90% of what human artists produce.

I've always found it much quicker to just... do the work myself. AI slows me down more than anything.

fair. I used to think that too, but I find at least for golang, the sota models write tests way faster than I would be able to. tdd is actually really possible with ai imo. except of course you get the scaffolding implementation (I haven't figured out a way to get models to write tests in a way that ensures the tests actually do something useful without an implementation).

Your final sentence is interesting. I'm not a strict doctrine adherent, but in TDD, don't you write some minimal test, then implement the system to pass the test?

yes, but I find it hard to constrain it to a minimal implementation. what usually happens is it writes some tests, then an implementation, and then according to the thinking, makes some modification. it works with a relatively precise prompt, but starts to go a bit off the rails when you say things in broad terms ("write tests to ensure concurrency works, and the implementation to ensure said tests are correct")

> even if AI was literally omnipotent and omniscient, you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do

no no no you don't get it, you would have ANOTHER AI for that

You're being sarcastic but if I "LLM as judge" one more time, I might jump off a bridge.

Also, it does appear that there are companies willing to YOLO themselves off a cliff with AI

It's not even about humans "needing" to be in the loop, but that humans "want" to be in the loop. AI is like a genius employee who has no ego and no desire to rise up the ranks, forever a peon while more willful colleagues surpass them in the hierarchy.

Until AI gets ego and will of its own (probably the end of humanity) it will simply be a tool, regardless of how intelligent and capable it is.

Humans need to be in the loop for the same reason other humans peer review humans pull requests: we all fuck up. And AI makes just as many mistakes as humans do. They just do so significantly quicker.

Yes, "Mecha-hitler" has no aspirations. /s

This is the opposite of both what the article is saying, and reality