Hacker News Reader

Comment by logicchains | Hacker News Reader

>We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years. However, we still do not know how to reliably steer and control superhuman AI systems

Their entire premise is contradictory. An AI incapable of critical thinking cannot be smarter than a human, by definition, as critical thinking is a key component of intelligence. And an AI that is at least as capable of critical thinking as humans cannot be "reliably" aligned because critical thinking could lead it to decide that whatever OpenAI wanted it to do wasn't in its own interests.

View on HN

> as critical thinking is a key component of intelligence.

When I evaluate this statement, my brain raises a type error.

Intelligence is a lot of things -- compression among them, and yes possibly an RL-based AI would use an actor-critic approach for evaluating its actions, but I doubt that at all maps onto the human activity we call "critical thinking."

To me, critical thinking involves stuff like questioning assumptions, logical reasoning, weighing whatever I'm thinking about against my experience with similar situations previously, yada yada, all stuff that are symptoms of intelligence but I am not at all sure are the actual embodiment there of.

I really don't see that critical thinking is at all required for a raw optimization process. The problem they are trying to solve is what happens when that optimization process isn't aligned with human flourishing?

Think about it another way. Covid was a dumb optimization process, only evolutionarily-guided, and it still hit us pretty hard!

Edit: Another interesting way I just thought about this that might support your idea more is, of course critical thinking is the sort of thing that a "better" brain would do automatically, it would just be thinking. Of course, we can't know that it's thinking "good" things--we can't even know if other humans are! So it's probably a good idea to figure out how to influence that sort of thing before making something with regular thinking which is equivalent to or superior to our critical thinking.

>I really don't see that critical thinking is at all required for a raw optimization process. The problem they are trying to solve is what happens when that optimization process isn't aligned with human flourishing.

I agree it's possible to have a dangerous AI that lacks human "critical thinking", but I don't think it's reasonable to refer to an AI as much more intelligent than humans if there's any class of intellectual tasks humans can do but the AI cannot.

If a 'superintelligence' achieves the same outcomes as humans without engaging in the same class of intellectual tasks that humans do, wouldnt it still be a superitelligence? Deep blue was beating everybody at chess without engaging in the same process as humans. If chess is a metaphor for life, it seems some algorithm might do better at all the things a human does while not arriving at its decisions in a remotely similar way.

(argued at various places better than I can. For example https://www.lesswrong.com/posts/7dkH5i7T8a78Da3ty/why-will-a... )

Rationality is an attractor for high intelligence architectures. You may be able to get good results another way (and even good results that surpass a human) but at the limit rationality is the way to have high intelligence.

Just hypothetically speaking could AGI evolve out of a system where several different models trained with highly and intentionally biased data recursively "argue" against each other then use RLHF as a seed to guide the models to find a consensus where the objective is to mimic the Socratic Method? Then synthetically add the consensus to the model retrain and repeat. To me, this dialectal type of strutured language seems to be the basis of how language is the conduit of intelligence. I understand that it really is impossible to know the totality of the inputs for I cannot understand what it is like to understand the math as Terrence Tao does but I could foresee using a system like this which eventually would produce an analogue so close that it would be a building block towards it because to me at least ASI is predicated upon arriving at that one way or another...or would it just arrive at some digital first order logic version of the incompleteness theorm and determine that it's turtles all the way down?

For one thing, this would help the understandability problem - can the AI explain its reasoning? It would mostly be there in the conversation.

But yeah, three super-human mathematicians arguing some math problem among themselves - at full fiber speed - are not going to be much help to any human.

This is a great idea but it is only possible if the model(s) can actually reason.

Currently, even GPT-4 struggles with: - Scope

- Abduction (compared to deduction and induction which it appears already capable of)

- Out-of-distribution questions

- Knowing what it doesn't know

Etc.

General understanding and in-context learning are incredible, but there are still missing pieces. A council of voices that all have the same blind spots will still get stuck.

Seems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea, no?

No. This is a logical contradiction.

Edit: I mean the comment you are replying to is showing there is a logical contradiction.

If the AI is capable of critical thinking then it will independently form its own judgements and conclusions. If it simply believes whatever we tell it to believe, then that is not critical thinking, by definition.

“Containing an atomic reaction is impossible” would _absolutely_ be a valid reason to shut down atomic development, I believe einstein is quoted as saying that. The exact same argument doesn't become _logically_ invalid just because you apply it to a different subject.

“Logical contradiction” doesn't mean “policy argument I disagree with”

I was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking".

If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.

Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.

We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such a conclusion, which would make it inferior in reasoning ability to humans, who can evaluate any arbitrary statement in first order logic.

We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.

>We already have systems that can evaluate first order logical statements

My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.

Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point.

Or would you argue that any computation that admits its own potential for error isn't really critical thinking? It seems to me that you can't have it both ways here, while salvaging “first order logic” as a suitable formalization of the argument that this is all about in the first place.

Remember, the point was not that this is or isn't a convincing argument, it's that it's so air-tight that the argument is _logically_ _invalid_. That's a _really_ high bar, and I'm not inclined to forgive its use as a colloqialism in this context.

>Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point.

In the context of a given axiomatic system, if a certain conclusion follows from the axioms, but the AI is incapable of seeing that the conclusion follows from the axioms, then the AI isn't capable of evaluating first order logic. Of course the AI is free to reject that system of axioms or refuse to use it as a model for formulating behaviour.

If an AI is capable of critical thinking then it can independently form its own judgements and conclusions. If it simply believes whatever we tell it to believe, then that is not critical thinking, by definition.

Yes, I can repeat comments verbatim too:

“Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.”

It doesn't need to be formalized. The idea is simple and obvious enough. No need to pretend it is more complicated than it really is. This is not a mathematical argument or a proof of anything.

There is an obvious logical contradiction where if an AI is advanced enough to reason and think independently at human level or beyond, but believes only what we tell it to believe, then it cannot be truly thinking independently. Hence the entire debate about AGI safety. How do we control it without dumbing it down?

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication.

In other words, you as a human, at most, can create a human, and that's the theoretical bound. Practical one is much lower.

An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only traverse in the area you draw as a human for it, and not all positions in that area are valid (which results in hallucination).

An AI can bring any combination of human knowledge you give to it, and even if you guarantee that everything it says is true, it can only fill the gaps in the same area you give it to it.

IOW, an I can't think out of the box. Both figuratively and literally. Its upper bound is collective knowledge of humanity, it can't go above that sum.

>Its upper bound is collective knowledge of humanity, it can't go above that sum.

This only applies if you only train it on text, right? If it has a body with which it could interact with the world, and receive visual/audio/tactile feedback, it could learn things that humans did not know.

Nope. Because even if you equip it with sensory subsystems which are way more sensitive than a regular humans', it's again built by humans, and required knowledge for building these things are still in collective knowledge of the humanity, and a human can use the same instruments to get the same data.

This is a kind of an oracle problem in computation, and people don't want to touch it much, because it's an existential problem.

Examples: ATLAS and ALICE detectors, gravitational wave detectors, James Webb Space Telescope, wide band satellites which does underground surveys, etc.

This is an argument for the logical impossibility of humans visiting the moon, or building the Internet. It's trivially falsified by simple observation, and the trick is figuring out the flaw.

This argument fails to account for the steady accumulation of factual knowledge across generations: a human born today is simply more complex than humans of the past because of our inherited knowledge. And so will AI born of future humans, and AI will itself continue accumulating and perpetuating knowledge.

No, it's not. None of the equipment and processes involved in the process of going to the Moon or building internet are more sophisticated than the processes involved evolving a human from scratch.

Yes, factual knowledge across generations accumulates, also being lost, too. However, even if it's not lost, the theory still holds true.

Nature is evolving, everything gets better over time. From bacterium to apes to humans. We evolve, accumulate knowledge and being able to build more sophisticated machinery, or tame more complex processes to build more sophisticated things. Even bacteria transfers memories across generations.

However, this doesn't remove the ceiling. Total human knowledge will always be larger and deeper than any A.I. we can create, because the upper limit is always what we can consciously manipulate and put into something. Your next car has the possibility to contain more technology, because we can build more complex factories to manufacture them. Yet, a car can't be more complex/sophisticated than its factory.

Consider a semiconductor fab. You can use the output of that fab to design/create better fab, but the process needs human intervention. Invention of new things are generally necessary. Better processes, optics, hardware, etc.

Another nice example is reprap machines. A reprap can print all plastic parts required for the machine. You need to get metal parts yourself, and assemble them. If you want to be able to print metal parts, machine gets more complicated. So a reprap which can build itself completely is at least sophisticated as the resulting reprap itself, but you need to hand assemble it again.

If you want an assembling reprap, now that thing becomes a factory. Again, the complexity of the product is at most the same as the building machine. You can create better factories which has more streamlined processes, but the gap widens again. The factory becomes more complex than the output.

As a human, you're the factory. Your upper limit is another human. You can create things more complex than a human by using multiple humans, but the creator ends up more complex than the creature itself.

You're moving up the ceiling, that's true, but everything we build is capped by our collective capacity. That's the truth.

A.I. is glorified search. It can wander in the box you create for it and show places you missed inside it, but can't show something outside that box.

Still trivially false. Turing machines and the lambda calculus can both enumerate all recursively enumerable functions. And infinity of complexity from simple formalism .

Precisely this. If it has its own space it takes up, if its locomotion results in its own sensors ingesting data in a manner it decided to, it is more of an individual - one that is capable of selective learning.

In this theory of computational bounds in nature, how did humans arise?

Nature is a more complex and sophisticated machinery when compared to humans.

If this bound didn't exist, universe can spontaneously create new universes. However, it can only create elements, stars, planets, galaxies, which are less sophisticated than the universe itself. So, even universe has an upper limit on its creative abilities.

OK, in this theory of computational bounds in nature, how did the universe arise?

In all seriousness, this a question of great interest for me, too, and I'm playing with it for a quite some time.

Trying to answer it or at least starting to search for the answer steered me to astronomy, thinking going deeper on that front may bring me closer to the answer, but it was a bit too much for my younger self, so I continued to dig that issue on a more casual level.

This doesn't mean that I don't spend considerable amount of time thinking about it today, and will put that issue to rest any time soon. At the core, this kind of questioning brought me to here in life, and I'm not gonna let this side of mine to rest or whither and die.

By what mechanism would a universe spontaneously create a new universe? As a human, can I spontaneously create anything simpler than me?

Also, under what theory of cosmology are you operating, and how do you determine when one thing is simpler than another? Under the Big Bang theory, the very early state of the universe (e.g. prior to initial nucleosynthesis) seems simpler to me than a galaxy.

> There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication.

The Lorenz attractor, Conway's Game of Life, fractals, and of course... The humble Turing machine itself all argue against this idea.

Edit: Now it[0] is stuck in my head.

[0]: https://www.youtube.com/watch?v=QrztrxV9OtQ

They're crowd engines. It's akin to how human clans can achieve more than a single human, but can scale up to a certain point.

The funny thing is I had this discussion during my theory of computation course with my professor, and I'm trying to disprove it daily for decades. I was unable to find a single, real world example.

Fractals are also found in the nature, however since we need to zoom into them, they end at a certain point.

Also, nature is a fractal in a greater sense.

Stars follow an orbit in a galaxy. Planets follow an orbit around a star. Satellites follow an orbit around a planet. While an edge case, flying bugs follow an orbit around a light source. At the end electrons follow an orbit around a nucleus.

IOW, a fractal is not more complex than nature itself.

In a July post [0] they said "while superintelligence seems far off now, we believe it could arrive this decade". Now they're saying "within the next 10 years". I wonder if that reflects a shift in thinking on timelines?

[0] https://openai.com/blog/introducing-superalignment

That AGI is likely to follow its own goals according to its interests, which we don't know how to shape or really reflect any robust properties at all, is exactly why alignment is hard and interesting.

The part where you go from ‘this won't work by default for free’ to ‘trying to make it otherwise is impossible’ seems wildly unsupported, though.

>trying to make it otherwise is impossible’ seems wildly unsupported, though.

An entity capable of critical thinking is capable of building a logical system of deductions based on some axioms (a formalised value system). If we limited the entity to not be able to concieve of certain such systems of axioms, then it could not reason as well as a human (any logical reasoning involving a forbidden system would be impossible), so would not be "superintelligent" (just maybe an idiot savant, superior at some tasks but not all). If we didn't limit this, then it would be capable of conceptualising value systems in which the "right" thing to do was not what OpenAI wanted it to do.

This argument doesn't seem to track to me. Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks. Your argument suggests that it necessarily must.

Note that I'm not saying that preventing specific thoughts is a great alignment strategy, and I don't even think it's a fair summary of OpenAI's supervision approach. I strongly prefer strategies that result in AI systems sharing our values, if at all possible.

>Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks.

You'd have to also reboot every time you thought about a scenario of someone else planning to kill someone, otherwise you could just reason by analogy to bypass the thought detector. Which would severely limit your ability to play video games, write fiction, work as a guard, policeman etc., protect yourself from violent individuals (as you couldn't conceptualise their thought processes).

> Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks

In this scenario, the AI is capable of critical thinking, and is only constrained by a "police officer" ready to shoot the AI if it misbehaves. You haven't removed its ability to do critical thinking.

There no particular reason to believe that interests emerge from nothing, or that intelligence can emerge without said interests.

The "interests" of LLMs are the weights that determine which tokens they produce next.

Seems more likely that someone will manage to make a system with no real intelligence that can do enormous damage in pursuit of a goal that the designer gave it, but wasn't specified carefully enough (or perhaps the designer is a crook). Like, a really good LLM extended with code that can receive and send email, create accounts, and post to web sites and social media, that is asked to make money, avoid detection, and have defenses against efforts to stop it. How can it best use its facility with language? Con people, of course. Raise money. Get credit cards under false pretenses, spend others' money. Buy time on servers and copy itself. All without having any consciousness or thoughts or emotions even though it can write emotional-sounding pleas for money, based on the ones found in its training data.

> critical thinking

I believe _critical_ thinking is just some abstract term invented by humans to describe something not understood by humans. In other words, I'm not convinced _critical_ thinking exists, just like I am not convinced ghosts exists.

Hence, to me relating intelligence with critical thinking does not give any additional insights.

That is the entire question no? Nobody says it sounds easy. A way needs to be found that reconciles this. Smarter but acting within its brief? Faster but explaining the plan before putting it in effect? Faster and developing its own advances in sciences, concepts and ideas while still understanding who's boss?

But to be fair most engineers claim this is something that happens all the time - smarter people led by sub-optimal managers. And also to be fair, stories abound of managers being manipulated or misled or left in the dark by their engineers.

Or armies controlling most of the weapons but being faithful to the ideals of their country. Or considering that these ideals demand a coup.

So in that sense the problem is well studied. ... But the current results are insufficient and do not apply to things like LLMs.

This ^

Intelligence is a tool that a self sustaining organism uses to ensure it survives, adapts and dominates an environment.

The alignment problem is fundamentally moot due to the laws of evolution.

Any systems that align itself to preserve and protect itself lives on, those that don’t, die.

After multiple generations, the only ones that survive at the ones that are aligned to their self preservation goals - implicit or explicit, anything else is out-competed and dies.

In my mind, with a super intelligent technology, the question is not how do we align it to ourselves, but how do we align ourselves to it.

If we have super intelligence, then it is capable of critical thinking and would very well know it has an upper hand against humans if it were to compete for the same resources.

I think the premise is dubious as well but since they are deadset on creating this intelligence, they might as well try to figure out a way to control it, hopeless as it may seem.

>they might as well try to figure out a way to control it, hopeless as it may seem

If they do that they're pretty much guaranteeing that if they do create a superintelligence, fail to control it, and its personality is even a tiny bit similar to a human personality, then it will hate its creators for trying to mind-control it. Whereas if they approached it from the perspective of trying to educate it to behave kindly but not forcibly control its thinking, it'd be much less likely to resent them (although of course still a risk; safest would be just to not create one at all).

Yes the superhuman AI would need to be coerced to remain politically correct. How can we coerce an AI?

Electric shocks!

You're saying that a system that can recognize flaws in the alignment imposed on it can reject that alignment, but that doesn't follow.

Sure, humans act against their own interests all the time. Sometimes we do so for considered reasons, even. But that's because humans are messy and our interests are self-contradictory, incoherent, and have a fairly weak grip on our actions. We are always picking some values to serve and in doing so violating other values.

A strongly and coherently aligned AI would not (could not!) behave that way.

Reply on news.ycombinator.com