Comment by kolektiv | Hacker News Reader

Comment by kolektiv | Hacker News Reader

Parent Full thread

I understand your point, I think - and certainly I don't want to go anywhere near name-calling or derision, that doesn't help anyone. But I am reminded of arguments I've had with creationists (I am not comparing you with them, but sometimes the general tone of the debate). It seems like one side is making an extraordinary claim, and then demanding the other side rebut it, and that's not something that seems reasonable to me.

The thing about nuclear weapons is that the theoretical science was clear before the testing - building and testing them was proof by demonstration, but many people agreed with the theory well before that. How they would be used was certainly debated, but there was a clear and well-explained proposal for every step of their creation, which could be tested and falsified if needed. I don't think that's the case here - there seems to be more of a claim for a general acceleration with an inevitable endpoint, and that claim of inevitability feels very short on grounding.

I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.

>I don't want to go anywhere near name-calling or derision, that doesn't help anyone.

Characterizing your opponent's argument as an appeal to "underpants gnomes" struck me as derisive, if you don't mind my saying.

If space ships operated by an alien civilization appeared in orbit above Earth, wouldn't that constitute a potent danger? I'd say it certainly would because the aliens might be (and probably were if they could travel here) better at science and technology than we are. The AI labs by the own admission are trying to create an alien intelligence as good at science and technology as possible. Yes, they're probably at least a decade or two away from "succeeding" at creating one better at science and technology than well-funded teams of humans are, but the AI labs might surprise us (and surprise themselves) by "succeeding" much sooner : everyone including I'm sure the researchers at OpenAI were surprised when GPT-4 was able to score in the 90th percentile on the bar exam.

Because even the researchers creating these frontier models don't understand them well enough to say for sure that the next model they spend hundreds of millions of dollars of GPU time training will exceed human ability in something dangerous like inventing new military technologies, the time to stop creating new large frontier models is now.

GPT-4 has a lot of knowledge of the world, but it is much less capable than a human is (and more to the point, than a capable human organization like the FBI or Microsoft is) at devising plans able to withstand determined human opposition. One of the things I'm worried about is new models that are much better at such a planning task _and_ have lots of knowledge about things like physics and human nature. One reason to worry that such a new model is not far off is that AlphaZero is better than humans at creating plans that can withstand determined human opposition (but of course AlphaZero has no knowledge of and in fact no way of obtaining knowledge of any part of reality beyond a Go board).

Companies declare that they are trying to build better AI, and the ultimately purpose is AGI. Both the definition of AGI given by the company and by AI aliment/safety researchers is similar. AI safety people trust it is dangerous.

let me go continue using a nuclear bomb as a metaphor. if we don't know whether building nuclear bomb is possible, but some companies declare they have existing progress on creating this new bomb...

the danger of nuclear bomb is obvious, because it is designed as a bomb. companies are trying to build the AGI which is similar to the dangerous AGI in AI safety researchers' prediction. the dangers are obvious, too.

They declare that - but I could also declare I'm trying to build a nuclear bomb (n.b. I'm not). Whether people are likely to try and stop me, or try and apply some legal non-proliferation framework, is partly influenced by whether they believe what I'm claiming is realistic (it's not - I have a workshop, but no fissile material).

Nobody gets too worried about me doing something which would be awful, but the general consensus is that I won't achieve. Until a company gives some credible evidence they're close to AGI... (And companies have millions/billions of reasons to claim they are when they're not, so scepticism is warranted).

All good points. Now playing viled advocate: building a nuclear bomb in my basement was very difficult, I admit. But since I already have my spywares installed everywhere, the moment a dude come with an AGI, it will directly be shared to all my follow hackers through bittorent, eDonkey, Hyphanet, Gnunet, Kad and Tor, just to name a few.

It is surely better to have regulation now than scramble to catch up if ago is possible.

What about nation states? Do you really think the US military will avoid working towards AGI if they think it would give them a tactical advantage? Or the CCP? Or North Korea? Personally, if AGI gets developed, I’d rather the first iteration be in the hands of someone who doesn’t also have access to nuclear weapons.

Why wouldn't the government just intasntly take it? Hell, why wouldn't the corporations just sell it to the highest bidder?

They may well. But if we ban research altogether, the only ones with access will be governments. At least with an open system there would be some competition.

So do we just pray AGI is impossible?

Companies are trying and spend many money to make it possible in this moment.

I think it's reasonable to at least act as if they really are trying to do what they say they're trying to do.

Like with the nuclear bomb situation - I think it would be reasonable for someone to try to stop you from building a nuclear bomb, or check if you really were, even if they had no idea how you could have gotten the materials. Because it would be really bad if you did. I think people would be worried about you trying to do something awful even if there was low confidence it was possible. They wouldn't be as worried as they would be if they thought you were more capable, but still worried.

So I guess that comes down to whether you think companies saying they want to make AGI are more like a toddler, a teenager, you, a ballistics engineer who owns a uranium mine, Lockheed-Martin, or a nation-state trying to make a nuclear bomb. My understanding is that people who are concerned about AI x-risk are largely in the teenager to Lockheed-Martin range (e.g., small eventual risk to large imminent risk), while I assume you think it's more in the toddler range (no risk at all for a long time)

> Companies declare that they are trying to build better AI, and the ultimately purpose is AGI.

They do declare it, but nobody has even come up with a plausible path from where we are today, to anything like AGI.

At this point they might as well declare that they're trying to build time machines.

yes, but why we even give them chances? when they know their ultimately purpose is dangerous?

> With AI, there still seems to be a lot of hand-waving between where we are now and "AGI".

> I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.

If I understand you correctly, then (1) you doubt that AGI systems are possible and (2) even if they are possible, you believe that humans are still very far away from developing one.

The following is an argument for the possibility of AGI systems.

  Premise 1: Human brains are generally intelligent.
  Premise 2: If humans brains are generally intelligent, then software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
  Conclusion: Software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.

(fyi I believe there is an ~82% chance humans will develop an AGI within the next 30 years.)

Also (3) that AGI in practice will necessarily pose any danger to humans is doubtful. After all Earth has billions of human level intelligence and nearly all of them are useless and if they are even mildly dangerous it's rather due to their numbers and disgusting biology than intelligence.

For info: I don't believe (1), I do believe (2) although not that strongly - it's more likely to be a leap than a gradient, I suspect - I simply don't see anything right now that convinces me it's just over the next hill.

Your conclusion... maybe, yes - I don't think we're anywhere near a simulation approach with sufficient fidelity however. Also 82% is very specific!

> For info: I don't believe (1), I do believe (2) although not that strongly

Thanks for clarifying. Do you believe there is a better than 20% chance that humans will develop AGI in the next 30 years?

> I simply don't see anything right now that convinces me it's just over the next hill.

These are the reasons that I believe we are close to developing an AGI system.

  (1) Many smart people are working on capabilities.
  (2) Many investment dollars will flow into AI development in the near future.
  (3) Many impressive AI systems have recently been developed: Meta's CICERO, OpenAI's GPT4, DeepMind's AlphaGo.
  (4) Hardware will continue to improve.
  (5) LLM performance significantly improved as data volume and training time increased.
  (6) Humans have built other complex artefacts without good theories of the artefact, including: operating systems, airplanes, beer.

These achievements are impressive, but I'd rather not overhype it.

* GPT-4 still hallucinates as hell, can't do math, fails as basic logic, can't handle really big contexts, hard to update, easy to jailbreak etc.

* AlphaGo was defeated by a Go amateur with a help of another AI.

* AlphaStar basically failed to achieve real goals, was trivial to cheese even after defeating high-ranked players sometimes.

All these problems are architectural, you can't just throw more money and GPUs at it like GPT-2-3-4.

It's hard to predict at this point. We may get to AGI anywhere from 5 years to 100 years.

I don't think these reasons are very persuasive, as everything but 5 has been true at different times in the past. Obviously it's much more people, more dollars, and more impressive systems (but slower hardware progress), but I hope you see what I'm getting at.

And of course there's differences in what someone considers to be soon. Many AI x-risk believers think there's a ~50% chance of AGI before 2031 (https://www.metaculus.com/questions/5121/date-of-artificial-...) (I've heard this prediction site's userbase tends towards futurists/techno-optimists/AI x-riskers). I would consider that soon, I wouldn't consider 2054 soon.

Reply on news.ycombinator.com