Hacker News Reader

When someone says "we can regulate x, but we must act quickly," it's important to remember that the "we" they are referring to are often so poorly informed on the subject that the average house cat could outperform them if tested on the material and they are spoon-fed talking points by corporate lobbyists who pay their campaign budgets.

Harari is a scare monger making bank off promoting the pop-culture "Terminator is bad" caricature of AI.

I was fairly upset with CBS' "60 Minutes" as I received a urgent message from one of my parents about the dangers of AI. It had such gems as:

Google CEO: "It's just predicting the next word." News Story Narrator: "But it doesn't feel like that--"

Yes, because how it "feels" is certainly objective material for a news story. And my favorite:

CBS News Correspondent: "You don't fully understand how it works. And yet, you've turned it loose on society?"

I actually dug into Giphy for a Frankenstein GIF for that one ...

sigh

Really getting sick of Fear-of-the-Other ...

Yup. Feelings don't care about your facts.

Still, I don't think that having a regulation that AI must reveal themselves when asked would be overstepping.

It would draw a red line between aligned and unaligned AI. It may not even be enforceable, but it would be a revealing characteristic that gives useful information, even when it lies. Especially when it lies.

> > We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai.

> I mean, maybe. Sure, LLMs can currently code shockingly well, and LLMs are just programs, so an LLM writing an LLM is def much more likely than a nuke building a nuke. But we don't _know_ if there are limits to what AIs can do. Or, to pick on the word "exponential," we don't know if there are limits to growth rates of AIs improving AIs!

It seems to me that the current approach to AIs can improve along only two axes: better training data, or better algorithms. I don't think this kind of AI can generate better training data for the next generation of itself, just by the nature of how it operates. It (currently) doesn't know truth from falsehood; how is it going to know what training data is better or worse than other data? And I don't think it can improve its algorithm either. It may be able to implement an algorithm that someone else specifies, but I'm not sure that it can write a working algorithm, let alone an improved working algorithm.

So I don't think that self-improving AI is currently on the table.

> ai can make exponentially more powerful ai

Is this true to any extent right now, or just blatantly incorrect assumptions on which to base clickbait and FUD?

The issue is whenever it becomes true it is likely way too late to regulate it.

For example let's imagine a hypothetical world that didn't have guns yet be we think they were possible. Well if you're waiting till one group made guns first you're going to be at their mercy going after their gun armed soldiers with swords.

Right now? Not true to any extent, no.

But it's not exactly "blatantly incorrect assumptions". It's more wishful thinking. (The "self-improving, exponentially more powerful" AI is currently the only hope for an AGI - except that there isn't currently any realistic hope of a self-improving, exponentially more powerful AI.)

Wishful? More like nightmare thinking, hence the desire for some people wanting to ensure that if it can happen, that it doesn't.

I think it's an assumption based on the fact that we don't seem to be anywhere near the maximum amount of computing capacity that we could be using to train AIs. It might be just throwing more compute at the problem, or it might be that we are nowhere close to the ideal algorithms / model architectures, and AI will aid the process of getting there.

To turn it around a bit, what is your null hypothesis? If it's that AI will not be able to make more powerful AI, why do you believe that?

Given that there are two cases, and smart people disagree about which one we're headed towards, and one could have very dire consequences while the other is no big deal, I think it's reasonable not to assume that the latter is true without exercising a lot of due diligence.

"We can apply more compute power" is something very different from "exponentially self improving". Assuming that "exponentially improving" will show up if given enough compute power is very much not a null hypothesis.

Ok, so there's self improving, where an AI tries to make itself smarter, and then there's just exponential improvement, where an AI does one or more of the following, with or without a human in the loop:

* acquires money through legitimate business and buys more computing power to train a smarter AI.

* acquires money through illegal means and does the same.

* designs a more efficient model architecture or training process that results in a smarter AI.

I don't know how far away from this we are, it's very hard to predict. I do predict that at first we'll see mostly things where a human is directing the AI, but there's selective pressure toward not having a human involved, because someone who just sets up an AI and lets it loose is going to have a thing that reacts faster and can work 24/7. Irresponsible businesses that let their AIs run rampant are going to make more money, and be able to afford better AIs. It's the same story that has played out again and again over history, Moloch (1) pushing us to do the selfish but harmful thing by rewarding those that do it.

Remember that exponential curves seem pretty slow at first, and it can be hard to tell when you are standing on one. It might be that we're still years away from exponential self-improvement, but I think it's worth taking seriously as a possibility. Even with just humans designing AIs, the amount of effort required has become exponentially less as we got more compute, designed better libraries and tools, people published research and copied it and built on it. We can't predict exactly what or when the next step will be, but I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic.

[1] <https://slatestarcodex.com/2014/07/30/meditations-on-moloch/>

> I don't know how far away from this we are, it's very hard to predict.

On this point, I am in total agreement with you.

One way to tell might be, what happens if we let an AI try to improve itself? Does it get better, or worse? My money currently is on "worse". Well, how much worse? How far away are we from it being able to improve itself? I think we are an enormous distance away, almost an immeasurable distance, but I could be wrong.

> I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic.

It's always logistic in the end. (But if we're on the first part of the curve, where it's exponential, that "in the end" bit doesn't matter.)

I believe AI could make exponentially better AI but do we have any evidence that this is likely, or could happen soon? Sounding alarm bells with no understanding of how likely it is seems irresponsible.

>> We can still regulate the new ai tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, ai can make exponentially more powerful ai.

I dropped in to comment on that bit too. You've described it well already, but I'd still like to add:

.. hypothetically, in the future. To be fair, in principle then it seems perfectly feasible that a sufficiently much more advanced nuke might invent an even more advanced nuke too, hypothetically, in the future.

Not really to detract from the rough direction of the argument in the post though, because despite the hype, imprecision and questionably declared speculation of the sort you pointed out, I think he's basically right to be concerned about the hazards of throwing even current and imminent "AI" into Humanity's communications and cultural evolution.

The only thing that can help is open-sourcing this technology; It is too dangerous in the hands of a monopoly or even a small oligopoly.