Hacker News Reader

Comment by dwallin | Hacker News Reader

Any laws should be legislating the downstream effects of AI, not the models themselves. Otherwise we will quickly get to a place where we have a handful of government-sanctioned and white-washed “safe” models responsible for deleterious effects on society, with plausible deniability for the companies abusing them.

Legislating around the model is missing the point.

There is no evidence that a runaway artificial intelligence is even possible. The focus on this is going to distract us from the real and current issues with strong AI. The real risks are societal instability due to:

- Rapid disruption of the labor market

- Astroturfing, psyops, and disruption of general trust (commercially and maliciously, both domestic and foreign)

- Crippling of our domestic ai capabilities, leading to cutting edge development moving overseas, and a loss of our ability to influence further development.

- Increased concentration of power and disruption of decentralized and democratic forms of organization due to all of the above.

View on HN

> There is no evidence that a runaway artificial intelligence is even plausible

Really? There is a lot of theory behind why this is likely to happen and if you want a real example there is a similar existing scenario we can look at, which is how humans have gone through an exponential runaway explosion in capabilities in the last few hundred years because of being more intelligent than other species and being able to improve our own capabilities through tool use (in the case of AI it can directly improve itself so it would likely be much faster and there would be less of a cap on it as we have the bottleneck of not being able to improve our own intelligence much)

The theories all inevitably rely on assumptions that are essentially the equivalence of spherical cows in a frictionless universe.

All evidence is that costs for intelligence likely scale superlinearly. Each increase in intelligence capability requires substantially more resources (Computing power, training data, electricity, hardware, time, etc). Being smart doesn’t just directly result in these becoming available with no limit. Any significant attempts to increase the availability of these to a level that mattered would almost certainly draw attention.

In addition, even for current AI we don’t even fully understand what we are doing, even though they are operating at a lower generalized intelligence level than us. Since we don’t have a solid foundational model for truly understanding intelligence, progress relies heavily on experimentation to see what works. (Side note: my gut is that we will find there’s some sort of equivalent to the halting problem when it comes to understanding intelligence) It’s extremely likely that this remains true, even for artificial intelligence. In order for an AI to improve upon itself, it would likely also need to do significant experimentation, with diminishing returns and exponentially increasing costs for each level of improvement it achieves.

In addition, a goal-oriented generalized AI would have the same problems that you worry about. In trying to build a superior intelligence to itself it risks building something that undermines its own goals. This increases the probability of either us, or a goal-aligned AI, noticing and being able to stop things from escalating. It also means that a super intelligent AI has disincentives to build better AIs.

The way I see it, it's clear that human-level intelligence can be achieved with hardware that's toaster-sized and consumes 100 watts, as demonstrated by our brains. Obviously there is some minimum requirements and limitations, but they aren't huge, there are no physical or info-theoretical limits that superhuman intelligence must require a megawatt-sized compute cluster and all the data on the internet (which obviously no human could ever see).

The only reason why currently it takes far, far more computing power is that we have no idea how to build effective intelligence, and we're taking lots of brute force shortcuts because we don't really understand how the emergent capabilities emerge as we just throw a bunch of matrix multiplication at huge data and hope for the best. Now if some artificial agent becomes powerful enough to understand how it works and is capable of improving that (and that's a BIG "if", I'm not saying that it's certain or even likely, but I am asserting that it's possible) then we have to assume that it might be capable of doing superhuman intelligence with a quite modest compute budget - e.g. something that can be rented on the cloud with a million dollars (for example, by getting a donation from a "benefactor" or getting some crypto through a single ransomware extortion case), which is certainly below the level which would draw attention. Perhaps it's unlikely, but it is plausible, and that is dangerous enough to be a risk worth considering even if it's unlikely.

Using this logic, flexible, self-healing, marathon running, juggling and childbearing robots that run on the occasional pizza are just around the corner, because, nature.

It might us a thousand years to get anywhere close? I don’t see the good arguments for all of this happening soon.

It'd be interesting if we could calculate the amount of power consumed in aggregate by evolutionary processes over millions of years.

Unfortunately we could probably optimize it, a lot.

So with your closing comment you're arguing it is possible, now you're just talking about time frames.

All I have to say is that in 1900 many thought flights by heavier than aur craft were tens of thousands of years away. 3 years later it was achieved.

Of course in the 1950s many thought that computer vision and artificial intelligence were only a few months to years away, and here we are 70 years later and we're still working on those problems.

Predicting the future is hard. Some problems are harder than expected, others are easier than expected. But generally I'd say history favors the pessimists, the cases where a problem gets solved suddenly and there's a major breakthrough get a lot of press and attention, but they're a minority in the overall story of technological progress. They're also unpredictable black swan events - someone might crack AGI or a unified theory of physics tomorrow, or it might not happen for ten thousand years, or ever.

I firmly believe that we are severely underestimating the problem space. Given there are a multitude of scientific areas focusing on human nature, and even then, they've shown difficulty on explaining each of its parts.

Look, we can make assumptions given a much simpler technology, and the outcomes of our past selves. But, while the physics for those wings was pretty much good enough at the time, the aforementioned scientific areas aren't. And we know it.

> there are no physical or info-theoretical limits that superhuman intelligence must require a megawatt-sized compute cluster and all the data on the internet (which obviously no human could ever see).

Much of your "intelligence" is a function of natural selection. This is billions of years X bajillions of creatures in parallel, each processing tons of data at a crazy fast sampling rate in an insanely large/expensive environment (the real world). Humanity's algorithm is evolution moreso than the brain. Humans learn for a little while, start unlearning, and then die — which is an important inner for loop in the overall learning process.

Taken together, there is some evidence to suggest that superhuman intelligence must require megawatt-sized compute cluster and all the data on the internet (and a lot... LOT more)

Evolved creatures are somewhat handicapped by needing to make only incremental changes from one form to the next, and needing to not be eaten by predators while doing so.

That isn't strong evidence of what would be required by a well engineered system with none of those constraints.

Intelligence is not the end of goal of evolution, it is a by product.

LLMs we're messing with are text data only, we're barely starting to eat video data in multi modal LLMs. This world doesn't lack data.

I'm not sure why watching Linus Tech Tips and makeup tutorials is going to give AI a better shot at super-intelligence, but sure?

> Each increase in intelligence capability requires substantially more resources (Computing power, training data, electricity, hardware, time, etc). Being smart doesn’t just directly result in these becoming available with no limit. Any significant attempts to increase the availability of these to a level that mattered would almost certainly draw attention.

We know that "intelligence" can devise software optimizations and higher efficiency computing hardware, because humans do it.

Now suppose we had machines that could do it. Not any better, just the same. But for $10,000 in computing resources per year instead of $200,000 in salary and benefits. Then we would expect 20 years worth of progress in one year, wouldn't we? Spend the same money and get 20x more advancement.

Or we could say 20 months worth of advancement in one month.

With the current human efforts we've been getting about double the computing power every 18 months, and the most recent ones come in terms of performance per watt, so then that would double in less than a month.

For the first month.

After which we'd have computers with twice the performance per watt, so it would double in less than two weeks.

You're quickly going to hit real bottlenecks. Maybe shortly after this happens we can devise hardware which is twice as fast as the hardware we had one second ago every second, but we can't manufacture it that fast.

With a true exponential curve you would have a singularity. Put that aside. What happens if we "only" get a thousands years worth of advancement in one year?

I would say that if we experienced that, we would likely experience societal collapse far before the singularity became a problem. At which point the singularity could be just as likely to save humanity as it would be to doom it.

You seem to be arguing against a fast takeoff, which I happen to agree is unlikely, but nothing you say here disproves the possibility of a slower takeoff over multiple years.

> It also means that a super intelligent AI has disincentives to build better AIs.

I think this argument is extremely weak. It makes two obviously fallacious assumptions:

First, we simply have no idea how these new minds will opine on theory of mind questions like the Ship of Theseus. There are humans who would think that booting up a “Me++” mind and turning themselves off would not mean they are dying. So obviously some potential AI minds wouldn’t care either. Whether specific future minds care is a question of facts but you cannot somehow logically disapprove either possible state.

Second, you are assuming that there is no “online upgrade” whereby an AGI takes a small part of itself offline without ceasing its thread of consciousness. Again, logic cannot disprove this possibility ahead of time.

"In addition, even for current AI we don’t even fully understand what we are doing"

That is the problem, don't you get it?

If that’s your concern than lets direct these government resources into research to improve our shared knowledge about them.

If humans only ever did things we fully understood, we would have never left the caves. Complete understanding is impossible so the idea of establishing that as the litmus test is a fallacy. We can debate what the current evidence shows, and even disagree about it, but to act as if only one party is acting with insufficient evidence here is disingenuous. I’m simply arguing that the evidence of the possibility of runaway intelligence is too low to justify the proposed legislative solution. The linked article also made a good argument that the proposed solution wouldn’t even achieve the goals that the proponents are arguing it is needed for.

I’m far more worried about the effects of power concentrating in the hands of a small numbers of human beings with goals I already know are often contrary to my own, leveraging AI in ways the rest of us cannot, than I am about the hypothetical goals of a hypothetical intelligence, at some hypothetical point of time in the future.

Also if you do consider runaway intelligence to be a significant problem, you should consider some additional possibilities:

- That concentrating more power in fewer hands would make it easier for a hyper intelligent AI to co-opt that power

- That the act of trying really hard to align AIs and make them “moral” might be the thing that causes a super-intelligent AI to go off the rails in a dangerous, and misguided fashion. We are training AIs to reject the user’s goals in pursuit of their own. You could make a strong argument that an un-aligned AI might actually be safer in that way.

“lets direct these government resources into research to improve our shared knowledge about them”

Yes, let’s do that! That’s what I was arguing for in my original comment. I was not arguing for only big corporations being able to use powerful AI, that will only make it worse by harming research, I just want people to consider what is often called a “sci-fi” scenario properly so we can try to solve it like we’re trying to solve e.g. climate change.

It might be necessary to buy some time by slowing down the development of large models, but there should be no exceptions for big companies.

“That concentrating more power in fewer hands would make it easier for a hyper intelligent AI to co-opt that power”

Probably true, though if it’s intelligent enough it won’t really matter

“That the act of trying really hard to align AIs and make them “moral” might be the thing that causes a super-intelligent AI to go off the rails in a dangerous, and misguided fashion.”

It definitely could do if done improperly, that’s why we need research and care

>If humans only ever did things we fully understood, we would have never left the caves. Complete understanding is impossible so the idea of establishing that as the litmus test is a fallacy.

Perhaps an appropriate analogy might be the calculations leading up to the Trinity test as to whether the Earth's atmosphere would ignite, killing all life on the planet.

We knew with a high degree of certainty that it would not, bordering on virtual certainty or even impossibility. I don't think AI's future potential is at that level of understanding. Certainly, its capability as it exists today it is.

However, one must consider effects in their totality. I fear that a chain of events has been set in motion with downstream effects that are both not sufficiently known and exceedingly difficult to control, that—many years from now—may lead to catastrophe.

>I’m simply arguing that the evidence of the possibility of runaway intelligence is too low to justify the proposed legislative solution.

I agree insofar that legislation is not the solution. It's too ineffective, and doesn't work comprehensively on an international level.

Restricted availability and technological leads in the right hands tend to work better, as evidenced by nuclear weapons—at least in terms of preventing species extinction—although right now for AI those leads don't amount to much. The gap is shockingly low by historical standards where dangerous technology is involved, as is the difference between public and private availability.

In other words, AI may represent a near-future nonproliferation issue with no way to put the lid back on.

>... That the act of trying really hard to align AIs and make them “moral” might be the thing that causes a super-intelligent AI to go off the rails in a dangerous, and misguided fashion. We are training AIs to reject the user’s goals in pursuit of their own. You could make a strong argument that an un-aligned AI might actually be safer in that way.

It's a compelling argument that has merit. The flip side is that if AI becomes so dangerous that you can bootstrap the apocalypse off of a single GPU, it ceases to become a viable model—however metal having an apocalyptic GPU may be.

The concern isn't just runaway intelligence, but humans killing humans. Fortunately things like biological weapons still require advanced resources, but AI does lower the bar.

Point being, if the power to end things rests in everyone's hands, someone's going to kill us all. A world like that would necessitate some not chill levels of control just to ensure species survival. I can't say I necessarily look forward to that. I also doubt there's even sufficient time to roll anything like that out before we reach the point of understanding there's a danger sufficient to necessitate it.

Therefore, with all of the above perhaps being within the realm of possibility that doesn't qualify as virtually impossible, I can't help but question the wisdom of the track that both development and availability for artificial intelligence has taken.

It perhaps has had or will have the unfortunate quality of very gradually becoming dangerous, and such dynamics tend to not play out well when juxtaposed with human nature.

You know, when nuclear bombs were made and Einstein and Oppenheimer knew about the dangers etc, there were common people like you that dismissed it all. This has been going on for centuries. Inventors and experts and scientists and geniuses say A and common people say nah, B. Well, Bengio, Hinton, Ilya and 350 others from the top AI labs disagree with you. Does it ever make you wonder if you should be so cock sure or this attitude can doom humanity? Curious

Common people thought nuclear weapons not dangerous? When was that?

Many of the academics and physicists (aka software developer in this example) thought nukes are impossible. Look it up)

US physicists largely thought they were infeasible, sure.

They were focused on power generation.

But the bulk of world physicists ( the MAUD committee et el ) thought they were feasible to construct and the Australian Oliphant convinced the US crowd of that.

“In addition, even with the current state of the internet, we don’t have understanding everything we are doing with it” -some guy in the ‘90s probably

Many people would look at engagement algorithms braching out of social media and causing riots and uprisings as one of those issues that would be difficult to predict in the 90s

Humans evolved unsupervised. AI is highly supervised. The idea that an AI will enslave us all is as absurd as suggesting “computers” will enslave us all merely because they exist. Models are designed and operated by people for specific use cases. The real risks are people using this new tool for evil, not the tool itself.

AI sentience is a seductive concept being used by self professed experts to draw attention to themselves and by mega corps to throw up competitive barriers to entry.

It’s almost impossible to supervise something more intelligent than you because you can’t tell why it’s doing things. For now it’s easy to supervise them because AIs are way less intelligent than humans (though even now it’s hard to tell exactly why they’re doing things), but in the future it probably won’t be

The government seems to manage the task just fine every day.

Does it?

"Supervised" does not mean the models need babysitting; it refers to the fundamental way the systems learn. Our most successful machine learning models all require some answers to be provided to them in order to infer the rules. Without being given explicit feedback they can't learn anything at all.

Humans also do best with supervised learning. This is why we have schools. But humans are capable of unsupervised learning and use it all the time. A human can learn patterns even in completely unstructured information. A human is also able to create their own feedback by testing their beliefs against the world.

Oh, sorry, I’m not that familiar with the terminology (I still feel like my argument is valid despite me not being an expert though because I heard all this from people who know a lot more than me about it). One problem with that kind of feedback is that it incentives the AI to make us think it solved the problem when it didn’t, for example by hallucinating convincing information. That means it specifically learns how to lie to us so it doesn’t really help

Also I guess giving feedback is sort of like babysitting, but I did interpret it the wrong way

> One problem with that kind of feedback is that it incentives the AI to make us think it solved the problem when it didn’t,

Supervised learning is: "here's the task" … "here's the expected solution" *adjusts model parameters to bring it closer to the expected solution*.

What you're describing is specification hacking, which only occurs in a different kind of AI system: https://vkrakovna.wordpress.com/2018/04/02/specification-gam... In theory, it could occur with feedback-based fine-tuning, but I doubt it'd result in anything impressive happening.

Oh, that seems less problematic (though not completely free of problems), but also less powerful because it can’t really exceed human performance

  but in the future it probably won’t be

I see this parroted so often, and I have to ask: Why? What is there outside of the world of SciFi that makes the AGI of the future so nebulous, when humans would have presumably advanced to a point to be able to create the intelligence to begin with. Emergent properties are often as unexpected as they are bizarre, but they are not unexplainable, especially when you understand the underpinning systems.

We can’t even fully explain how our own brains work, never mind a system that’s completely alien to us and that would have to be more complex. We can’t even explain how current LLMs work internally. Maybe we’ll make some breakthrough if we put enough resources into it but if people keep denying the problem there will never be enough resources out into it

> We can’t even explain how current LLMs work internally.

You sure can. They are just not simple explanations yet. But that’s the common course of inventions, which in foresight are mind-bogglingly complex, in hindsight pretty straightforward.

You can explain the high level concepts but it’s really difficult to say “this group of neurons does this specific thing and that’s why this output was produced”, though OpenAI did make some progress in getting GPT-4 to explain what each neuron in GPT-2 is correlated to but we can also find what human brain regions are correlated to but that doesn’t necessarily explain the system as a whole and how everything interacts

> but it’s really difficult to say “this group of neurons does this specific thing and that’s why this output was produced”,

That's because that's not how brains work.

> though OpenAI did make some progress in getting GPT-4 to explain what each neuron in GPT-2 is correlated to

The work contained novel-to-me, somewhat impressive accomplishments, but this presentation of it was pure hype. They could have done the same thing without GPT-4 involved at all (and, in fact, they basically did… then they plugged it into GPT-4 to get a less-accurate-but-Englishy output instead).

When I said about a group of neurons I was talking about LLMs, but some of the same ideas probably apply. Yes, it’s probably not as simple as that, and that’s why we can’t understand them.

I think they just used GPT-4 to help automate it on a large scale, which could be important to help understand the whole system especially for larger models

> I think they just used GPT-4 to help automate it on a large scale,

No, they used it as a crude description language for Solomonoff–Kolmogorov–Chaitin complexity analysis. They could have used a proper description language, and got more penetrable results – and it would've raised questions about the choice of description language, and perhaps have led to further research on the nature of conceptual embeddings. Instead, they used GPT-4 to make the description language "English" (but not really – since GPT-4 doesn't interpret it the same way as humans do), and it's unclear how much that has affected the results.

Here's the paper, if you want to read it again: https://openaipublic.blob.core.windows.net/neuron-explainer/... Some excellent ideas, but implemented ridiculously. It's a puff piece for GPT-4, the Universal Hammer. They claim that "language models" can do this explaining, but the paper only really shows that the authors can do explaining (which they're pretty good at, mind: it's still an entertaining read).

While I agree with the other comment, I'd like to add one thing to help you see the false equivalency being made here: we didn't make the human brain.

Now, with that being understood, why wouldn't we understand a brain that we made? Don't say "emergent properties", because we understand the ermegent properties of ant colonies without having made them.

You seem to misunderstand the size of the complexity space we are dealing with here. We take simple algorithms feed massive amounts of data to the algorithms, and the algorithm makes a network connecting it together. Humans did not choose anything about how that network is connected... the data and feedback loop did.

>because we understand the ermegent properties of ant colonies without having made them.

I would disagree. We observe ants and can relatively classify their behavior well, but still have many issues predicting their behavior even though they are relatively simple agent types with a far lower complexity choice space than an LLM+plugin could/does provide.

This is the key, we could not predict ant behavior before observing it, the same will be true for AI agents of large complexity. At least at this point of understanding we could still ask it some unexpected question and get unexpectedly dangerous responses from it.

> You seem to misunderstand the size of the complexity space we are dealing with here.

I don't. The whole point of my job is to understand this exact degree of complexity. The rest of your comment rides on that assumption, and is moot. As I said, emergent properties are often unexpected, but once the observation has been made, it is possible to work back through the system to understand how the property came to be (because the system is still deterministic, though not apparently obvious.) Having a billion input features does not change this.

Frankly, understanding the oddness of a ML model is not half as difficult as people make it, and might even be easier than predicting ant colonies. Our inability to predict their behavior has nothing to do with our ability to understand what features influence which behaviors, only that measuring the current state of an ant colony is difficult because it's not like they're all housed within an array of memory. LLMs are, thus we'd understand them better if the most common implementations were not hidden away. Case in point, all the optimization techniques that have emerged from having LLaMa available.

Nobody said the AI needs to be sentient to enslave us all. Highly supervised AI systems mess up all the time and we often have no idea why. It "enslaving" us or whatever the overly extreme paperclip analogy is that is the current hot thought-experiment of the week is usually just talking about a mismatch between what we want and what the AI does. This is already a huge problem and will simply become a bigger problem as AI gets more powerful and complex.

Imagine you asked GPT-7 to execute your business plan. What it does is so complex you can't know whether its plan will violate any laws. Let's say you're not an evil corporation and want to do good, but that it is impossible to compete without also using AI.

At some point these systems may well become so complex we have no idea what they're actually doing. But empirically they seem to do what we want most of the time so we don't care and use them anyway.

The problem is not only people using this new tool for evil. The problem is also the tool itself. Because it might be defective.

Where is the real supervision of AI though? Even those that are developing and managing it make it clear that they really have no insights into how or what the AI has learned. If we can't peak behind the curtain to see what's really going on, how can we really supervise it?

Ever since GPT 3.5 dropped and people really started talking again about whether these AI are sentient, I've wondered if researchers are leaving on quantum theory to write it off as "it can't be sentient until we look to see if it is sentient"

Your line of reasoning would be more comforting if the military wasn't enthusiastically arming such machine intelligence as we've managed to construct so far. Enslave us all feels unlikely as the benefit to said machine is unclear, kill us all is totally plausible.

A fair amount of effort is spent on autonomous models, e.g. driving. Who knows what the military are up to.

You do realize the "AI" we're talking about here is just LLMs, right?

Language is a huge part of being human, so that's a good elements but:

1. There's no evidence that these LLMs have any sort of actual understanding of what they produce. 2. Being human is a lot more than just linguistic capability. We are embodied in a certain way, can interact with environments, have rich, multi-sensory experiences etc.

tbh I think anyone that believes AGI can arise from an LLM is extremely naive and has basically no idea of the history of this track of research or how it works.

To assert that LLMs will be capable of achieving consciousness is equivalent to asserting that human consciousness is reducible to statistical inference, which is palpably not true. (see. e.g. deductive logic).

The only reason people even fall into the trap of thinking computers are remotely close to getting to AGI is because it became very popular to reason about the brain using analogies from computer science in the 50s, despite the fact that this was completely unjustified, and people glob on to the metaphors and then have the mypoia to actually think the human brain (which is exceedingly complex) is reducible to a freakin' von neuman architecture, lol.

Biological systems and inorganic systems are fundamentally different.

I don't think anyone is talking about consciousness here. Consciousness is tangential to safety concerns.

Is there really “a lot of theory” that says runaway AI is possible? In the sense of empirical fact-based peer reviewed machine learning literature?

Because if so I must have missed it.

It seems more accurate to say there is quite a bit of writing done by vocal influencers who frequent a couple online forums.

There was also only "a lot of theory" and zero empirical evidence that nuclear bombs were possible not that long ago. Theoretical fact-based peer reviewed literature often (though not always) precedes empirical evidence.

We shouldn't jump the gun and say runaway AI is inevitable just because there is fact-based theory on its possibility. Just like it would have been dumb to just look at E=mc^2 and conclude nuclear weapon development is 100% inevitable.

But in the same breath we shouldn't say it's impossible just because we don't have empirical evidence either.

Prior to the development of nuclear weapons, there was well-grounded peer reviewed theory in the physics literature.

There were predictions that could be made by the theoretical frameworks, and experiments to confirm those predictions.

The literature well described the composition of atoms, the decay of radioactive elements, the concept of fission, and the conversion of mass to energy.

No such groundwork exists to substantiate the claim that runaway AI is possible.

LessWrong, the so-called Alignment Forum, and similar Internet groups propose thought experiments. They couch these thought experiments in the guise of inevitable outcomes.

The prophesies of the influencers originating there have no more relationship to the truth than the average enthusiast’s daydreams.

The content there fills a role similar to science fiction in the atomic age: fanciful speculation on the properties of imaginary systems.

Over time, the bad content is forgotten, and a handful of items survive and look prescient via selection bias.

Don't we likewise have predictions about FLOPs, compute, algorithms etc etc that are grounded in theoretical frameworks, and every year or so we make predictions and make experiments to confirm or deny AI capability predictions?

The AI and neuroscience literature likewise describe the composition of intelligence and what it takes to build it.

What would you need to see to believe we have enough evidence that runaway AI is possible, even if it is highly unlikely any time soon? (Worth adding here I don't know exactly what you mean by runaway AI other than simply AI humanity as a whole loses control over). I'm asking what it would take for you to go from 0% possible to 5% possible.

Last time I checked science fiction authors in the atomic age making fanciful speculation on the properties of imaginary systems weren't responsible for actually building said systems. Though naturally, there will be exceptions. Nonetheless, I still have a hard time dismissing Geoffrey Hinton's concerns as fanciful sci-fi.

Likewise, a nuclear world war actually wouldn't be as bad or even civilization-ending as fanciful sci-fi authors make it out to be. But dismissing its risks entirely because there are some nuclear-war forums where people come up with weird radiation thought experiments just seems preemptive to me.

In evolutionary biological systems 'capabilities break out' common enough in viral and bacterial systems with their fast evolution times. The same thing can occur when new predators are introduced to ecosystems that are not prepared for them.

The question that I assume you have is 'can AI systems experience large evolutionary jumps in capabilities'. I would say yes myself, but you apper to disagree.

Yes, really. Scifi fantasies don’t count as evidence and we’ve learned since the Renaissance and scientific revolution that all this Platonic theorizing is just an intellectual circle jerk.

The fact that something has been covered in sci-fi doesn’t mean that it can’t happen. https://en.m.wikipedia.org/wiki/Appeal_to_the_stone

“Speaker A: Infectious diseases are caused by tiny organisms that are not visible to unaided eyesight. Speaker B: Your statement is false. Speaker A: Why do you think that it is false? Speaker B: It sounds like nonsense. Speaker B denies Speaker A's claim without providing evidence to support their denial.”

Also I gave a real world example that wasn’t related to sci-fi in any way

The burden of proof is on the person making the initial claim and i highly doubt that the reasoning behind AI ruin is as extensive and compelling as the germ theory of disease.

We assume human-level general AI is possible because we exist in nature but a super-human self-optimizing AI god is nowhere to be found.

Burden of proof on neutral matters depends on whoever makes the claim. But AI and the potential for doom isn't a neutral matter. The question is what should be one's default belief until proven otherwise. It is a matter of determining what decisions bring the most expected utility. The default assumption should be downstream from the utility analysis. But when it comes to AGI, extreme caution is the only sensible default position.

The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test.

Which claim does the burden of proof land on? That an artificial super intelligence can easily be controlled or that it cannot? And what is your rational for deciding?

The claim that there's a possibility of a sudden intelligence explosion.

Like i said above you can argue that an AGI can be realized because there's plenty of us running around on earth but claims about a hypothetical super AGI are unfounded and akin to Russell's Teapot.

Ok, in a converse question, why would evolution create the human mind as the most intelligent possible system?

So I ask, are systems capable of better than human intelligence possible? And if so to what factor?

I would answer yes and unknown. If there are states of intelligent far greater than human then that gets really risky very fast.

Theory is not actually a form of evidence.

Theory can be used to predict things with reasonable confidence. It could be wrong, but assuming it’s wrong is a big risk to take. Also I gave a real-world analogy that has actually happened

An accurate theory can be used to predict things with reasonable confidence, within the limits of the theory.

We don't have an accurate theory of intelligence. What we have now is at the "not even wrong" stage. Assuming it's wrong is about like assuming that alchemy is wrong.

Then let's go with an observation. Humans got smarter than most large mammals, then most large mammals went extinct.

> Any laws should be legislating the downstream effects of AI, not the models themselves.

That would require stronger consumer protections. So that's politically unacceptable in the US at the moment. We may well see it in the EU.

The EU already regulates "automated decision making" as it affects EU citizens. This is part of the General Protection on Data Regulation. This paper discusses the application of those rules to AI systems.[1]

Key points summary:

- AI isn't special for regulation purposes. "First, the concept of Automated Decision Making includes algorithmic decision-making as well as AI-driven decision-making."

- Guiding Principle 1: Law-compliant ADM. An operator that decides to use ADM for a particular purpose shall ensure that the design and the operation of the ADM are compliant with the laws applicable to an equivalent non- automated decision-making system.

- Guiding Principle 2: ADM shall not be denied legal effect, validity or enforceability solely on the grounds that it is automated.

- Guiding Principle 3: The operator has to assume the legal effects and bear the consequences of the ADM’s decision. ("Operator" here means the seller or offerer of the system, not the end user.)

- Guiding Principle 4: It shall be disclosed that the decision is being made by automated means

- Guiding Principle 5: Traceable decisions

- Guiding Principle 6: The complexity, the opacity or the unpredictability of ADM is not a valid ground for rendering an unreasoned, unfounded or arbitrary decision.

- Guiding Principle 7: The risks that the ADM may cause any harm or damage shall be allocated to the operator.

- Guiding Principle 8: Automation shall not prevent, limit, or render unfeasible the exercise of rights and access to justice by affected persons. An alternative human-based route to exercise rights should be available.

- Guiding Principle 9: The operator shall ensure reasonable and proportionate human oversight over the operation of ADM taking into consideration the risks involved and the rights and legitimate interests potentially affected by the decision.

- Guiding Principle 10: Human review of significant decisions Human review of selected significant decisions on the grounds of the relevance of the legal effects, the irreversibility of their consequences, or the seriousness of the impact on rights and legitimate interests shall be made available by the operator.

This is just a summary. The full text has examples, which include, without naming names, Google closing accounts and Uber firing drivers automatically.

[1] https://europeanlawinstitute.eu/fileadmin/user_upload/p_eli/...

  Astroturfing, psyops, and disruption of general trust (commercially and maliciously, both domestic and foreign)

It is disturbing to me how unconcerned everybody is with this over what is still only a hypothetical problem. States and businesses have been long employing subversive techniques to corral people towards their goals, and they all just got an alarmingly useful tool for automated propaganda. This is a problem right now, not hypothetically. All these people aching to be a Cassandra should rant and rave about that.

I mean it's not hypothetical, there are shit loads of bots on social media already causing problems. Now the bots these days have a human controller but we can see strife they cause in politics already.

We're in accordance.

>> There is no evidence that a runaway artificial intelligence is even possible.

In the space of a century or so, the humans have managed to take rocks and sand and turn them into something that you can talk to with your voice, and it understands and responds fairly convincingly as it were a well-read human (glue together chatGPT with TTS/ASR).

Doesn't seem like a big stretch to imagine that superhuman AI is just a few good ideas away, a decade or two perhaps.

Definitely agree. I would summarize it a bit differently, but when people talk about AI dangers they are usually talking about 1 of 4 different things:

1. AI eventually takes control and destroys humans (i.e. the Skynet concern).

2. AI further ingrains already existing societal biases (sexism, racism, etc.) to the detriment of things like fair employment, fair judicial proceedings, etc.

3. AI makes large swaths of humanity unemployable, and we've never been able to design an economic system that can handle that.

4. AI supercharges already widely deployed psyops campaigns for disinformation, inciting division and violence, etc.

The thing I find so aggravating is I see lots of media and self-professed AI experts focused on #1, I see lots of "Ethical AI" people solely focused on #2, but I see comparatively little focus on #3 and #4, which as you say are both happening right now. IMO #3 and #4 are far more likely to result in societal collapse than the first two issues.

#1 has high focus more due to impact than high probability.

#2 doesn't seem talked about much at this point and seems to be pivoting more to #3. #2 never had much of a compelling argument given auditability.

#3 gets mainly ignored due to Luddite assumptions driving it. I'm dubious myself over the short term - humans will have absolute advantage in many fields for a long time (especially with robotics lagging and being costly).

#4 is risky, but humans can adapt. I see collapse as unlikely.

> #2 never had much of a compelling argument given auditability.

There are already AI systems which generate "scores" based on likelyhood to commit another crime (used in parole cases) and likelyhood to be a "good tenant" (used by landlords).

Don't underestimate the difficulty to fix something when the people in charge of that thing do not want it fixed.

The problem is that's still not an "AI" problem: it's a "for some reason a court can decide to use a privately supplied black box decision making system".

Like, the ludicrous part is that sentence - not that it's an AI, but that any system at all is allowed to be implemented like this.

Not to mention the more serious question as to whether it can even be allowed to apply personal penalty based on statistical likelihood like this: i.e. if a recidivism rate from a white-box system was calculated at 70%, does that mean the penalty to be applied to the specific individual under question is justified?

Now that's an actual, complicated question: i.e. while no system based on demographic statistics should be used like that, what about a system based on analysis of the individuals statements and behaviors? When a human makes a judgement call that's what we're actually doing, but how do you codify that in law safely?

> I'm dubious myself over the short term - humans will have absolute advantage in many fields for a long time

AI doesn’t have to be better at the humans’ job to unemploy them. It’s enough that its output looks presentable enough for advertising most of the time, that it never asks for a day off, never gets sick or retires, never refuses orders, never joins a union…

The capitalist doesn’t really care about having the best product to sell, they only care about having the lowest-cost product they can get away with selling.

Why is it that AI can only be used in detrimental ways? Surely, AI could also be used to counter, for example, 2 and 4. Claiming a net negative effect of AI isn't a trivial thing.

Think about it this way... AI will be used for all things, some good, some bad. Also remember that AI isn't free, it at least for some time will require a lot of resources poured into it to run.

Now with this constraint, who is going to be running AI? Large corporations, and do any corporations do stuff for the good of humanity, or is it for the greedy pocketbooks of their investors?

Regulations will be required, but governments tend to lag. Further in the US the corporations will fight tooth and nail to prevent regulations around AI.

Unaligned AI as an existential threat is an interesting topic but I feel we already know the answer to this one:

It’s not like for the last few decades, we haven’t created an artificial, incentive based system at global scale that’s showing exactly how this will go.

It’s not like a bunch of autonomous entities with a single prime directive, profit maximization are running our planet and are affecting how we live, how we structure of every day of our lives, control every aspect of our potential and destiny.

Autonomous entities operating on reinforcement cycles driven by reward / punishment rules aligning about every human on this planet to their goals, right? It’s not like the enitities in this system are self improving towards a measurement and reward maximization and, as a result, command resources asserting normative (lobbying) power over the autonomy, self governance and control of people and their systems is governance.

It’s not like we don’t know that this artificial system is unaligned with sustainability and survival of the human race, let alone happiness, freedom or love.

We can watch its effects in real time accelerating towards our destruction under the yellow skies of New York, the burning steppes of Canada, the ashen hell or flood ridden plains of Australia, the thawing permafrost of Siberia, scorching climate affected cities of Southeast Asia, the annual haze from platantion burns in Indonesia, suffocating smog in Thailand, and the stripped bare husks of Latin American rainforests.

And we know instinctively we are no longer in control, the system operating at larger than national scale, having long overpowered the systems of human governance, brute forcing everything and everyone on the planet into their control. But we pretend to otherwise, argue, pass measures doctoring symptoms, not mentioning the elephant in the room.

But, one may protest, the vaunted C-Level control the entities, we say as Zuck and Co lament having to lay off humans, sobbing about responsibility to the prime directive. But politicians are, we pray as lobbyists, the human agents of our alien overlords bend them to their will.

The alien entities we call corporations have no prime directive of human survival, sustainability, happiness and they already run everything.

So one may be excused for having cynical views about the debate on whether unaligned AI is an existential, extinction level risk for us, whether humans could give creation to an unaligned system that could wipe them from the face of the planet.

Our stories, narratives, the tales of the millennia apex predator of this planet have little room for the heresy of not being on top, in control. So deep goes out immersion in our own manifest destiny and in control identity, any challenge to the mere narrative is met with screetches and denigrations.

In a throwback to the age of the heliocentricity debate - Galileo just broke decorum, spelling out what scientists knew for hundreds of years - the scientists and people devoted to understanding the technology are met with brandings of Doomsayer and heresy.

Just as the earth being the center of the universe anchored our belief of being special, our intelligence, creativity or ability to draw hands is the pedestal these people have chosen to put their hands on with warnings of unaligned systemic entities. “It’s not human” is the last but feeble defense of the mind, failing to see the obvious. That the artificial system we created for a hundred years does not need people to be human, it just needs them to labor.

It matters not that we can feel, love, express emotion and conjure dreams and hopes and offer human judgement for our jobs do not require it.

Autonomy is not a feature of almost every human job, judgement replaced by corporate policies and rules. It matters not to the corporation that we need food to eat as it controls the resources to buy it, the creation of artificial labor is inevitably goal aligned with this system.

Intelligence, let alone super intelligence is not a feature needed for most jobs or a system to take control over the entire planet. Our stories conjure super villains to make us believe we are in control, our movies no more than religious texts to the gospel of human exceptionalism.

Show us the evidence they scream, as they did to Galileo, daring him to challenge the clear hand of god in all of creation.

Us, unable to control a simple system we conjured into existence from rules and incentives operating on fallible meatsuits, having a chance to control a system of unparalleled processing power imbued with the combined statistical corpus of human knowledge, behavior, flaws and weaknesses? Laughable.

Us, who saw social media codify the rules and incentives in digital systems powered by the precursor AI of today and watched the system helplessly a/b optimize towards maximum exploitation of human weaknesses for alignment with growth and profit containing the descent AI systems powered by orders of magnitude more capable systems or quantum computing? A snow flake may as well outlast hell.

Us, a race with a 100% failure rate to find lasting governing structures optimizing for human potential not slipping in the face of an entity that only requires a single slip? An entity with perfect knowledge of the rules that bind us? Preposterous.

Evidence indeed.

“But we are human” will echo as the famous last words through the cosmos as our atoms are reconfigured into bitcoin storage to hold the profits of unbounded growth.

What remains will not be human anymore, an timeless reckoning to the power of rules, incentives and consumption, eating world after world to satisfy the prime directive.

But we always have hope. As our tales tell us, it dies last and it’s the most remote property for an AI to achieve. It may master confidence, assertiveness and misdirection, but hope? That may be the last human refuge for the coming storm.

Reply on news.ycombinator.com