Comment by gonehome | Hacker News Reader

Comment by gonehome | Hacker News Reader

Parent Full thread

The recent paper about using gpt-4 to give more insight into its actual internals was interesting, but yeah the risks seem really high at the moment that we'd accidentally develop unaligned AGI before figuring out alignment.

Out of the options to reduce that risk I think it would really take something like this, which also seems extremely unlikely to actually happen given the coordination problem: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

You talk about aligned agents - but there aren't any today and we don't know how to make them. It wouldn't be aligned agents vs. unaligned, it's only unaligned.

I don't think spreading out the tech reduces the risk. Spreading out nuclear weapons doesn't reduce the risk (and with nukes at least it's a lot easier to control the fissionable materials). Even with nukes you can still create them and decide not to use them, not so true with superintelligent AGI.

If anyone could have made nukes from their computer humanity may not have made it.

I'm glad OpenAI understands the severity of the problem though and is at least trying to solve it in time.

What is the "recent paper about using gpt-4 to give more insight into its actual internals?"

https://openai.com/research/language-models-can-explain-neur...

Unaligned doesn't really seem like it should be a threat. If it's unaligned it can't work toward any goal. The danger is that it aligns with some anti-goal. If you've got a bunch of agents all working unaligned, they will work at cross-purposes and won't be able to out-think us.

This is a misunderstanding of what AI alignment problems are all about.

Alignment != capability

Think a paperclip maximizing robot that in its process of creating paperclips kills everyone on earth to turn them into paperclips.

Corporations like Saudi Aramco are already doing that. You don't need a superintelligent AI, corporations that maximize profit are already sufficient as misaligned superhuman agents.

You can't maximize profit without customers, they must be aligned with someone.

They're aligned with the military-industrial complex. The US military is one of the biggest consumers of fossil fuels[1] and it's the same with other nations and their energy use. So profitable is not the same as aligned with human values.

1: https://en.m.wikipedia.org/wiki/Energy_usage_of_the_United_S...

> The US military is one of the biggest consumers of fossil fuels

I guess this phrasing is up for debate, but according to the source linked "the DoD would rank 58th in the world" in fossil fuels.

Is that a huge amount of fossil fuel use? Absolutely. But one of the biggest?

> According to the 2005 CIA World Factbook, if it were a country, the DoD would rank 34th in the world in average daily oil use, coming in just behind Iraq and just ahead of Sweden.

Sure, the phrasing could be debated but the fact that it even ranks close to actual nation states is already problematic. The US military is basically an entire nation state of its own. This is nothing new if you're old enough to have observed the kind of damage it has done but it demonstrates my point about profit and alignment. Profits are very often misaligned with human values because war is extremely profitable.

US DOD fuel use being the level of Sweden doesn’t seem problematic to my envelope-math; it seems to reflect the size of the entities involved.

Iraq is a now broken third word country/economy in recovery so not a great comparable to US. Sweden is small but a good comparable culturally/development-wise. US is 331 million people. It spends 3% of GDP on military. 3% of 331m is 10 million. Sweden is 10 million people. U.S. military fuel use is in line with Sweden’s.

I could be off here (DOD!=US military?), corrections welcome, but I wouldn’t even be shocked if a military entity uses 3-10x more fuel than a civilian average and above math puts us surprisingly close to 1x.

Math seems correct but US military also includes conglomerates and companies like Palantir and Anduril (main reason it is described as an industrial complex is because there is no clear distinction between corporations and how their activities are tied up with military spending and energy use).

Oh there's no denying the US military has ballooned to the size of a small to medium-sized country. That alone is a huge issue for me personally - I do agree with our country having any form of standing military but that precedent was abandoned 80 years ago.

I'm not sure how to properly compare the military of one country with the entirety of a country ~1/30th the size. On the surface it doesn't seem crazy for those to have similar budgets or resource use.

The comparison is in terms of energy use since at the end of the day that is the fundamental currency of all techno-industrial activity. The point is that the global machinery that is currently guiding civilizational progress is fundamentally anti-life. It constantly grows and subsumes whatever energy resources are accessible without any regard for negative externalities like pollution and environmental degradation. This is why I don't take AI alarmism seriously because the problem is not the AI, the problem is the organization of techno-industrial civilization and its focus on exponential growth.

It's only going to keep getting worse and the AI alarmism is not doing anything to address the actual root causes of the crisis. If anything, AI development might actually make things more sustainable by better allocating and managing natural resources so retarding AI progress is actually making things worse in the long run.

I think those really are separate concerns that should both be given more attention.

There's a strong correlation between GDP growth and oil use, that's a huge problem and one that likely can't be solved without fundamentally revisiting modern economic models.

AI poses it's own concerns though, everything from the alignment problem to the challenge of even having to define what consciousness even is. AI development won't inherently make allocating natural resources easier - with the wrong incentive model and lack of safety rails AI could find its own solution to preserving natural resources that may not work out so well for us humans.

The current model is already destructive and most of the market is managed by artificial agents. Schwab will give you a roboadvisor to manage your retirement account so AI is already managing large chunks of the financial markets. Letting AI manage not just the financial aspects but things like farmland is an obvious extension of the same principle and since AIs can notice more patterns it's going to become basically a necessity because global warming is going to make large parts of existing farmlands unmanageable. Floods and droughts are becoming more common and humans are very bad at figuring out the weather so there will be an AI agent monitoring weather patterns and allocating seeds to various plots of land to maximize yields.

Bill Gates has bought up a bunch of farmland and I am certain he will use AI to manage them because manual allocation will be too inefficient[1].

1: https://www.popularmechanics.com/science/environment/a425435...

Bit of an interesting thought experiment there, could a corporation maximize profit without customers? I wonder if we can find any examples of this type of behavior...

Yes, but a profit maximizer doesn’t need to eliminate all humans to become a big problem.

In fairness, corporations can still be fraudulent.

No, I understand what you're saying, I just think you're wrong. To be a little clearer: you're assuming a single near-omnipotent agent randomly selects an anti-goal and is capable of achieving it. If we instead create 100 near-omnipotent agents odds are that the majority will be smart enough to recognize that they have to cooperate to achieve any goals at all. Even if the majority have selected anti-goals, it's likely that the majority of the anti-goals will be at cross-purposes. You'll also have a paperclip minimizer, for example. Now, the minimizers are a little scary but these are thought experiments and the goals will not be so simple (nor do I think it would be obvious to anyone including the AIs which ones have selected which goals.) The AIs will have to be liars if they select anti-goals, and they will have to not only lie to us but lie to each other, which makes coordination very hard bordering on impossible.

In some ways this is a lot like Bitcoin, in that people think that with enough math and science expertise you can just reason your way out of social problems. And you can, to an extent, but not if you're fighting an organized social adversary that is collectively smarter than you. 7 billion humans is a superintelligence and it's a high bar to be smarter than that.

It’s worth reading about the orthogonality thesis and the underlying arguments about it.

It’s not an anti-goal that’s intentionally set, it’s that complex goal setting is hard and you may end up with something dumb that maximizes the reward unintentionally.

The issue is all of the AGIs will be unaligned in different ways because we don’t know how to align any of them. Also, the first to be able to improve itself in pursuit of its goal could take off at some threshold and then the others would not be relevant.

There’s a lot of thoughtful writing that exists on this topic and it’s really worth digging into the state of the art about it, your replies are thoughtful so it sounds like something you’d think about. I did the same thing a few years ago (around 2015) and found the arguments persuasive.

This is a decent overview: https://www.samharris.org/podcasts/making-sense-episodes/116...

> the first to be able to improve itself in pursuit of its goal could take off at some threshold and then the others would not be relevant.

Thanks for reminding me that I need to properly write up why I don't think self-improvement is a huge issue.

(My thought won't fit into a comment, and I'll want to link to it later).

Alignment is about agreement with human preferences and desires, not internal consistency. e.g. An AI that wanted to exterminate humanity could work towards that goal, but it would be unaligned (unaligned with humanity). Alignment is basically making sure humanity is fine with what the AI does.

Humanity has more than one alignment...

Ya, why do you think there are alarm bells sounding off everywhere right now…

The capabilities are coming fast. There is no alignment.

The most likely alignment we will get is the alignment m of money to power.

Yes, that's part of the reason why alignment is such a huge problem.

You can imagine an AI that answers questions and helps you get things within reason that doesn't hurt anyone else plus corrections for whatever problems you imagine with this. That's roughly an aligned AI. It will help you build a bomb as a fun experiment, but would stop you from hurting someone with the bomb.

Apart from some obvious cases that everyone agrees with, alignment is not a big problem it is an incoherent one. It can’t be “solved” any more than the problem of what the best ice cream flavor is can be solved.

Humanity doesn’t have unified interests or shared values on many things. We have different cultural memories and different boundaries. What to some is an expression of a fundamental right is an affront.

At the limit sure there’s variance, but our shared selected history has a lot in common, something a non-human intelligence would not get for free: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...

I’m also not a moral relativist, I don’t think all values are equivalent, but you don’t even need to go there - before that point a lot of what humans want is not controversial and the “obvious” cases are not so obvious or easy to classify.

And I'm less concerned about emergent alignment with an anti-goal (paperclip optimization) than I am with a scenario like ransomware designed by malicious humans using a super AI aligned with an anti-goal.

Reply on news.ycombinator.com