Hacker News Reader

Comment by arugulum | Hacker News Reader

Allow me to present Altman's wager:

- If OpenAI does not achieve AGI, and you invested in it, you lose some finite money (or not, depending on the value of their other R&D)

- If OpenAI does not achieve AGI, and you did not invest in it, you saved some finite money, which you could invest elsewhere for finite returns

- If OpenAI achieves AGI and you invested in it, you get infinite returns, because AGI will capture all economic value

- If OpenAI achieves AGI and you did not invest in it, you get negative infinite returns, because all other economic value is obliterated by AGI

Therefore, one must invest.

View on HN

Has the same issue as Pascal's. Competing AGI projects (gods) exist and their believers might be the ones reaping the infinite rewards, not to mention the distinct possibility the AGI (god) doesn't actually see rewarding believers as its highest priority, and might choose to share its infinite rewards with people who aren't part of the inner circle, or even punish people who joined inner circles with the express intent of elevating themselves above ordinary people :)

Actually makes a bit more sense from the traditional view that AGI projects might not reach their goal but the well run ones are likely to have very commercially valuable byproducts anyway. If we were getting Star Trek economics out of it, who'd be interested in entirely obsolete concepts like "economic value" and "100x returns" anyway?

Agreed. It seems extremely naive that they believe they can control the power of AGI, plus it is borderline ridiculous that their goal for AGI is to capture economic value.

I'm skeptical of OpenAI, but to be charitable I don't think economic value is the primary goal for their AGI. Their position seems to be that it needs to be one of several necessary goals in order to repay investors for helping them scale up to achieve it in the first place.

It also has an issue not present in Pascal's, which is you are multiplying by infinity in a context where it doesn't actually make sense.

The payoff from AGI may be incalculable, but it isn't infinite both in itself and in Altman's ability to enjoy the rewards it promises. Once the value becomes finite, a whole heap of risk-reward logic kicks in that the Wager wants to sweep under the rug.

As a concrete example, following this Altman's wager would result in Altman giving all his wealth to the first beggar on the street who mumbles that he might be able to run an AGI project - the possibility that the beggar can achieve that is, technically speaking, nonzero. Multiply that by infinity and you have a great expected return (infinite, in fact). However, practically speaking, the risk will overwhelm the large-but-finite payoff.

Infinity is bigger than people think :P

Two other possibilities that Altman's wager as presented does not take into account:

1. The government could change the rules (i.e. redistribute wealth) in response to the new economic reality brought about by AGI

2. The AGI may decide to retain the economic value it captures for itself

According to the true believers, the payoff for AGI is infinite because the superintelligence will be capable of literally anything (or at least simulating it well enough that it doesn't matter). To them, it is Pascal's wager.

Religious beliefs can be weird.

>Has the same issue as Pascal's.

I get the feeling that some people (maybe not this commenter) are missing the point I'm making.

Yes, Pascal's wager is flawed - which is why I'm modeling the Altman's wager after it.

I thought I had made the wording and respective payoff statements sufficiently tongue-in-cheek, but I guess this is Poe's law biting back.

tbf I wasn't 100% sure whether you were tongue-in-cheek or had just spent way too much time taking variants of this argument seriously on LessWrong...

(also, as much as I'm also tongue in cheek when talking about AGIs punishing [infinite numbers of simulations of] Sam Altman for backing the wrong AGI project, I'm dead serious about them potentially being able to turn a decent profit selling advanced data processing to Silicon Valley startups. And let's face it, Sam's a much better CEO to be able to help them achieve that than AI Rapture anyway)

> If OpenAI achieves AGI and you invested in it, you get infinite returns

The article states that this is explicitly false. By design, OpenAI's investor returns are capped at 100x.

I find it odd that 3 other people have also replied to your comment and I'm the first one to mention that OpenAI has explicitly capped investor returns to prevent an explosion in inequality. I wonder how many people read the full article before commenting.

I wonder if that was poor wording from the reporter. They write “capped” but then describe a liquidation preference.

The cap is one of the reasons this wager might actually be great marketing.

- General hubris (ambition?) weeds out skeptics (curmudgeons?) In the audience right away. Not the intended buyer!

- Every minute a potential investor spends contemplating the impact of a 100x return cap is a moment they are framing on that outcome.

- OpenAI already proved that they can advance the cutting edge on specific AI tasks. In that context, AGI is a good smokescreen (or the kind of North star that you navigate toward without reaching). It's attractive to idealistic investors and new hires, but also justifies pragmatic research.

I don't put weight on the AGI goal, but I fully believe that Sam can help find business models in the research projects that can be executed with a critical mass of talent and capital.

Once the superintelligence achieves godhood, ye will remember yr true faithful.

You left off the assumptions that AGI will be a superintelligence and one capable of capturing all value. There's no evidence to believe either one. Instead, it will probably be pretty dumb like humans are without the decades of supervised training we get and our interactions with the world. Further, the second a smart one starts getting devastating results, there will be political push-back to do something about it. People might cast votes to make rules about them that limit their power or rate of development.

Which gets us back to the real risk that people like Altman are too detached from reality to get: the masses being damaged by laws and companies controlled by tiny few, aka a plutonomy. That's where we're at. That's what's causing most problems people face. For example, CEO's trying to get bonuses do layoffs while folks like Altman wonder how AI might hurt jobs. If they're really worried, the smart folks need to pool all their resources together to combat the ability of special interests to bribe politicians and get away with it. Then, incentive structures that do less damage to employees and consumers as the companies grow. That be a start on addressing real problems vs those they're making up.

I am very enthusiastic about AGI but I think it's strange how most other enthusiasts seem to assume that AGI automatically becomes (extremely) super-intelligent. I can't rule out the possibility of that being engineered deliberately over a period of months or years, assuming it's possible, but it seems like it is in no way automatic.

At the very least the system would need extensive training. No reason to believe that the initial versions will have some super-human superspeed self-training ability to absorb a lifetime of information in a very short period.

Also OpenAI seems to have their main strategy for ensuring it's safe as just being the first group to progress towards it and then witholding their research except for select "safe" partners. This seems like it can only make the deployment less democratic rather than necessarily safer.

As far as made up versus real problems, my guess is for someone like Altman who is benefiting so much from the system, it is hard for his worldview to really acknowledge extreme flaws such as fundamental corruption.

I think the only certainty here is the need to spend a lot of money on compute - whether from third party cloud services or from fans manufacturing chips you designed.

The description of some company or system capturing all future value sounds more like a singleton than general intelligence. Instead of calling themselves OpenAI maybe they could change their name to something that reflects that direction.

> Further, the second a smart one starts getting devastating results, there will be political push-back to do something about it. People might cast votes to make rules about them that limit their power or rate of development.

My fear of AGI is that it will not be able to be stopped once deployed, as you're saying. It's irreversible. It will know that we want to shut it down, and so it will be able to copy itself onto other devices (think about the hacking capabilities of AGI for a moment), or any other method of survival.

The current approach is to regulate it after it is invented. While this worked for cars and planes and many other inventions, AGI is different for the reason above.

In fact, with that in mind, a scary thought is that any group of researchers who are cognizant of that would hide their creation of AGI if successful, assuming they were motivated by profit. Thus it would remain woefully unregulated.

> think about the hacking capabilities of AGI for a moment

We're discussing the hypothetical skills and capabilities of a thing which is fundamentally science fiction. The rules are treated as arbitrary.

I don't see a priori why an AGI would be intrinsically good at hacking, or even why it would be capable of exponentially improving itself.

This is the problem I see with any discussion of AGI. The game's rules don't matter, so we can define whichever properties we'd like for the sake of argument. There's no skin in the game to counteract that, because we have no conception of what AGI will actually be like - nor if it's even possible.

As it stands, in your comment and the rest of this thread I see a variety of leaps and jumps to scenarios which seem completely undefended.

I try to stay grounded by comparing them to both prior AI's, regular brains, savant intelligence, and combinations of them playing to each's strengths. When I do this, I realize that we already have systems in place preventing, reducing, and containing damage from all kinds of smart humans. We react to their schemes in areas like finance in cat and mouse game. The AGI's would probably be no worse than dealing with them so long as they're designed to be hard or impossible to copy themselves.

Let me try - briefly - to explain why that is nonsense.

If AGI ever comes to fruit it will cause such a huge disruption that anything that came before it will not be useful in trying to predict aspects of the world after it. For all we know the people that backed it will be hunted down like rats for extermination.

What if AGI really leads to Skynet and the downfall of humanity? Then you’ve realized infinite negative returns in a success condition.

Not that I’m saying it will, but Pascal’s wager is known to be a huge fallacy.

It depends how much you worry about society going south because of climate breakdown (and how much you expect AGI to help or worsen the situation).

My current view is that the current political hot potato status of AGI means it won't be developed anytime before the climate crisis really kicks in. At this point fiat money becomes worthless due to emergency measures. Your best bet now to help your future self is to spend to prepare civilisation/your country/your neighbourhood/yourself for the ordeal.

Also due to the current political hot potato nature of AGI I would expect it to be seized/controlled by governments if anyone gets anywhere near close to developing it.

So I see little upside for investing and likely having a reduced capacity for adaptation if you do invest. You may have different ideas about the speed of development or badness of the climate breakdown though.

I say we still have to try. Mankind won't get another shot at it.

Trying for agi could be part of a reasoned response to climate breakdown.

But I am not going count on getting a reasoned response any time soon.

The reasoned response to climate breakdown is to start taking it seriously at a global level.

AGI could help with that by culling humans, or at least seriously constraining human behavior. We're perfectly capable of doing that ourselves, although such behavior tends not to be popular.

Realistic scenarios in which AGI solves climate change for us, in time, seem highly unlikely.

> The reasoned response to climate breakdown is to start taking it seriously at a global level.

Agreed! That should be the number one priority.

I also agree that it is unlikely that AGI will fix climate change for us in time.

What it might help with is adaptation. If all our attempts in trying to control the giant intricate set of climatic and ecological feedback loops we are disturbing don't work, then it might help us with designing appropriate technologies to cope with what the world has become.

>> If OpenAI achieves AGI and you invested in it, you get infinite returns, because AGI will capture all economic value

AGI would render ALL humans redundant... Investors too.

"Evolution is cleverer than you are" - Orgel's rule.

Pascal's wager is actually logically flawed.

If there are two separate AGI invented, and you could only have chosen to invest in one, then where's the infinite capture of value?

See this video on a reasonable argument about this: https://youtu.be/JRuNA2eK7w0

That's why they're using Lisp instead of Pascal! ;)

AGI may not capture any economic value, it may capture value in the same sense that when I go to work I capture value that my daughter does not. Yet my daughters play is neither impeded or accentuated by my work - she does her own thing regardless while I mess round with software.

This assumes that bourgeois nostrums such as property rights still exist in a post-agi world.

The whole point of the singularity is that it changes _everything_ and we have no idea what's on the other side of it.

Why would the New Mind give a shit about petty human notions like investment?

Would a $0.01 investment be good enough then to partake in infinite returns?

That's basically the same wager as Roko's Basilisk, or any religion. But, the same problem as with religion, is, which God/AI company to believe/invest in...

Almost, if not for this:

> OpenAI has become a “capped profit” company, with the promise of giving investors up to 100 times their return before giving away excess profit to the rest of the world.

Here's a better philosophical wager:

- If AGI is possible within our lifetimes, we will at the time it happens all pretty much live in a post-scarcity economy and will all share the rewards.

- If AGI turns out not to be possible within our lifetimes, you'll have wanted to invest that money in a way that benefits you.

If either of the latter two of these outcomes came to pass then the question of investing would be moot as money would become pointless (as there is no further value to exchange).

He's capped return at 100x, though.

Reply on news.ycombinator.com