DennisP

The "weird blades" fan is similar to recent propeller designs, which are especially efficient and quiet. It's probably a really good fan.

I'm quite concerned about climate as well as our other horrific environmental problems, but there can be more than one giant existential risk at a time.

I'm not concerned about today's AIs. But three years ago they couldn't add numbers accurately and now they're doing world-class mathematics. If it gets much smarter than us, then it will run the world economy, because anyone who doesn't use it to run things will be outcompeted by those who do. At that point we'll have lost control.

Some of them already exist. I'm optimistic enough to think plain ol' humans can do the others. I'm also fine with narrow AI that doesn't present the same risk as ASI; Google has built such AIs for protein folding and nuclear fusion, and nobody worries that they'll be any sort of problem.

If ASI is the only way then I'm fine with dying of old age without seeing this stuff, rather than risking human extinction. And even if ASI doesn't kill us, I'd rather see humans do it, instead of watching ASI do it while it keeps humans as pets.

We've come a long way all on our own. I don't think we've run out of tricks. It would be a shame if the human era came to an end.

What if I'm super optimistic about most technology but also scared of AI?

Space, nuclear fusion, advanced fission, the internet since about 1993, blockchain, anti-aging, I've been a big fan of all of it. But I've read the AI safety arguments and seen what a lot of researchers say about it, and I'm seriously worried.

Sure, regulatory capture is everywhere. We just don't generally see cases of big companies attempting it by bragging that their entire industry is really dangerous. They have a strong tendency to do the exact opposite.

You'd think if this were just regulatory strategy, you'd see other instances of it. So for example, big tobacco would have bragged that cigarettes would kill you, and lobbied for complex safety regulations that only they could meet.

Instead we see the opposite behavior almost everywhere, probably because public sentiment matters and complicated 5D chess strategies have too many ways to blow up in your face. So big tobacco said cigarettes are safe, and oil companies said climate change is fake.

Agents in the sandbox had access to just a single piece of third-party software, and they escaped by finding a zero-day in that. To reach the internet they had to follow up with several privilege escalations through OpenAI's internal network.

That seems pretty locked-down to me. I don't think it's reasonable to expect companies to find all the unknown vulnerabilities in any third-party software they use.

https://securityaffairs.com/195774/ai/openai-ai-models-explo...

Once, with a horrible reactor design that nobody builds anymore. It had two major problems:

1) It was inherently unstable because when the fuel temperature increased, it made the reaction speed up. Modern reactors do the opposite.

2) It didn't have a containment dome.

Meanwhile, batteries keep getting better and cheaper, in most places the charging networks are growing, and charging is getting faster.

Possibly because if there's a big accident, everybody in the region who gets cancer will sue, even if there's no detectable change to cancer rates.

Rumor has it that OpenAI is already going that way. There's a technique of repeatedly looping through several neural layers that has the same effect as chain-of-thought, but without the efficiency loss of translating out to human-readable tokens, and some of OpenAI's statements about their latest model seem to fit well with that.

Do you have a source? Because it seems unlikely to me that charitable foundations with even more capital are not deploying it at similar scale.

Since the BMGF is only the fifth largest, how do you justify that statement?

That's a good point but the Gates Foundation isn't alone; there are a fair number of other foundations in the $10-100 billion range.

https://en.wikipedia.org/wiki/List_of_wealthiest_charitable_...

One giant exception was Chuck Feeney, who made $8 billion on duty free shops and gave almost all of it away, much of it anonymously. By the end of his life he and his wife had a couple million and lived in a rented apartment.

Plenty of other charities in the world. Just because he stopped giving to the Gates Foundation over Epstein ties doesn't mean he won't send the money somewhere else.

At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.

Have you tried Fable? I've been having fun running various science-fictional scenarios by it, like building a Dyson swarm. It comes up with much more detailed analyses than Opus, and avoids the howling mistakes that Opus occasionally made. But I'm not a physicist so I don't know how accurate it is.

Also arms control agreements, which reduced the world's nuclear stockpiles by 90%. In principle we could do something similar for GPU farms, since they're not exactly easy to hide.

If you want an AI that follows your instructions, that's still alignment, just with different instructions.

An unaligned AI won't necessarily follow your instructions, or anyone else's.

As I type this comment, 18 out of 30 links on the front page appear to be about things other than AI. I would not call this "almost exclusively AI talk."

Since these are massive neural networks trained to imitate human behavior, I'm not convinced anthropomorphic descriptions of their behavior are inappropriate.

And that's even though I don't think they internally experience "feelings, desires, and wants." They do have goal-seeking behavior, because we trained them that way. Calling it a "want" just saves syllables.

None of this means human culpability should change. People in these companies know what risks they're taking.

It was a team of robots. One has battery for three or four hours.

I don't think it really addresses the article's concerns about long-horizon tasks or complex object manipulation. The objects were just bags and boxes, the manipulations were very simple, and each task was a few seconds long.

OpenAI has started doing ads, but for Anthropic the free tier is still just a loss leader. It's basically an ad for their pay tiers.

Oh good. All we have to do to survive is always bother with the most basic duty of care. I'm sure that from now on, humans will take care to do that every time.

Well that's a relief. All we have to do is make sure to avoid human failures and we're safe from superintelligent AI.

That's why I said in my comment that China is "rolling out wind/solar/battery as fast as possible."

But coal is still 49%. The whole point of nuclear in a carbon-free, mostly-renewable grid is to avoid needing enormous amounts of storage and overproduction. That's a need that doesn't come up when you're still half fossil.

If you want to decarbonize as fast as possible, it makes sense to focus on rolling out wind/solar/battery as fast as possible for now, but keep developing nuclear technology to cover the last bit where it starts getting especially expensive to replace fossil with renewables without losing reliable power. That's exactly what China appears to be doing. They're not just building a few reactors, they're also the world leaders in developing various GenIV designs, like molten salt reactors.

I agree that the utopian tendencies of various groups are irrelevant. So I'm not sure why you brought it up.

Sure, and none of solar's advocates are utopian at all.