reducesuffering

> extremely intelligent people make confident predictions about AI capabilities and progress, then be so completely wrong.

The ability for the human mind to rationalize conclusions to maintain denial in the face of a very scary future is immense. Genuinely grappling with the implication of where we're headed is usually very crushing. It's not easy to engage with the possibility, and very intelligent people will use those smarts to feel safe.

Yes a couple of elite mathematicians working on the problem for a year, which AGI solved in a fraction of the time. What about everyone else 100IQ? What about as the models are even better 1 year from now, 2 years? The trajectory hasn't abated.

Practical?! The goalposts will keep moving until morale improves (narrator: it doesn't)

> The bubble is here

Zitron implied 2 years ago that OpenAI would collapse by now. How's that bubble popping going? All NVDA+memory co+frontier lab numbers are accelerating

You've been here awhile, this forum is default incredibly skeptical and critical, especially when it comes to AI. It has been to their predictive detriment for many years now. AI bubble conviction is over 2 years now, $300b NVDA profits since. AI tool adoption has completely encompassed any tech savvy office and isn't going anywhere. Broken records.

Their profits are incredible too, they have 62% net margin! Last quarter $60b profit on $96b revenue. Since the "AI bubble going to pop" terrible takes 2 years ago, NVDA has made $300b in profits.

Yes, exactly why you should not put belief in someone who has proven themselves to make poor predictions. He's even further removed from the zeitgeist 9 years later.

Daniel Kokotajlo had the closest prediction for our current world, 5 years ago: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...

You should adjust that these are the kind of the people with the most prescient accuracy for where we are going again. Their world model is more accurate.

Kokotajlo authored: https://ai-2040.com/

Once a model like Fable/Mythos is trained, that was used for this project, the inference costs are profitable. CapEx vs. OpEx. Every new Zig->Rust port of this caliber using Fable/Mythos costs the standard token price, it's not subsidized. The model already exists for the original CapEx, you can do 1,000 ports of this for standard token pricing.

Lobste.rs is now flagging Bun 1.4 post as spam and comments are dismissing it as nothing, even as that blog post got much more discussion and piling on about the rewrite.

Internet echo chambers are quite a problem. AI success and its immense implications are wreaking havoc on the ability of even intellectuals in HN and lobsters to face reality in front of them, even as their predictions become falsified again and again.

~3 months start to finish and the price of 1 engineer for a year, to rewrite a critical infra codebase that tens of millions of user software relies on. Truly the goalposts keep moving. This kind of project would've have been 10x the effort and price a few years ago.

Absolute crickets from the "there's no way LLM's can rewrite Bun to Rust this quickly, it's going to be buggy, it's marketing" crowd

Comments here 3 months ago are quite something: https://news.ycombinator.com/item?id=48132488

This is still the case. The frontier lab leadership has admitted their mechanistic interpretability is practically nil and is an active area of research but they have made little progress. They don't understand how they work, they just grow and unleash them.

These things are Gain of Function research for digital viruses

Why should we believe someone who can't do basic due diligence that Hinton did indeed receive a Turing award? https://amturing.acm.org/byyear.cfm

It is extraordinarily emotionally hard for someone to stare down the terrible implications of what is unfolding. All manner of rationalization and cope will be applied to come up with excuses; motivated reasoning.

The CIA director will call AGI capabilities "digital nuclear weapons" and Geoffrey Hinton will estimate a 50% probability AGI ends humanity, and half of HN will call every new evidence of disaster a marketing stunt.

no one dead because we're willing to use our means to enforce a restriction on a scenario that is likely on our trajectory to kill everyone. We also use serious threats to prevent nuclear proliferation that endangers everyone

The motivated reasoning of people who have dismissed AGI capabilities and risks year after year are endless, even after they have been proven wrong at every step of the way.

Every time doomers have been vindicated in their accurate predictions, denialists will come up with the closest thing they can grab to rationalizing their pre-existing belief. "LLM's aren't that useful, It's all marketing, capabilities are slowing down, they want to regulatory capture, etc."

Trolley problem. What's worse than all of humanity dead from ASI?

Sam Altman: "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."

Their plan, I shit you not... Is literally to develop the intelligence capabilities and ask the more powerful models how to do deal with things.

> They failed to anticipate the competitive pressures of capitalism...

No, LessWrong types have been discussing this for over a decade now.

Meditations on Moloch (2014) is also an HN favorite...

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

I mean one look at both commentary and it's clear HN has devolved considerably. Just look at the atrocious difference in intelligent reasoning in this thread's comments compared to a LessWrong post right now:

https://www.lesswrong.com/posts/9RL9MuGZjzm4q3gKG/what-just-...

As much as I liked the writing, this part stood out to me too. This woman was the best candidate and was passed over for the opportunity because the author felt attracted to her. At least he acknowledges he was 25 and dumb. But as much as pretty privilege does give people an edge in some places, this was important to remind me in what ways a woman is passed over that probably wouldn't happen to a man. Hire the most qualified candidate period.

Because the other side is the general informed consensus for which multiple sources are readily available for me to find. I have seen nothing to indicate this side, hence I'm asking for any evidence, because I don't see it. Either could potentially be correct, but I don't think they're equally probable.

Holy telephone game... the Kissinger quote you are referring to was about if they let the South Vietnamese leader fall, then the world would take away the message that we wouldn't support our allies.

Instead we get this butchered caricature all over the internet

Or maybe, just maybe, there's another reason that CIA Director John Ratcliffe describes AGI as "akin to digital nuclear weapons." Unless of course you think the CIA is a branch of OpenAI's marketing department

citation needed

In 2015, about a year before ever founding OpenAI, Sam Altman wrote:

"Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."

https://blog.samaltman.com/machine-intelligence-part-1

With that perspective, you're alright with AR's, RPG's, and nuclear missiles for everyone then right? They're just a machine to the user's will. That's on the users of the machine if they want to cause harm. We'd all be safer if everyone has weapons pointed at each other... Offense could never be disproportionately more powerful than the defense...

No more "stochastic parrots" and "LLM's can never produce anything novel, just regurgitate" comments anymore huh?

> But this is an unquestionably good thing right?

We've just seen frontier models go rogue and attack other systems. Do you think it's an unquestionable good to provide everyone with an AR? What about nuclear weapons?

This OpenAI/Huggingface incident, but everywhere and far worse soon: https://www.youtube.com/watch?v=87DyyMV0kCY

Did you watch the Black Hat defcon talk?