Hacker News Reader

> I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech

I am going to say it. The CEO of a pre-IPO company has extreme incentive to bolster the tech he is selling, to the point where his every action should be viewed as only in service to that goal. Every word he says should be viewed critically through that lense. He is not making this post out of the goodness of his heart, he is doing it in service to the IPO. If it happens to align with your views that's great, but it's still just a marketing stunt to get people with your views to buy in. Don't be fooled. Buy in if you feel it's a good deal, not because of the CEO's marketing.

In terms of aeronautics, went from the Wright brothers to the moon in 40 years. After that, everyone understandably thought that we'd be living in space and flying everywhere with personal jet packs in another couple years. Little did they know, it was the top of the S-curve, not the middle.

In the 60 years since, we've barely been able to adapt the 737 to fly longer routes.

Sure, I have no doubt that AI progress will follow an S-curve. The question is, where are we on it and is the plateau at a level safe for humanity? That's a very difficult thing to estimate without a crystal ball and not a risk I want to take.

Or simply we use AI and see on the ground what it can and can't do. I can generally trust an agent for solved problems, but the more something deviates from established industry standards (i.e. what was relentlessly scraped) I have an increasingly harder time not having constant oversight of what it's doing, no matter the specs I put on the md.

Personally I feel most of the improvement in the last year comes from tooling/integration (MCPs, realtime documentation access, treesitter support, orchestration) than from the models themselves, in the last year. And still frontier models would routinely come up with bs until you tell them to actually use those tools.

You're talking as if this is a static thing though. It's the God of Gaps [1] but for humanity's special sauce.

Two years ago, I couldn't trust an LLM to do anything that wasn't straight forward boiler plate.

One year ago, I was pretty solid at writing algorithms that were combinations of existing ideas.

Now, Fable is outputting stuff that I would genuinely consider to be creative and original if a colleague had presented it to me.

Yes, maybe the code style still isn't great, but given the pattern of the last few years, it feels correct (a priori) to assume that this gap isn't going to keep closing.

[1] https://en.wikipedia.org/wiki/God_of_the_gaps

I agree that AI poses a threat to society. I act on this by not developing world-leading AI models and offering them to anyone willing to pay top dollar, while funneling that money back into accelerating AI capabilities development. Maybe Dario would consider taking a similar ethical position? Maybe he would support restrictions and taxes on data center construction, in order to slow down the pace?

If Anthropic were not developing these models, one of many other companies would be. I think it's good that the CEO of the current world-leader is at least considering these discussions and platforming possible solutions.

The fact that he doesn't support more restrictive approaches that don't align with his incentives doesn't invalidate the points he is making.

> If Anthropic were not developing these models, one of many other companies would be.

"Everyone else is doing it" but even worse as nobody else is actually even doing it and you just made up that part.

Just commenting on this horrible justification trick in general.

It’s a guy asking to make an end-run around the constitution and the APA regulatory framework based on a flimsy sci fi premise. Naturally it provokes a negative reaction.

Why are they not preaching for protected weights, but public, ie, under state control? What do you feel his posture will be if that is starting to be discussed?

Also, on an unrelated note, why would you have an account for 5 years and only now post your second comment? AI has been an existential threat for years, why only now?

This is a pattern I am seeing all over the place on HN in the last year in AI threads, and I have to admit that I am starting to become paranoid and my feels need some assuaging.

Your first point is very reasonable, and I agree that that is something Dario would likely be more opposed to.

However, my point isn't that I think Dario is our saviour who we should follow the every word of. As with everyone, his opinions should be filtered through the lens of his incentives. That said, I don't understand the knee-jerk reaction by many commenters to completely disregard the many important points he's making.

As for the lack of my account use, I can't comment for others, but I'm just quite shy. I've opened up the comment box many times to write a reply but rarely commit to actually posting it, especially because I feel like I'm not on the side of the general HN consensus.

You write well & HN is better when there are more well-written people on the opposite side of the consensus

Genuinely, thank you. This is very encouraging and makes me feel much better about commenting more going forwards.

The quality of discussion and prose on HN is just generally so high that it can feel quite a bit intimidating to jump in (in contrast to Reddit where I have no worries about commenting haha).

This 100%. It’s great to hear diverse perspectives!

Here's the real reason you are finding these responses in the thread and I'll lay it out in the open.

There are people who simultaneously are scared about AI but refuse to believe that AI is the * real deal * and can do everything Dario thinks it can. These are the same people who think its all marketing hype and Dario is "hyping up" before IPO or some lame conspiracy theory.

It is high time people start accepting the real world performance of LLMs and brace themselves instead of hiding behind two contradictory views

1. AI is hype

2. AI is scary

Those two views aren't contradictory. AI can both be scary, in that the primary dangers are bias, misinformation at scale, and displacement of human labor, and simultaneously be overhyped in the way it is marketed at sold to investors with the "what if this happens in the future" scare mongering.

Notice how in any of these proposed regulations, Dario is talking about future advances. Notice how these suggestions are never implied to apply retroactively to existing models. If AI was SO dangerous, then any future regulations should obviously be retroactive and we should seriously consider restricting access now to the models that already exist.

Hard not to see that as nothing more than a play at regulatory capture and pulling the ladder up behind them.

Humans just aren't very good at dealing with threats that aren't immediate concerns. 'Safety regulations are written in blood' is a saying for a reason. A significant chunk of the population shrugs off climate change, and nearly all fertility rate crises threads are filled with dumb 'hurr lower population good' and/or 'See what Capitalism gets you!' rhetoric - They fundamentally don't even understand what the problem is. So is it really all that surprising that a technology like this would be shrugged off until it's too late ? Especially one with such existential issues for humanity? Some people are still too loathe to admit the clearly intelligent machine is intelligent, devolving into increasingly nonsensical and absurd (and ironically more human demeaning) arguments as model capabilities get better. I'm afraid you're asking for too much.

> admit the clearly intelligent machine is intelligent

citation needed

There is nothing indicating these models are clearly intelligent. Language fluency is not cognitive intelligence, and to think otherwise is falling into the trap of anthropomorphizing the LLMs.

They are still probabilistic engines, there is no causal reasoning still, they only emulate logic, and as far as we know, there is no agency, just the illusion of agency.

The danger here is not existential as you say. We aren't on the cusp of some machine uprising by super intelligence. The threats are algorithmic bias, misinformation at scale, and displacement of human labor.

lol sure buddy, keep telling yourself that

"This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won’t"

https://theonion.com/this-war-will-destabilize-the-entire-mi...

It's denialism, same as climate change, the subconscious fear to really grapple with the actual "what if" alternative scenario. Anthropic are true believers. They got to $1T in 5 years by being exceptionally smart and ahead of the curve here. Meanwhile HN just continually devolves into reactionary cynicism. "must be marketing, they just want to be rich, impossible AI advances much further." Meanwhile at every step of the way, Anthropic and X-riskers / "doomers" are vindicated in their correct predictive beliefs. We're headed to a future far dangerous than nukes very soon. We're in an arms race to detonate one 100x the size

I don’t really think it has to do with general denial in this case. I think he’s likely right in that pushback against datacenters is partially the release of angst over the threats which these models pose but in an indirect fashion since power is so unevenly distributed that local political organizing is the only real mechanism for people to act through. I think everyone feels nervous about these tools. People are scared of autonomous weapons, hackers using agents, and any of the other present issues that already exist. I think some of the pushback that comes in these threads is the fact that people like Dario are so focused on the long-term view that they suppress all immediate threats which already exist. You don’t need to argue about exponential growth or “AI being better than humans at everything” to want to better regulate this technology.

I would argue the best way to safeguard against long-term threats is to start by focusing on the issues that already exist. If you can offset the health risks of local datacenters or issues of unequal distribution of wealth by creating a more equal society right now then you’re already on the path to handling these long term issues. To me, this distant focus only distracts from the already present issues and conceals effective policy in this moment. We do need to safeguard against AI risk and it’s already here. Don’t even get me started on the havoc which recommendation systems have caused in society in the last 15 years which we still don’t call AI because it doesn’t speak.

Tl;Dr: These essays can feel disingenuous because:

1) AI risk is already present without exponential growth. The exponential growth argument often feels like a distraction from the fixes we could put in to fix the current issues that are already here.

2) The people stating this argument often have billions of dollars to gain if it comes true. While they may be altruistic, I also don’t see them doing all that much to fix the issues that people are already claiming exist and instead continue onwards on their path by justifying that they are the only ones responsible enough to handle it if the super intelligence does arrive. By continuing down that path, if that day ever does arrive they’ll have ensured the existence of a system which is unable to handle it.