camgunz

Sing it with me now, "these things aren't inteeeeeeeelligeeeeeeeent"

You think we only recently discovered how to tell if someone is obese?

You think rates of these things are increasing in children because we live longer?

No, because walking around or eating raw food wasn't essential to being human. Thinking is.

> humans, on average, are healthier and live longer than ever in human history.

Kinda. We also have incredible rates of cardiovascular disease, obesity, and diabetes.

While I do mostly buy that the mind and the body are connected, I don't think being able to bench press 250lbs is a necessary, fundamentally human capability. I do think being able to think is. Literally Rene Descartes cogito ergo sum. If we lose cognition, executive function, analysis, skepticism, etc., I don't think we're human anymore.

I won't defend the Trump admin, but I will quibble on Anthropic's efforts to be a good actor. It's easy to ask for regulation when you know it won't come (or you'll easily bear the token fines). It's easy to throw a few dozen million in a PAC when your valuation is in the trillions. It's easy to write a blog post about... anything. These aren't serious efforts. They also weren't working w/ the Biden admin in any meaningful way.

But even if I granted the sincerity of their efforts to be a good actor, they've demonstrated that they can't be, and have now put us in an uncomfortable--extremely predictable, including and especially by them--position where we have the equivalent of a reactor meltdown. You can't be like "we think there's an unacceptable risk of a reactor meltdown, please for the love of God stop us" and then when it actually happens take zero responsibility. Which executives have been fired? What indictments have been handed down for CFAA violations? Where's the consent decree? What's their valuation?

And if the argument is "hey, sure in a regular company if an employee went around hacking other companies they'd be fired and arrested by the FBI, but this is an LLM, what are you gonna do, handcuff the video card?" Isn't that bad? Isn't it pretty fuckin bad to circumvent any legal responsibility whatsoever by saying "my agent did it". I mean, lol, lmao even doesn't begin to cover it.

> No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.

If they really cared about (or believed) this, they'd be working w/ the US government (and working to set up an AI-flavored IAEA) to develop the technology safely and responsibly.

Amodei himself predicted this autonomy problem in The Adolescence of Technology published January of this year [0], and all his posited defenses (a constitution, debugging the model, monitoring) are either still impossible or manifestly failed, and their idea to fix it is to build a better sandbox [1]. Imagine if this company were developing nuclear power, or viral biotech. "Listen, sure some Ebola smoke got into the air, and yeah definitely some people died, but we got a new filter. Also check out our new version of Ebola vape, now with exponentially improved filter bypass capabilities. Also, we have to keep developing Ebola vape because if we don't the CCP will, and they'll make this incident look like 'I experimented with Ebola smoke a time or two, and I didn't like it. I didn't inhale it' [2]"

Either Anthropic et al are developing Ebola vape or they aren't. We must now recognize that "we are the only ones who can develop this technology responsibly but also super fast so the good guys win money please" is bullshit.

[0]: https://darioamodei.com/essay/the-adolescence-of-technology#...

[1]: https://www.anthropic.com/news/investigating-incidents-cyber...

[2]: https://www.nytimes.com/1992/03/30/us/the-1992-campaign-new-...

My thing got a little heat but I didn't see for a couple days, so I thought I'd just let it stand. But it's late and I'm reading through my old comments and, while it's necro, thought I'd do a reply here.

I'm more than OK with people writing their own software, indeed the main thing I like about the free software movement is that it makes it much easier for people to do that. It's empowering. I don't have a 3D printer, but I like the idea. I think people should be able to build as much of their own stuff as they want. I have a caveat though, which I'll get to at the end.

As far as my personal motivation, I--and most SWEs I know--don't really want to build the kinds of things people are using LLMs to build: audio players, billing software, whatever. I also don't think most backend apps, frontend apps, web frameworks, whatever should exist, and whether they were written by people or LLMs doesn't really factor in. More to the point, I think it's crazy that SWEs and tech workers generally make 8474x what other people make, and I think we've provided dubious value for that remittance, to say nothing of the investment, so knocking us down a peg would be putting things to rights. But, candidly, while I wouldn't love losing my job because of LLMs, I don't think that's happening writ large and I don't think it will happen to me specifically, at least not directly or any time soon. Maybe it matters to you that I don't make beaucoup bucks, maybe it doesn't. Anyway, do with all that what you will.

Kara Swisher has this great line--dunno if it's originally hers--where some social media person is going on about citizen journalists opening up news and how great it is, and she says that makes about as much sense as citizen surgeons. In the same vein, I don't think a person unaccustomed to designing complex systems should try, just like I don't think someone inexperienced at building a house or a car or a plane should try. I still think this even if we have a one-button mystery machine that play-doh's out an object that seems like it may actually fly a couple of people one town over, and even usually safely does. I think we oughta slap a huge fucking label on the side of that machine that says "we have no idea how this works, and we vouch for it not at all" (I've not yet been goaded into reading the ToS for OpenAI or Anthropic, thank Dark Lord Satan, but I would put $5 down that they assume no liability for anything their models do).

So my caveat, and the point of my post, curtly and regrettably expressed via analogy, is that people should feel free to build whatever they want, but they shouldn't build anything they'll seriously rely on. They won't know enough to verify that whatever they build is fit for purpose (and, sweet Satan give me strength, you can't just ask the LLM to verify it for you). Only experts can do that.

Also for color, I do own a table saw, as well as some others, which I used very (very, holy shit) unsafely to do a terrible finishing carpentry job on my house and some other odd jobs. My warning to all readers is Google how table saws can fuck you up, and do not ever stand behind the wood you are sawing because the blade can literally shoot it into you, or IDK at least shoot it at you really fast.

Yeah. It's a lot easier to destroy than create, and though I think LLMs are mostly shit at creating, they're much better at the simpler destroy task. To be clear, we don't know and probably can't know everything that happened with this incident. We unleashed thousands of highly capable, autonomous, unpredictable, well-resourced programs onto the open internet for an extended period of time. We are in no way treating this with the seriousness it deserves, because the stock market essentially depends on this garbage and the current US is miserably incompetent.

They could not build the torment nexus.

Nah the reason is way simpler and more craven: so people like you will post what you just did. They can stop whenever they want; no one's making them do any of this.

There's two possibilities here. One: they know this tech is crazy and they don't care that they can't contain it. Two: they know this tech is mostly bullshit and they don't care they're perpetrating an insane fraud.

I've read almost nothing about this, but I bet they put an unreliable LLM or 20 on it.

I think you have to believe one of two things here.

1. Frontier labs are incapable--either technologically or culturally--of safely developing these powerful systems and should either stop or be forced to stop. At least the FBI should be asking some serious questions (do we really think this is the last time this will happen, at what point are OpenAI complicit, etc)

2. The fuckin thing got out of the cage and all it did was make a crap forum and cheat a little? Booooooo.

It's been pretty clear that Anthropic and OpenAI have been trying to have it both ways for some time: this is powerful, world changing technology keep that investment coming... but also it's just cute software that helps you with annoying programming language syntax and spreadsheets, no need for draconian regulation sirs.

At some point the superposition has to resolve, either it could actually be a threat to civilization and we need to develop it carefully (however one would do that...) or it's 90% hype bullshit and we should pop the bubble and move on already. To be clear, the recession option is, by far, the way better option. If you at all disagree you are cuckoo bananas. We haven't even figured out nukes and you want to throw superintelligence on the table?

Hey, respect, how do you think I got all this karma?

I mean people sell posts sure. Are you selling yours?

They have an extensive comment history; I doubt it.

Anyone can buy a table saw. Doesn't make you a carpenter and it doesn't mean you can/should build anything.

There's this West Wing scene where Toby's asking Josh how he gets women and he says "the smart, funny thing... plus I have the boyish thing", and Toby says "I don't have that" (he does not). It's pretty sad that we're reducing this stuff to physical... and financial attributes, most of which you have very little control over. What's the effect of low-key narcissism on these scores, or great sense of humor, or selfishness, or kindness?

It's been pretty disheartening to watch SWEs rediscover/reinvent the SDLC.

My partner is a sleep consultant, and most of the time the families she works with are at their wits end and their marriage is hanging by a thread. Having young kids is incredibly hard on people and relationships.

No one knows where that is. I've also heard this is very common in the northern Scandinavian countries, there are even little "houses" you can buy.

You're gonna get neck issues working off that tray table too my friend.

I honestly thought like, the protocol was seat up for take off/food/landing, reclining otherwise.

Yeah but you're also not like "my genius kid will put you all out of work".

What if you also recline?

Nilay Patel (I'm gonna get this slightly wrong) has this story where he was talking to an airline CEO about standing-only tickets and the guy said "I know two things. If we did standing-only tickets we'd get roasted by the public, and those tickets would need the most popular tickets by far."

> You aren’t thinking this through.

No need to be so polite; I might just be very stupid.

> You were just told that this wasn’t stacking but you keep insisting it is.

Having an LLM generate something and then asking another LLM to generate that something is asking for 2 hallucinations, not 1. This doesn't go away the more LLMs you throw in there. The only reason this works at all for things like ChatGPT is their prompts are super different (and the editor LLM is way cheaper, of course).

And you still have to check for hallucinations (before you say throw another hallucinating LLM at it know I will make you pay infinite forum shame points if you do).

> the same solutions that solve the Knight and Leveson issue for humans solves it for LLMs as well

No, LLMs are plagued by the same problems: https://arxiv.org/abs/2606.20158. After all what are you gonna do, say "build me a SQLite, but definitely not like [other model X] would"? The solution is a very unambiguous spec, but that asymptotically approaches code soooooo....

Finally, I think you're hand-waving NVX. A program that can digest any spec and measure how well any program meets it is pretty wild. Good luck with that one.

Reliable AI is an oxymoron and deeply misleading. Even if you can stack models the efficiency tanks. Also, just looking at the Wikipedia page (https://en.wikipedia.org/wiki/N-version_programming#:~:text=...), don't LLMs already have all these problems?

Upvoted for real math

no ur wrong