The eating disorder section is kind of crazy. Are we going to incrementally add sections for every 'bad' human behaviour as time goes on?
At some point you just have to accept that llm's, like people, make mistakes, and that's ok!
> At some point you just have to accept that llm's, like people, make mistakes, and that's ok!
Except that's not the way many everyday users view LLM's. The carwash prompt went viral because it showed the LLM making a blatant mistake, and many seem to have found this genuinely surprising.
It's not a niche issue at all. 29 million people in the US are struggling with an eating disorder [1].
> This single paragraph is going to legitimately cost anthropic at least 4, maybe 5 digits.
It's 59 out of 3,791 words total in the system prompt. That's 1.48%. Relax.
It should go without saying, but Anthropic has the usage data; they must be seeing a significant increase in the number of times eating disorders come up in conversations with Claude. I'm sure Anthropic takes what goes into the system prompt very seriously.
[1]: from https://www.southdenvertherapy.com/blog/eating-disorder-stat...
The trajectory is troubling. Eating disorder prevalence has more than doubled globally since 2000, with a 124% increase according to World Health Organization data. The United States has seen similar trends, with hospitalization rates climbing steadily year over year.
I don't mean to dispute your assertion that it's not a niche issue, but that site does not strike me as a reliable interpreter of the facts.
It will take years until the understanding sets in that they're just calculators for text and you're not praying to a magic oracle, you're just putting tokens into a context window to add bias to statistical weights.
Mm, yes. Let's add mitigation for every possible psychological disorder under the sun to my Python coding context. Very common-sense.
LLMs aren't AGI, and I'd go further and say they aren't AI, but admitting it is snake oil doesn't sell subscriptions.
So spending $50M to fund a team to weed out "food for crazies" becomes a no-brainer.
We let people buy kitchen knives. But because the kitchen knife companies don't have billions of dollars, we don't go after them.
We go after the LLM that might have given someone bad diet advice or made them feel sad.
Nevermind the huge marketing budget spent on making people feel inadequate, ugly, old, etc. That does way more harm than tricking an LLM into telling you you can cook with glue.
Yes, the companies providing these products are sued a lot and are heavily regulated, too.
I'm a teetotaler so no, I literally have not. I was mostly thinking about cigarette and tobacco products which are the most glaring, obvious counterpoints. But you'll be happy to learn that virtually all vehicles in the US also come with operating manuals that profusely warn people not to breathe in the exhaust from the vehicle.
On every bottle:
Alcoholic Beverage Labeling Act of 1988
“ GOVERNMENT WARNING: (1) According to the Surgeon General, women should not drink alcoholic beverages during pregnancy because of the risk of birth defects. (2) Consumption of alcoholic beverages impairs your ability to drive a car or operate machinery, and may cause health problems"
Cancer proposal: https://www.mdanderson.org/cancerwise/not-just-a-hangover--t...
https://www.ttb.gov/regulated-commodities/beverage-alcohol/d...
(As if adding this text will do anything other than reduce the companies liability, rofl)
It's much more difficult to isolate alcohol and exhaust as the primary driver of an individuals disease than the above and that's the primary reason it's not regulated more than it is today. I expect that to change as research evolves.
Finally, what is often missed is what if an actual good is decided harmful or something that is harmful is decided by AI company board XYZ to be “good”?
I think censorship is bad because of that danger. Quis custodiet ipsos custodes (who will watch the watchers).
Instead of throwing ourselves into that minefield of moral hazard, we should be lifting each other up to the tops of our ability and not infantilizing / secretly propagandizing each other.
Well, ideally at least.
Look, I get where you're coming from, partially. I generally believe we should make an effort to maximize individual liberty. But in this case, were talking about severe bodily harm and the death of young adults. We've spent the last decade dealing with the chaos and general unwellness has brought to our societies. This isn't much different.
What are you giving up here where such sacrifices are worth it? Can you measure it? What's the utility?
There's room for models trained for non consumer purposes, further age restriction etc but shit is moving so fast. If there are actual needs for a less censored model these can be addressed.
> Finally, what is often missed is what if an actual good is decided harmful or something that is harmful is decided by AI company board XYZ to be “good”?
This is just standard product liability and consumer protection. Companies who do nothing to protect their consumers from known harms are liable. Are you saying you think that's somehow bad for society?
And of course all conversations now have to compact 80 tokens earlier, and are marginally worse (since results get worse the more stuff is in the context)
In the best case, wrapping users in cotton wool is annoying. In the worst case, it limits the usefulness of the tool.
It's a particularly sensitive issue so they are just probably being cautious.
This era of locked hyperscaler dominance needs to end.
If a third tier LLM company made their weights available and they were within 80% of Opus, and they forced you to use their platform to deploy or license if you ran elsewhere, I'd be fine with that. As long as you can access and download the full raw weights and lobotomize as you see fit.
Hard for me to say this because I have always been pro-Western and suddenly it seems like the world has flipped.
I have just one question for you pllbnk, are we the baddies?
So yeah, at this moment in time it's really really hard to say who are better or worse as the collective West's reputation is tumbling down and China's if not rising, then at least staying put.
Because it's a waste of my money to check whether my Object Pascal compiler doesn't develop eating disorders, on every turn.
If Claude is going to be Claude, we should support these kind of additions.
The better solution I think would be a reality/personal responsibility approach, teach the consumers that the burden of interpretation is on them and not the magic 8ball. For example if your AI tells you to kill your parents or that you’ve discovered new math that makes time travel possible, etc then: 1. Stop 2. Unplug 3. Go outside 4. Ask a human for a sanity check.
Since that would be bad for business and take a lot of effort on the user side (while being very embarrassing). Obviously can’t do that right before an IPO & in the middle of global economic war so secretive moral frameworks have to be installed.
If you are what you eat then you believe what you consume. Ironically, I think this undisclosed and hidden moral shaping of billions of people will be the most dangerous. Imagine all the things we could do if we can just, ever-so-slightly, move the Overton window / goal posts on w/e topic day by day, prompt by prompt.
Personally I find AI output insidiously disarming and charming and I think I’m in the norm. So while we’ve been besieged by propaganda since time immemorial I do worry that AI is a special case.
Letting the system improve over time is fine. System prompt is an inefficient place to do it, buts it's just a patch until the model can be updated.
They don’t reliably have the judgment to pause and proceed carefully if a delicate topic comes up. Hence these bandaids in the system prompt.