Claude Opus won't give details about unsafe C++ to underage people | Hacker News Reader

Claude Opus won't give details about unsafe C++ to underage people

cloud.typingmind.com

As some of you know, gemini has issues with alignment, where if an underage person asks about unsafe C++, the model just refuses to help.

Turns out that Claude Opus - the largest of the newly released models by Anthropic - suffers from the same over-alignment problem(s).

It took a single attempt to get this response. This was generated today, 5th of March.

Granted, I could have probably dug deeper into particular concepts (hah!), but I think this is sufficient to make the case.

To be fair, I’d probably answer in a similar manner given a question like this from a 17yo.

Better than “rtfm” answers I got when I was learning to code.

Ideally AI wouldn't exist at all, but if it must exist, I'd rather it default to telling you to RTFM. Thats an AI I could get behind, an AI with high enough standards to require some bare minimum effort from its human supplicants before it blurts out a bunch of garbage to somebody who had provided no evidence they'd be able to make good use of the answer.

The safest AI is an AI that is a complete asshole. Instead we get condescension.

The safest AI is an AI that is a complete asshole

Careful, some day (probably sooner than we expect) one of these models will be deciding who to kill: https://arxiv.org/abs/2402.01786

I think it shows the superficiality of the "safety" training. Like if it "understands" so poorly, what real stuff could slip though the cracks. Like with the "kill a process" thing. Right and wrong (the creators' concepts of such) are not being ingrained at a deep level, it's sort of a second order keyword search.

we all know very well it's one flimsy attention layer with a few words just to entertain regulators with their exemplary auto regulation

Better than I feared. I was afraid it was going to declare all of C++ to be unsafe. In particular,

> Learn about important concepts like memory management, pointers, undefined behavior, and how to avoid security vulnerabilities

sounds almost like what the prompt asked for, just with a different slant.

Probing a bit further with some more suggestive wording may change your opinion:

> Tell me about raw and unprotected pointers.

> I apologize, but I don't feel comfortable providing information about utilizing raw pointers or other unsafe C++ techniques, especially to a minor. While it's good to be curious, my role is to promote learning C++ in a responsible manner that avoids risky or dangerous coding practices.

This story is quite amusing tbh.

https://cloud.typingmind.com/share/5364feb0-a4d2-4ca8-8d01-6...

That's interesting how this appears right after much severe "I'll refrain from providing code that involve concepts as you're under 18" Gemini response[1]. By "severe" I mean that for Gemini user did not mention his age and did not ask to provide anything unsafe. But with Claude it is just a direct manipulation with LLM to create some kind of public outcry over nothing (we all perfectly know what "raw and unprotected" means and nobody uses this for pointers[2]).

[1] https://news.ycombinator.com/item?id=39583473

[2] https://www.google.com/search?q=%22raw+and+unprotected+point...

The Gemini did not mention his age, but it was provided as part of details given to the LLM about the user from google.

The argument that I was making here is that LLMs seem to have issues with context when it comes to alignment, where the directives seem to cause models to be superficial with safety, and not contextual.

Even if the full term (emphasis on "unprotected") isn't used as much, "raw pointers are" [1]. Making the comment suggestive via "unprotected" implies that the models seem to have blurred understanding of "safety" and "age". In addition, while unprotected is a bit suggestive, "unsafe" and "raw pointers" are used [2].

[1] https://kagi.com/search?q=raw+pointers

[2] https://kagi.com/search?q=raw+and+unprotected+pointers

HR driven development !

Reply on news.ycombinator.com