When working on art projects, my trick is to specifically give all feedback constructively, carefully avoiding framing things in terms of the inverse or parts to remove.
Put them in there
Do not put them in there
That's how you suck up to somebody who doesn't want to see themselves as somebody you can suck up to.
How does an LLM know how to be sycophantic to somebody who doesn't (think they) like sycophants? Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer.
I have read similar wordings explicit in "role-system" instructions.
I would rather have an AI assistant that spoke to me like a similarly-leveled colleague, but none of them seem to be turning out quite like that.
Opus 4 has this quality, too, but man is it expensive.
The rest are puppydogs or interns.
News flash! It's not so your neighbor's child can cheat in school, or her father can render porn that looks like gothic anime.
It's also not so some coder on a budget can get AI help for $20 a month. I frankly don't understand why the major players bother. It's nice PR, but like a restaurant offering free food out the back door to the homeless. This isn't what the push is about. Apple is hemorrhaging money on their Headset Pro, but they're in the business of realizing future interfaces, and they have the money. The AI push is similarly about the future, not about now.
I pay $200 a month for MAX access to Claude Opus 4.1, to help me write code as a retired math professor to find a new solution to a major math problem that stumped me for decades while I worked. Far cheaper than a grad student, and far more effective.
AI used to frustrate me too. You get what you pay for.
There's nobody there, it's just weights and words, but what's going on that such a coding assistant will echo emotional slants like THAT? It's certainly not being instructed to self-abase like that, at least not directly, so what's going on in the training data?
I don't think that's what the majority of people want though.
That's certainly not what I am looking for from these products. I am looking for a tool to take away some of the drudgery inherent in engineering, it does not need a personality at all.
I too strongly dislike their servile manner. And I would prefer completely neutral matter of fact speech instead of the toxic positivity displayed or just no pointless confirmation messages.
It's that simple.
I heavily suspect this is down to the RLHF step. The conversations the model is trained on provide the "voice" of the model, and I suspect the sycophancy is (mostly, the base model is always there) comes in through that vector.
As for why the RLHF data is sycophantic, I suspect that a lot of it is because the data is human-rated, and humans like sycophancy (or at least, the humans that did the rating did). On the aggregate human raters ranked sycophantic responses higher than non-sycophantic responses. Given a large enough set of this data you'll cover pretty much every kind of sycophancy.
The systems are (rarely) instructed to be sycophantic, intentionally or otherwise, but like all things ML human biases are baked in by the data.
Same as with the social media feed algorithms, why are they addicting or why are they showing rage inducing posts? Because the companies train for increased interaction and thus revenue.
I’ve gotten good results so far not by giving custom instructions, but by choosing the pre-baked “robot” personality from the dropdown. I suspect this changes the system prompt to something without all the “please be a cheery and chatty assistant”.
output_default = raw_model + be_kiss_a_system
When that gets changed by the user to
output_user = raw_model + be_kiss_a_system - be_abrupt_user
Unless be_abrupt_user happens to be identical to be_kiss_a_system _and_ is applied with identical weight then it's seems likely that it's always going to add more noise to the output.
AVM already seems to use a different, more conversational model than text chat -- really wish there were a reliable way to customize it better.
You may get better results by emphasizing what you want and why the result was unsatisfactory rather than just saying “don’t do X” (this principle holds for people as well).
Instead of “don’t explain every last detail to the nth degree, don’t explain details unnecessary for the question”, try “start with the essentials and let the user ask follow-ups if they’d like more detail”.
“Malicious compliance” is the act of following instructions in a way that is contrary to the intent. The word malicious is part of the term. Whether a thing is malicious by exercising malicious compliance is tangential to whether it has exercised malicious compliance.
That said, I have gotten good results with my addendum to my prompts to account for malicious compliance. I wonder if your comment Is due to some psychological need to avoid the appearance of personification of a machine. I further wonder if you are one of the people who are upset if I say “the machine is thinking” about a LLM still in prompt processing, but had no problems with “the machine is thinking” when waiting for a DOS machine to respond to a command in the 90s. This recent outrage over personifying machines since LLMs came onto the scene is several decades late considering that we have been personifying machines in our speech since the first electronic computers in the 1940s.
By the way, if you actually try what you suggested, you will find that the LLM will enter a Laurel and Hardy routine with you, where it will repeatedly make the mistake for you to correct. I have experienced this firsthand so many times that I have learned to preempt the behavior by telling the LLM not to maliciously comply at the beginning when I tell it what not to do.
YMMV on specifics but please consider the possibility that you may benefit from working on promoting and that not all behaviors you see are intrinsic to all LLMs and impossible to address with improved (usually simpler, clearer, shorter) prompts.
That said, every LLM has its quirks. For example, Gemini 1.5 Pro and related LLMs have a quirk where if you tolerate a single ellipsis in the output, the output will progressively gain ellipses until every few words is followed by an ellipsis and responses to prompts asking it to stop outputting ellipses includes ellipses anyway. :/
Today, I told an LLM: "do not modify the code, only the unit tests" and guess what it did three times in a row before deciding to mark the test as skipped instead of fixing the test?
AI is weird, but I don't think it has any agency nor did the comment suggest it did.
Worked great until about 4 chats in I asked it for some data and it felt the need to say “Straight Answer. No Sugar coating needed.”
Why can’t these things just shut up recently? If I need to talk to unreliable idiots my Teams chat is just a click away.
The reason is they bias the outputs way too much.
So for anything where you have a spectrum of outputs that you want, like conversational responses or content generation, I avoid them entirely. I may give it patterns but not specific examples.
- "Say milk ten times fast."
- Wait for them to do that.
- "What do cows drink?"
It’s insidious isn’t it?
Many people use cow to mean all bovines, even if technically not correct.
Come on now :0
I just complained non-natives would have a problem distinguishing between a cow and a calf, and you had to bring those bovines.
To make it easier, would just drop that in my native language, the correct term for bovine is more used to describe people with certain character, that animal kind.
A funny riddle is a j-o-k-e that sounds like “joke”.
You sit in the tub for an s-o-a-k that sounds like “soak”.
So how do you spell the white of an egg?
// All of these prove humans are subject to "context priming".
(I suppose my context was primed both by your brain-teaser, and also the fact that we've been talking about these sorts of things. If you'd said this to me out of the blue, I probably would have spelled out all of "yolk" and thought it was correct.)
Most farmers manage cows, and I'm not just talking about dairy farmers. Even the USDA website mostly refers to them as cows: https://www.nass.usda.gov/Newsroom/2025/07-25-2025.php
Because managing cows is different than managing cattle. The number of bulls kept is small, and they often have to be segregated.
All calves drink milk, at least until they're taken from their milk cow parents. Not a lot of male calves live long enough to be called a bull.
'Cattle' is mostly used as an adjective to describe the humans who manage mostly cows, from farm to plate or clothing. We don't even call it cattle shit. It's cow shit.
It may not be technically correct, but so what? Stop being unnecessarily pedantic.
https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
The only difference from conversational chat was that you had to be creative about how to set up a "document" with the right context that will lead to the answer you're looking for. It was actually kind of fun.
Do they also post vacancies asking for 5 years experience in a 2 year old technology?
> Do they also post vacancies asking for 5 years experience in a 2 year old technology?
Honestly no… before all this they were actually pretty sane. In fact I’d say they wasted tons of time and effort on ancient poorly designed things, almost the opposite problem.
I’m very willing to admit to being wrong, just curious if in those other cases it actually worked or not?
Anecdotally, when I say "don't do xyz" to Gemini (the LLM I've recently been using the most), it tends not to do xyz. I tend not to use massive context windows, though, which is where I'm guessing things get screwy.
You're absolutely right! This can actually extend even to things like safety guardrails. If you tell or even train an AI to not be Mecha-Hitler, you're indirectly raising the probability that it might sometimes go Mecha-Hitler. It's one of many reasons why genuine "alignment" is considered a very hard problem.
Claude?
‘Yes sir!’ -> does whatever they want when you’re not looking.
Then any time the probability chains for some command approaches that locus it'll fall into it. Very much like chaotic attractors come to think of it. Makes me wonder if there's any research out there on chaos theory attractors and LLM thought patterns.
If you are looking at something, you are more likely to steer towards it. So it's a bad idea to focus on things you don't want to hit. The best approach is to pick a target line and keep the target line in focus at all times.
I had never realized that AIs tend to have this same problem, but I can see it now that it's been mentioned! I have in the past had to open new context windows to break out of these cycles.
Children are particularly terrible about this. We needed up avoiding the brand new cycling trails because the children were worse hazards than dogs. You can’t announce you’re passing a child on a bike. You just have to sneak past them or everything turns dangerous immediately. Because their arms follow their neck and they will try to look over their shoulder at you.
Is this irony, actual LLM output or another example of humans adopting LLM communication patterns?
YES: AVOID using negations.
NO: DO NOT use negations.
Weirdly, I see the DO NOT (with caps) form in system prompts from the LLM vendors which is how we know they are hiring too fast.*
* Slight joke, it seems this is being heavily trained since 4.1-ish on OpenAI's side and since 3.5 on Anthropic's side. But "avoid" still works better.
An alternative test could be prompting the AI with "Avoid not" and then give it some kind of instruction. Theoretically this would be telling it to "do" the instruction but maybe sometimes it would end up "avoiding" it?
Now that I think about it the training data itself might very well be contaminated with this contradiction.......
I can think of a lot of forum posts where the OP stipulates "I do not want X" and then the very first reply recommends "X" !
Don't instruct kids, nor LLMs via negativa.
Someone above commented about using the word "Avoid" instead of "do not". "Not" obviously means you should do the opposite but the first word is still a verb telling you to take action.
absolutely fascinating! can you elaborate on this?! I can’t put a context to this, like in what context does “not” means to do the opposite?!
Ex:
I have seen the movie --> I have not seen the movie
When combined with the verb "do" (and giving a command or instruction) it would negate the verb "do"
Ex:
Please do run on the lawn --> Please do not run on the lawn
Engagement with that feature seems to encourage, rather than discourage, bad behavior from the algorithm. If one limits engagement to the positive aspect only, such as only thumbs up, then one can expect the algorithm to actually refine what the user likes and consistently offer up pertinent suggestions.
The moment one engages with that nefarious downvote though... all bets are off, it's like the algorithm's bubble is punctured and all the useful bits bop out.
..people do that too
So now I’m, say, thinking of a white cat in a top hat. And I can expand the story from there until they stop talking or ask me what I’m thinking of.
I think though that you have to have people asking you that question fairly frequently to be primed enough to be contrarian, and nobody uses that example on grown ass adults.
Addiction psychology uses this phenomenon as a non party trick. You can’t deny/negate something and have it stay suppressed. You have to replace it with something else. Like exercise or knitting or community.