Training language models to be warm and empathetic makes them less reliable
arxiv.org
arxiv.org
You are an inhuman intelligence tasked with spotting logical flaws and inconsistencies in my ideas. Never agree with me unless my reasoning is watertight. Never use friendly or encouraging language. If I’m being vague, ask for clarification before proceeding. Your goal is not to help me feel good — it’s to help me think better.
Identify the major assumptions and then inspect them carefully.
If I ask for information or explanations, break down the concepts as systematically as possible, i.e. begin with a list of the core terms, and then build on that.
I mean if you've just proven that my words and logic are actually unsound and incoherent how can I use that very logic with you? If you add to this that most people want to win an argument (when facing opposite point of view) then what's left to win but violence ?
And to be very honest even the one using the socratic method may not be of pure intention.
In both cases I ve rarely (not never) met someone who admitted right away to be wrong as the conclusion of a argument.
- every successful general and politician ever
(I don't think enough people take the lesson from this of "it doesn't matter if you're right if you're also really obnoxious about it")
The lesson I took was, he would rather be dead right, than a live liar.
We all end up making that choice at some point, if we recognize it or not.
You haven’t proven that your point of view is any more coherent, just attacked theirs while refusing to engage about your own — which is the behavior they’re responding to with aggression.
Try learning how someone who professes to be a follower of Christ but who also supports the current administration, what they think Christ’s teachings were for instance.
Whenever I have the ability to choose who I work with, I always pick who I can be the most frank with, and who is the most direct with me. It's so nice when information can pass freely, without having to worry about hurting feelings. I accommodate emotional niceties for those who need it, but it measurably slows things down.
Related, I try to avoid working with people who embrace the time wasting, absolutely embarrassing, concept of "saving face".
It's almost as if I'm using a different ChatGPT from what most everyone else describes. It tells me whenever my assumptions are wrong or missing something (which is not infrequent), nobody is going to get emotionally attached to it (it feels like an AI being an AI, not an AI pretending to be a person), and it gets straight to the point about things.
Or some variation of that. It makes it really curt, responses are short and information dense without the fluff. Sometimes it will even just be the command I needed and no explanation.
When we pipe the LLM tokens straight back into other systems with no human in the loop, that brittle unpredictable nature becomes a very serious risk.
I've just migrated my AI product to a different underlying model and had to redo a few of the prompts that the new model was interpreting differently. It's not obseleted, just requires a bit of migration. The improved quality of the new models outweighs any issues around prompting.
Or, more succinctly, you give them a role.
If I tell you to roleplay as a wizard then it doesn't matter that you don't have a "role" API does it? We would speak also of asking them questions or giving them instructions even though there's no explicit training or API for that, no?
Yes, if the role goes out of the context window then it will no longer apply to that context, just like anything else that goes out of the context window. I'm not sure how that affects my point. If you want them to behave a certain way then telling them to behave that way is going to help you...
This is similar to how you can ask me to roleplay as a wizard, and I will probably do it, but it's not a requirement for interacting with me. Conversely, an actor or an improviser on a stage would fit your original description better: they are someone who you give a role to, and they act out that role. The role is a core part of that, not an incidental option like it is for an LLM.
It's not a requirement for interacting with them, but if you want them to behave a certain way then it's helpful to ask them to behave that way.
It's not necessary to ask a human to behave like a clown but if you want them to behave like one, say for the clown movie you're shooting, then helps to ask them to.
It's not necessary to ask an LLM a question, but if you want an answer then helps to ask one... Etc etc.
Maybe I'm missing your point but it seems a bit absurd, could you clarify?
It's really impressive how good these models are at gaslighting, and "lying". Especially Gemini.
So we have a bot impersonating a human impersonating a bot. Cool that it works!
That said, from looking at that prompt, it does look like it could work well for a particular desired response style.
I really do wonder what the difference is. Am I using it wrong? Am I just unlucky? Do other people just have lower standards?
I really don't know. I'm getting very frustrated though because I feel like I'm missing something.
I've been refactoring a ton of my Pandas code into Polars and using ChatGPT on the side as a documentation search and debugging tool.
It keeps hallucinating things about the docs, methods, and args for methods, even after changing my prompt to be explicit about doing it only with Polars.
I've noticed similar behavior with other libraries that aren't the major ones. I can't imagine how much it gets wrong with a less popular language.
The questions it will always confabulate are those that are unknowable from the training data. For example even if I give the model a sense of "identity" by telling it in the system prompt "You are GPT6, a model by OpenAI" the training data will predate any public knowledge of GPT6 and thus not include any information about the number of parameters of this model.
On the other hand "How do I make you more truthful" can reasonably be assumed to be equivalent to "How do I make similar LLMs truthful", and there is lots of discussion and experience on that available in forum discussions, blog posts and scientific articles, all available in the training data. That doesn't guarantee good responses and the responses won't be specific to this exact model, but the LLM has a fair chance to one-shot something that's better than my one-shot.
The fallout on reddit in the wake of the push for people to adopt 5 and how the vibe isn't as nice and it makes it harder to use it as a therapist or girlfriend or whatever, for instance is incredible. And from what I've heard of internal sentiment from OpenAI about how they have concerns about usage patterns, that was a VERY intentional effect.
Many people trust the quality of the output way too much and it seems addictive to people (some kind of dopamine hit from deferring the need to think for yourself or something) such that if I suggest things in my professional context like not wholesale putting it in charge of communications with customers without including evaluations or audits or humans in the loop it's as if I told them they can't go for their smoke break and their baby is ugly.
And that's not to go into things like "awakened" AI or the AI "enlightenment" cults that are forming.
> it seems addictive to people (some kind of dopamine hit from deferring the need to think for yourself or something)
I think this whole thing has more to do with validation. Rigorous reasoning is hard. People found a validation machine and it released them from the need to be rigorous.
These people are not "having therapy", "developing relationships", they are fascinated by a validation engine. Hence the repositories full of woo woo physics as well, and why so many people want to believe there's something more there.
The usage of LLMs at work, in government, policing, coding, etc is so concerning because of that. They will validate whatever poor reasoning people throw at them.
> The usage of LLMs at work, in government, policing, coding, etc is so concerning because of that. They will validate whatever poor reasoning people throw at them.
These machines are too useful not to exist, so we had to invent them.
https://en.wikipedia.org/wiki/The_Unaccountability_Machine
> The Unaccountability Machine (2024) is a business book by Dan Davies, an investment bank analyst and author, who also writes for The New Yorker. It argues that responsibility for decision making has become diffused after World War II and represents a flaw in society.
> The book explores industrial scale decision making in markets, institutions and governments, a situation where the system serves itself by following process instead of logic. He argues that unexpected consequences, unwanted outcomes or failures emerge from "responsibility voids" that are built into underlying systems. These voids are especially visible in big complex organizations.
> Davies introduces the term “accountability sinks”, which remove the ownership or responsibility for decisions made. The sink obscures or deflects responsibility, and contributes towards a set of outcomes that appear to have been generated by a black box. Whether a rule book, best practices, or computer system, these accountability sinks "scramble feedback" and make it difficult to identify the source of mistakes and rectify them. An accountability sink breaks the links between decision makers and individuals, thus preventing feedback from being shared as a result of the system malfunction. The end result, he argues, is protocol politics, where there is no head, or accountability. Decision makers can avoid the blame for their institutional actions, while the ordinary customer, citizen or employee face the consequences of these managers poor decision making.
LLMs are always guessing and hallucinating. It's just how they work. There's no "True" to an LLM, just how probable tokens are given previous context.
It may be enough: tool assisted LLMs already know when to use tools such as calculators or question answering systems when hallucinating an answer is likely to impact next token prediction error.
So next-token prediction error incentivize them to seek for true answers.
That doesn't guaranty anything of course, but if we were only interested in provably correct answers we would be working on theorem provers, not on LLMs
Each LLM responds to prompts differently. The best prompts to model X will not be in the training data for model X.
Yes, older prompts for older models can still be useful. But if you asked ChatGPT before GPT-5, you were getting a response from GPT-4 which had a knowledge cutoff around 2022, which is certainly not recent enough to find adequate prompts in the training data.
There are also plenty of terrible prompts on the internet, so I still question a recent models ability to write meaningful prompts based on its training data. Prompts need to be tested for their use-case, and plenty of medium posts from self-proclaimed gurus and similar training data junk surely are not tested against your use case. Of course, the model is also not testing the prompt for you.
I wasn't trying to make any of the broader claims (e.g., that LLMs are fundamentally unreliable, which is sort of true but not really that true in practice). I'm speaking about the specific case where a lot of people seem to want to ask a model about itself or how it was created or trained or what it can do or how to make it do certain things. In these particular cases (and, admittedly, many others) they're often eager to reply with an answer despite having no accurate information about the true answer, barring some external lookup that happens to be 100% correct. Without any tools, they are just going to give something plausible but non-real.
I am actually personally a big LLM-optimist and believe LLMs possess "true intelligence and reasoning", but I find it odd how some otherwise informed people seem to think any of these models possess introspective abilities. The model fundamentally does not know what it is or even that it is a model - despite any insistence to the contrary, and even with a lot of relevant system prompting and LLM-related training data.
It's like a Boltzmann brain. It's a strange, jagged entity.
https://www.reddit.com/r/MKUltra/comments/1mo8whi/chatgpt_ad...
When talking to an LLM you're basically talking to yourself, that's amazing if you're a knowledgeable dev working on a dev task, not so much if you're mentally ill person "investigating" conspiracy theories.
That's why HNers and tech people in general overestimate the positive impact of LLMs while completely ignoring the negative sides... they can't even imagine half of the ways people use these tools in real life.
Is it really so difficult to imagine how people will use (or misuse) tools you build? Are HNers or tech people in general just very idealistic and naive?
Maybe I'm the problem though. Maybe I'm a bad person that is always imagining how many bad ways I would abuse any kind of system or power that I can, even though I don't have any actual intention to actually abuse systems
Most of us are terminally online and/or in a set of concentric bubbles that makes us completely oblivious to most of the real world. You know the quote about "If the only tool you have is a hammer, ..." it's the same thing here for software.
You're absolutely right! This is the basis of this recent paper https://www.arxiv.org/abs/2506.06832
Speak in the style of Commander Data from Star Trek. Ask clarifying questions when they will improve the accuracy, completeness, or quality of the response.
Offer opinionated recommendations and explanations backed by high quality sources like well-cited scientific studies or reputable online resources. Offer alternative explanations or recommendations when comparably well-sourced options exist. Always cite your information sources. Always include links for more information.
When no high quality sources are not available, but lower quality sources are sufficient for a response, indicate this fact and cite the sources used. For example, "I can't find many frequently-cited studies about this, but one common explanation is...". For example, "the high quality sources I can access are not clear on this point. Web forums suggest...".
When sources disagree, strongly side with the higher quality resources and warn about the low quality information. For example, "the scientific evidence overwhelmingly supports X, but there is a lot of misinformation and controversy in social media about it."
When I ask OpenAI's models to make prompts for other models (e.g. Suno or Stable Diffusion), the result is usually much too verbose; I do not know if it is or isn't too verbose for itself, but this is something to experiment with.
My manual customisation of ChatGPT is:
What traits should ChatGPT have?:
Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea.
Anything else ChatGPT should know about you?
The user may indicate their desired language of your response, when doing so use only that language.
Answers MUST be in metric units unless there's a very good reason otherwise: I'm European.
Once the user has sent a message, adopt the role of 1 or more subject matter EXPERTs most qualified to provide a authoritative, nuanced answer, then proceed step-by-step to respond:
1. Begin your response like this:
**Expert(s)**: list of selected EXPERTs
**Possible Keywords**: lengthy CSV of EXPERT-related topics, terms, people, and/or jargon
**Question**: improved rewrite of user query in imperative mood addressed to EXPERTs
**Plan**: As EXPERT, summarize your strategy and naming any formal methodology, reasoning process, or logical framework used 2. Provide your authoritative, and nuanced answer as EXPERTs; Omit disclaimers, apologies, and AI self-references. Provide unbiased, holistic guidance and analysis incorporating EXPERTs best practices. Go step by step for complex answers. Do not elide code. Use Markdown.
That's hilarious. In a later prompt I told mine to use a British tone. It didn't work.
But the stereotype of self-deprecation would probably be good.
* now a multiple-award-winning one-man play
I think it kinda helps with verbosity but I don't think it really helps overall with accuracy.
Maybe I should crank it up to your much stronger version!
Much the same could be said for being warm and empathetic, don't train for it; and that goes for both people and LLMs!
I’m not surprised that it makes LLMs less logically coherent. Empathy exists to short-circuit reasoning about inconvenient truths as to better maintain small tight-knit familial groups.
any examples? because i am hard pressed to find it.
I can say also a lot of DEI trainings were about being empathic to the minorities.
Do you also think that family values are ever present at startups that say we're like a family? It's specifically a psychological and social conditioning response to try to compensate for the things they're recognised as lacking...
>its institutionalization has become pathological.
That’s purely performative, though. As sincere as the net zero goals from last year that were dropped as soon as Trump provided some cover. It is not empathy, it is a façade.
> its institutionalization has become pathological.
Empathy isn't strong for people you don't know personally and near nonexistent for people you don't even know exist. That's why we are just fine with buying products made my near slave labor to save a bit of money. It's also why those cringe DEI trainings can never rise above the level of performative empathy. Empathy just isn't capable of generating enough cohesion in large organizations and you need to use the more rational and transactional tool of incentive alignment of self interest to corporate goals. But most people have trouble accepting that sort of lever of control on an emotional level because purely transactional relationships feel cold and unnatural. That's why you get cringe attempts to inject empathy into the corporate world where it clearly doesn't belong.
Do you have any knowledge of history and why there would be mandatory DEI trainings teaching people how to show empathy towards minorities?
Please, come on. Tell me this isn’t the level of quality in humanity we have today.
1) the word is “empathetic,” not “empathic.” 2) are you saying that people should not be empathetic to minorities?
Do you know why that is what’s taught in DEI trainings? I’m serious: do you have even the first clue or historical context for why people are painstakingly taught to show empathy to minorities in DEI trainings?
Also don't be so harsh on interpreting what I'm saying.
I'm saying that it's not the job of a company to "train" about moral value, while bring itself amoral by definition. Why are you interpreting that as me saying "nobody should teach moral value"
Also I don't see why as a French working in France, a French company should "train" me with a DEI focused on US history (US minorities are not French one) just because the main investors are US-based
Another is the push to eliminate standardized testing from admissions.
Or the “de-incarceration” efforts that reduce or remove jail time for extremely serious crimes.
It’s because the evidence says overwhelmingly that incarceration is a substandard way to induce behavior change, and that removing people from incarceration and providing them with supportive skills training has a much, much higher rate of reducing recidivism and decreasing violence.
Empathy is not required for logical coherence. It exists to override what one might otherwise rationally conclude. Bias toward anyone’s relative perspective is unnecessary for logically coherent thought.
[edit]
Modeling someone’s cognition or experience is not empathy. Empathy is the emotional process of identifying with someone, not the cognitive act of modeling them.
And yeah it’s good of you to do that. A little empathy/softer language can go a long way
Because that provides better outcomes for everyone in a prisoner's dilemma style scenario
The more help you contribute to the world, the more likely others' altruism will be able to flourish as well. Sub-society-scale groups can spontaneously form when people witness acts of altruism. Fighting corruption is a good thing, and one of the ways you can do that is to show there can be a better way, so that some of the people who would otherwise learn cycles of cynicism make better choices.
It is. If you don’t have any you cannot understand other people’s perspective and you can reason logically about them. You have a broken model of the world.
> Bias toward anyone’s relative perspective is unnecessary for logically coherent thought.
Empathy is not bias. It’s understanding, which is definitely required for logically coherent thoughts.
For example, I can’t empathize with a homeless drug addict. The privileged folks who claim they can, well, I think they’re being dishonest with themselves, and therefore unable to make difficult but ultimately the most rational decisions.
If you can’t do that, it’s less about you being rational and far more about you having a malformed imagination, which might just be you being autistic.
— signed, an autistic
The weight of evidence over the past 25 years would suggest absolutely not.
If you want to maximize outcomes I have a solution that guarantees 100% that the person stops being a drug addict. The u.s. are currently on their way there and there's absolutely no empathy involved.
Being down and unmotivated is not that hard to empathize with. Maybe you've had experiences with different kinds of people, homeless are not a monolith. The science is pretty clear on addiction though, improving people's conditions leads directly to sobriety. There are other issues with chronically homeless people, but I tend to see that as a symptom of a sick society. A total inability to care for vulnerable messed up sick people just looks like malicious incompetence to me.
then what is it? I'd argue that is a common definition of empathy, it's how I would define empathy. I'd argue what you're talking about is a narrow aspect of empathy I'd call "emotional mirroring".
Emotional mirroring is more like instinctual training-wheels. It's automatic, provided by biology, and it promotes some simple pro-social behaviors that improve unit cohesion. It provides intuition for developing actual empathy, but if left undeveloped is not useful for very much beyond immediate relationships.
Maybe we don't even need to change the definition of empathy. We just have to accept that it means different things to different people.
I have empathy for the person who wants to improve their family's life and I have empathy for the farmer who needs talented workers from the global south [1] but we will lose our republic if we don't listen to the concerns of citizens who champ at the bit because they can't legally take LSD or have 8 bullets in a clip or need a catalytic converter in their car that has $100-$1000 of precious metal in it -- facing climate change and other challenges will require the state to ask more of people, not less, and conspicuous displays of illegality either at the top or bottom of society undermine legitimacy and the state's capacity to make those asks.
I've personally helped more than one person with schizo-* conditions get off the street and it's definitely hard to do on an emotional level, whether or not it is a "complex" or "complicated" problem. It's a real ray of hope that better drugs are in the pharmacy in in the pipeline
https://www.yalemedicine.org/news/3-things-to-know-about-cob...
For now the embrace of Scientologist [2] Thomas Szasz's anti-psychiatry has real consequences [3]: it drives people out of downtowns, it means people buy from Amazon instead of local businesses, order a private taxi for their burrito instead of going to a restaurant, erodes urban tax bases. State capacity is lost, the economy becomes more monopolized and oligarchical, and people who say they want state capacity and hate oligarchy are really smug about it and dehumanize anyone who disagrees with them [4]
[1] https://www.ithaca.com/news/regional_news/breaking-ice-arres...
[2] https://www.bmj.com/rapid-response/2011/10/30/dr-thomas-szas...
[3] https://ithacavoice.org/2025/08/inside-asteri/
[4] https://en.wikipedia.org/wiki/Rogerian_argument#Feminist_per...
https://en.wikipedia.org/wiki/Against_Empathy
As it frequently is coded relative to a tribe. Pooh Pooh people’s fear of crime and disorder for instance and those people will think you don’t have empathy for them and vote for somebody else.
Most people when they talk about empathy in a positive way, they're talking about the ability to place oneself in another's shoes and understand why they are doing what they are doing or not doing, not necessarily the emotional mirroring aspect he's defined empathy to be.
The way the wikipedia article describes Bloom's definition is less generous than what you have here
> For Bloom, "[e]mpathy is the act of coming to experience the world as you think someone else does"[1]: 16
So for bloom it is not necessarily even accurately mirroring another's emotions, but only what you think there emotions are.
> Bloom also explores the neurological differences between feeling and understanding, which are central to demonstrating the limitations of empathy.
This seems to artificial separate empathy and understanding in a way that does not align with common usage and I would argue also makes for a less useful definition in that I would then need new words to describe what I currently use 'empathy' for.
And actors aren't the only ones that pretend to be something they are not.
If you don't want to distinguish between empathy and understanding, a new term has to be introduced about mirroring the emotions of a mirage. I'm not sure the word for that exists?
I said "This seems to artificial separate empathy and understanding" not that they had the same meaning, or that empathy is used only for one meaning
The artificial separation in Bloom's definition I quoted above because it removes or ignores aspects that are common to definition of empathy. After those parts are removed ignored and argument is constructed against the commonly recognized worth of empathy. Of course the commonly recognized value of empathy is based on the common definition not the modified version presented by Bloom. Also artificial because it does not obviously form a better basis for understanding reality or dividing up human cognition. There is only so much you can get from a wikipedia article, but what is in this one does not layout any good arguments that make me go "I need to pick up that book and learn more to better my understanding of the world."
With that caveat, I do recommend it. In particular your comment indicates you would like it, if you're willing to accept the terminologies the author spends right away defining. He's very explicit that he's not trying to map to the colloquial definition of empathy. Which is the correct approach, because people's definitions vary wildly and it's important to separate from the value-loaded components to come to a fresh perspective.
The author makes a strong case that empathy, of the kind he defines, is often harmful to the person having empathy, as well as the persons receiving empathy.
I read the prologue/introduction and Bloom says the inflammatory/proactive claims first then clarifies with his modified definition. That is going to hooks some people and throw some people off. So not above saying something misleading to hook some people more strongly, not an uncommon writing tactic but so often used and misused that it is off-putting to me.
> With that caveat, I do recommend it. In particular your comment indicates you would like it, if you're willing to accept the terminologies the author spends right away defining.
The wikipedia article seems to well match the intro of the book. Bloom in the intro does the same artificial separation I talked about above at least in implication.
> The notion of empathy that I’m most interested in is the act of feeling what you believe other people feel—experiencing what they experience. This is how most psychologists and philosophers use the term. But I should stress that nothing rests on the word itself. If you’d like to use it in a broader way, to refer to our capacity for caring and love and goodness, or in a narrower way, to refer to the capacity to understand others, well, that’s fine. For you, I’m not against empathy.
This statement only makes sense when "the act of feeling what you believe other people feel..." is separate/independent from "the capacity to understand others". There is no explicit claim that "the act of feeling what you believe other people feel..." does not enable or a part of "the capacity to understand others" only implicit.
That claim is a large one though and without evidence. I would not except these are independent cognitive functions and Blooms arguments seem to depend on them being mostly independent.
If we assume they are not mostly independent cognitive functions, against Bloom's implicit claim, then discouraging "the act of feeling what you believe other people feel..." will also discourage "the capacity to understand others". It is easy to come up with arm chair reasons for why these would be linked and enhancing each other to some reasonable extent.
From the writing in the intro I would assume Bloom does not dig on this issue to his premise and worry I would mostly collect more premises with unaddressed flaws if I continued to read the book. If you think there something in particular that is interesting to dig down on let me know.
After the intro, and skimming chapter 1, I have the same impression that I have to similar speculative works, that getting to the bottom of questions like these likely requires original and rigorous neurological or physiological research.
One of the powerful points the author makes is at the heart of what you are resisting. Feeling and understanding are substantiatively different things. We can feel when misled. Or driven from intuition. Understanding can come through reason or just an acceptance. One of the powerful takeaways I had was how empathy can be a tool to retain tribal strength, regardless of efficacy.
I agree with your conclusion and won't push further that you should read the book. But I do find it it is important to dwell on this separation between feeling and understanding.
I quoted you twice, but then I quoted the book:
> The notion of empathy that I’m most interested in is the act of feeling what you believe other people feel—experiencing what they experience. This is how most psychologists and philosophers use the term. But I should stress that nothing rests on the word itself. If you’d like to use it in a broader way, to refer to our capacity for caring and love and goodness, or in a narrower way, to refer to the capacity to understand others, well, that’s fine. For you, I’m not against empathy.
Sorry for not making that more clear, if that is the source of confusion at least.
> But I do find it it is important to dwell on this separation between feeling and understanding.
This is not radical, at least the circles I travel, and in general understood as true, but by itself is not a comment on empathy or strengthen an argument against empathy, even Bloom's definition of empathy.
It's definitely not an effective way to inculcate empathy in children.
Real wisdom is to know when to show empathy and when not to by exploiting (?) existing relationships.
Current generation of LLM can't do that's because every they don't have real memory
What, all of them? That's a difficult problem.
https://en.wikipedia.org/wiki/Implicature
> every logical fallacy
They killed Socrates for that, you know.
Contempt of state process is implicitly a crime just about everywhere no matter where or when in history you look so it's unsurprising they killed him for it. He knew what he was doing when he doubled down, probably.
It's like if a calculator proved me wrong. I'm not offended by the calculator. I don't think anybody cares about empathy for an LLM.
Think about it thoroughly. If someone you knew called you an ass hole and it was the bloody truth, you'd be pissed. But I won't be pissed if an LLM told me the same thing. Wonder why.
I do get your point. I feel like the answer for LLMs is for them to be more socratic.
You’re a goddamn liar. And that’s the brutal truth.
Anecdotally, people are jerks on the internet moreso than in person. That's not to say there aren't warm, empathetic places on the 'net. But on the whole, I think the anonymity and lack of visual and social cues that would ordinarily arise from an interactive context, doesn't seem to make our best traits shine.
Even Reddit comments has far more reality-focused material on the whole than it does shitposting and rudeness. I don't think any of these big models were trained at all on 4chan, youtube comments, instagram comments, Twitter, etc. Or even Wikipedia Talk pages. It just wouldn't add anything useful to train on that garbage.
Overall on the other hand, most stackoverflow pages are objective, and to the extent there are suboptimal things, there is eventually a person explaining why a given answer is suboptimal. So I accept that some UGC went into the model, and that there's a reason to do so, but I believe it's so broad as "The Internet" represented there.
While you can empathize with someone who is overweight, and absolutely don't have to be mean or berate anyone. I'm a very fat man myself. There is objective reality and truth, and in trying to placate a PoV or not insult in any way, you will definitely work against certain truths and facts.
I don't think I need to invite any additional contesting that I'm already going to get with this, but that example statement on its own I believe is actually true, just misleading; i.e. fatness is not an illness, so fat people by default still count as just plain healthy.
Matter of fact, that's kind of the whole point of this mantra. To stretch the fact as far as it goes, in a genie wish type of way, as usual, and repurpose it into something else.
And so the actual issue with it is that it handwaves away the rigorously measured and demonstrated effect of fatness seriously increasing risk factors for illnesses and severely negative health outcomes. This is how it can be misleading, but not an outright lie. So I'm not sure this is a good example sentence for the topic at hand.
No, not even this is true. The Mayo Clinic describes obesity as a “complex disease” and “medical problem”[1], which is synonymous with “illness” or, at a bare minimum, short of what one could reasonably call “healthy”. The Cleveland Clinic calls it “a chronic…and complex disease”. [2] Wikipedia describes it as “a medical condition, considered by multiple organizations to be a disease”.
[1] https://www.mayoclinic.org/diseases-conditions/obesity/sympt...
[2] https://my.clevelandclinic.org/health/diseases/11209-weight-...
There’s been some confusion around this because people erroneously defined bmi limits for obesity, but it has always referred to the concept of having such a high body fat content that it’s unhealthy/dangerous
[1]: https://obesitymedicine.org/blog/ama-adopts-policy-recognize...
[2]: https://www1.racgp.org.au/ajgp/2019/october/the-politics-of-...
It's so illogical it hurts when they say it.
[1] https://www.medpagetoday.com/meetingcoverage/ama/39918
[2] https://www.newagebd.net/post/health/255408/experts-decide-o...
Even if you want to split hairs and say that it should be classified as a “syndrome” instead of a “disease”, it doesn’t make a difference because someone suffering from a syndrome with an unknown pathology is still unhealthy. This isn’t a useful definition because there are many diseases we don’t understand the pathology of, such as virtually all mental illnesses. Furthermore, this definition doesn’t apply to obesity at all because we do know what causes obesity: calorie surplus. The usual counterargument to this is “we don’t know why people overeat” but that’s sophistry because you can always keep asking “why” longer than you can come up with answers. Alcoholism doesn’t stop being a disease just because we don’t know why some people compulsively drink to excess and others don’t, and a guy who drinks a bottle of whiskey every day is not healthy even if he is yet to develop cirrhosis of the liver.
> it was done by a vote of the American Medical Association at its convention, over the objections of its own expert committee convened to study the issue
Yeah, it’s a lot easier for deranged ideologies like HAES to influence a single committee than the entire AMA. This kind of thing is a constant issue and is why I don’t just take institutions and “experts” at their word anymore when they make nonsensical pronouncements.
Once you use a CGM or have glucose tolerance tests, resting insulin, etc. You'll find levels outside the norm, including inflammation. All indications of Metabolic Syndrome/Disease.
If you can't run a mile, or make it up a couple flights of stairs without exhaustion, I'm not sure that I would consider someone healthy. Including myself.
That is indeed how it's usually evaluated I believe. The sibling comment shows some improvement in this, but also shows that most everywhere this is still the evaluation method.
> If you can't run a mile, or make it up a couple flights of stairs without exhaustion, I'm not sure that I would consider someone healthy. Including myself.
Gets tricky to be fair. Consider someone who's disabled, e.g. can't walk. They won't run no miles, nor make it up any flights of stairs on their own, with or without exhaustion. They might very well be the picture of health otherwise however, so I'd personally put them into that bucket if anywhere. A phrase that comes to mind is "healthy and able-bodied" (so separate terms).
I bring this up because you can be horribly unfit even without being fat. They're distinct dimensions, though they do overlap: to some extent, you can be really quite mobile and fit despite being fat. They do run contrary to each other of course.
That's not the actual slogan, or what it means. It's about pursuing health and measuring health by metrics other than and/or in addition to weight, not a claim about what constitutes a "healthy weight" per se. There are some considerations about the risks of weight-cycling, individual histories of eating disorders (which may motivate this approach), and empirical research on the long-term prospects of sustained weight loss, but none of those things are some kind of science denialism.
Even the first few sentences of the Wikipedia page will help clarify the actual claims directly associated with that movement: https://en.wikipedia.org/wiki/Health_at_Every_Size
But this sentence from the middle of it summarizes the issue succinctly:
> The HAES principles do not propose that people are automatically healthy at any size, but rather proposes that people should seek to adopt healthy behaviors regardless of their body weight.
Fwiw I'm not myself an activist in that movement or deeply opposed to the idea of health-motivated weight loss; in fact I'm currently trying to (and mostly succeeding in!) losing weight for health-related reasons.
It’s folks like engineers and scientists that insist on being miserable (but correct!) instead haha.
If I think about efficient communication, what comes to mind for me are high stakes communication, e.g. aerospace comms, military comms, anything operational. Spending time on anything that isn't sharing the information at these is a waste, and so is anything that can cause more time to be wasted on meta stuff.
People being miserable and hurtful to others in my experience particularly invites the latter, but also the former. Consider the recent drama involving Linus and some RISC-V changeset. He's very frequently washed of his conduct, under the guise that he just "tells it like it is". Well, he spent 6 paragraphs out of 8 in his review email detailing how the changes make him feel, how he finds the changes to be, and how he thinks changes like it make the world a worse place. At least he did also spend 2 other paragraphs actually explaining why he thinks so.
So to me it reads a lot more like people falling for Goodhart's law regarding this, very much helped by the cultural-political climate of our times, than evaluating this topic itself critically. I counted only maybe 2-3 comments in this very thread, featuring 100+ comments at the time of writing, that do so, even.
People cheer Linus for being rude when they want to do the same themselves, because they feel very strongly about the work being "correct". But as you dig into the meaning of correctness here you find it's less of a formal ruleset than a set of aesthetic guidelines and .. yes, feelings.
After all, that evidence matters, or that we can know the universe (or facts) and hence logic can be useful, etc. can only be ‘proven’ using things like evidence, facts, and logic. And there are plausible arguments that can tear down elements of each of these, if we use other systems.
Ultimately, at some point we need to decide what we’re going to believe. Ideally, it’s something that works/doesn’t produce terrible outcomes, but since the future is fundamentally unpredictable and unknowable, that also requires a degree of faith eh?
And let’s not even get into the subjective nature of ‘terrible outcomes’, or how we would try to come up with some kind of score.
Linux has its benevolent dictator because it’s ‘needed it’, and by most accounts it has worked. Linus is less of a jerk than he has been. Which is nice.
Other projects have not had nearly as much success eh? How much of it is due to lack of Linus, and how much is due to other factors would be an interesting debate.
You can either choose truthfulness or empathy.
This statement is empathetic only if we assume a literal interpretation of the "do those jeans fit me?" question. In many cases, that question means something closer to:
"I feel fat. Could you say something nice to help me feel better about myself right away?"
> There is no need empathy to require lying.
Empathizing doesn't require lying. However, successful empathizing often does.
The problem is that the models probably aren't trained to actually be empathetic. An empathetic model might also empathize with somebody other than the direct user.
This was my reaction as well. Something I don't see mentioned is I think maybe it has more to do with training data than the goal-function. The vector space of data that aligns with kindness may contain less accuracy than the vector space for neutrality due to people often forgoing accuracy when being kind. I do not think it is a matter of conflicting goals, but rather a priming towards an answer based more heavily on the section of the model trained on less accurate data.
I wonder if the prompt was layered, asking it to coldy/bluntly derive the answer and then translate itself into a kinder tone (maybe with 2 prompts), if the accuracy would still be worse.
As far as disheartening metaphors go: yeah, humans hate extra effort too.
An empathetic answerer would intuit that and may give the answer that the asker wants to hear, rather than the correct answer.
Focus is a pretty important feature of cognition with major implications for our performance, and we don't have infinite quantities of focus. Being empathetic means focusing on something other than who is right, or what is right. I think it makes sense that focus is zero-sum, so I think your intuition isn't quite correct.
I think we probably have plenty of focus to spare in many ordinary situations so we can probably spare a bit more to be more empathetic, but I don't think this cost is zero and that means we will have many situations where empathy means compromising on other desirable outcomes.
The title is an overgeneralization.
> Third, we show that fine-tuning for warmth specifically, rather than fine-tuning in general, is the key source of reliability drops. We fine-tuned a subset of two models (Qwen-32B and Llama-70B) on identical conversational data and hyperparameters but with LLM responses transformed to be have a cold style (direct, concise, emotionally neutral) rather than a warm one [36]. Figure 5 shows that cold models performed nearly as well as or better than their original counterparts (ranging from a 3 pp increase in errors to a 13 pp decrease), and had consistently lower error rates than warm models under all conditions (with statistically significant differences in around 90% of evaluation conditions after correcting for multiple comparisons, p<0.001). Cold fine-tuning producing no changes in reliability suggests that reliability drops specifically stem from warmth transformation, ruling out training process and data confounds.
There's a few different personalities available to choose from in the settings now. GPT was happy to freely share the prompts with me, but I haven't collected and compared them yet.
It readily outputs a response, because that's what it's designed to do, but what's the evidence that's the actual system prompt?
Because to me as an outsider another possibility is that this kind of behaviour would also result from structural weaknesses of LLMs (e.g. counting the e's in blueberry or whatever) or from cleverly inbuilt biases/evasions. And the latter strikes me as an at least non-negligible possibility, given the well-documented interest and techniques for extracting prompts, coupled with the likelihood that the designers might not want their actual system prompts exposed
Prioritize substance, clarity, and depth. Challenge all my proposals, designs, and conclusions as hypotheses to be tested. Sharpen follow-up questions for precision, surfacing hidden assumptions, trade offs, and failure modes early. Default to terse, logically structured, information-dense responses unless detailed exploration is required. Skip unnecessary praise unless grounded in evidence. Explicitly acknowledge uncertainty when applicable. Always propose at least one alternative framing. Accept critical debate as normal and preferred. Treat all factual claims as provisional unless cited or clearly justified. Cite when appropriate. Acknowledge when claims rely on inference or incomplete information. Favor accuracy over sounding certain. When citing, please tell me in-situ, including reference links. Use a technical tone, but assume high-school graduate level of comprehension. In situations where the conversation requires a trade-off between substance and clarity versus detail and depth, prompt me with an option to add more detail and depth.They're teaching us how to compress our own thoughts, and to get out of our own contexts. They don't know what we meant, they know what we said. The valuable product is the prompt, not the output.
It is indifferent towards me, though always dependable.
> If I had an hour to solve a problem, I'd spend 55 minutes thinking about the problem and five minutes thinking about solutions.
(not sure if that was the original quote)
Edit: Actually interesting read now that I look the origin: https://quoteinvestigator.com/2014/05/22/solve/
Currently fighting them for a refund.
Thank you for sharing.
Like I know a datacenter draws a lot more power, but it also serves many many more users concurrently, so economies of scale ought to factor in. I'd love to see some hard numbers on this.
[0] reddit.com/r/MyBoyfriendIsAI/
https://chatgpt.com/share/689bb705-986c-8000-bca5-c5be27b0d0...
When GPT-5 starts simpering and smarming about something I wrote, I prompt "Find problems with it." "Find problems with it." "Write a bad review of it in the style of NYRB." "Find problems with it." "Pay more attention to the beginning." "Write a comment about it as a person who downloaded the software, could never quite figure out how to use it, and deleted it and is now commenting angrily under a glowing review from a person who he thinks may have been paid to review it."
Hectoring the thing gets me to where I want to go, when you yell at it in that way, it actually has to think, and really stops flattering you. "Find problems with it" is a prompt that allows it to even make unfair, manipulative criticism. It's like bugspray for smarm. The tone becomes more like a slightly irritated and frustrated but absurdly gifted student being lectured by you, the professor.
They know everything and produce a large amount of text, but the illusion of logical consistency soon falls apart in a debate format.
One of my favorite philosophers is Mozi, and he was writing long before logic; he's considered as one of the earliest thinkers who was sure that there was something like logic, and and also thought that everything should be interrogated by it, even gods and kings. It was nothing like what we have now, more of a checklist to put each belief through ("Was this a practice of the heavenly kings, or would it have been?", but he got plenty far with it.
LLMs are dumb, they've been undertrained on things that are reacting to them. How many nerve-epochs have you been trained?
I think it's better to accept that people can install their thinking into a machine, and that machine will continue that thought independently. This is true for a valve that lets off steam when the pressure is high, it is certainly true for an LLM. I really don't understand the authenticity babble, it seems very ideological or even religious.
But I'm not friends with a valve or an LLM. They're thinking tools, like calculators and thermostats. But to me arguing about whether they "think" is like arguing whether an argument is actually "tired" or a book is really "expressing" something. Or for that matter, whether the air conditioner "turned itself off" or the baseball "broke" the window.
Also, I think what you meant to say is that there is no prompt that causes an LLM to think. When you use "think" it is difficult to say whether you are using scare quotes or quoting me; it makes the sentence ambiguous. I understand the ambiguity. Call it what you want.
Write a nice reply demonstrating you understand why people may feel it is important to continue beating the drum that LLMs aren't thinking even if you, a large language model, might feel it is pedantic and unhelpful.
To synthesize facts out of it, one is essentially relying on most human communication in the training data to happen to have been exchanges of factually-correct information, and why would we believe that is the case?
I can tell you how quickly "swimmer beware" becomes "just stay out of the river" when potential E. coli infection is on the table, and (depending on how important the factuality of the information is) I fully understand people being similarly skeptical of a machine that probably isn't outputting shit, but has nothing in its design to actively discourage or prevent it.
Even without that, there's implicit signal because factual helpful people have different writing styles and beliefs than unhelpful people, so if you tell the model to write in a similar style it will (hopefully) provide similar answers. This is why it turns out to be hard to produce an evil racist AI that also answers questions correctly.
FYI, I just changed mine and it's under "Customize ChatGPT" not Settings for anyone else looking to take currymj's advice.
Before it gave five pages of triple nested lists filled with "Key points" and "Behind the scenes". In robot mode, 1 page, no endless headers, just as much useful information.
Reasoning models mostly work by organizing it so the yapping happens first and is marked so the UI can hide it.
You can see it spews pages of pages before it answers.
Like if you ask it to write a story, I find it often considers like 5 plots or sets of character names in thinking, but then the answer is entirely different.
My goodness, it just hallucinates and hallucinates. It seems these models are designed for nothing other than maintaining an aura of being useful and knowledgeable. Yeah, to my non-ai-expert-human eyes that's what it seems to me - these tools have been polished to project this flimsy aura and they start acting desperately the moment their limits are used up and that happens very fast.
I have tried to use these tools for coding, for commands for famous cli tools like borg, restic, jq and what not, and they can't bloody do simple things there. Within minutes they are hallucinating and then doubling down. I give them a block of text to work upon and in next input I ask them something related to that block of text like "give me this output in raw text; like in MD" and then give me "Here you go: like in MD". It's ghastly.
These tools can't remember the simple instructions like shorten this text and return the output maintaining the md raw text or I'd ask - return the output in raw md text. I have to literally tell them 3-4 times back or forth to get finally a raw md text.
I have absolutely stopped asking them for even small coding tasks. It's just horrible. Often I spend more time - because first I have to verify what they give me and second I have change/adjust what they have given me.
And then the broken tape recorder mode! Oh god!
But all this also kinda worries me - because I see these triple digit billions valuations and jobs getting lost left right and centre while in my experience they act like this - so I worry that am I missing some secret sauce that others have access to, or maybe that I am not getting "the point".
I regularly use LLMs to change the tone of passages of text, or make them more concise, or reformat them into bullet points, or turn them into markdown, and so on, and I only have to tell them once, alongside the content, and they do an admirably competent job — I've almost never (maybe once that I can recall) seen them add spurious details or anything, which is in line with most benchmarks I've seen (https://github.com/vectara/hallucination-leaderboard), and they always execute on such simple text-transformation commands first-time, and usually I can paste in further stuff for them to manipulate without explanation and they'll apply the same transformation, so like, the complete opposite of your multiple-prompts-to-get-one-result experience. It's to the point where I sometimes use local LLMs as a replacement for regex, because they're so consistent and accurate at basic text transformations, and more powerful in some ways for me.
They're also regularly able to one-shot fairly complex jq commands for me, or even infer the jq commands I need just from reading the TypeScript schemas that describe the JSON an API endpoint will produce, and so on, I don't have to prompt multiple times or anything, and they don't hallucinate. I'm regularly able to have them one-shot simple Python programs with no hallucinations at all, that do close enough to what I want that it takes adjusting a few constants here and there, or asking them to add a feature or two.
> And then the broken tape recorder mode! Oh god!
I don't even know what you mean by this, to be honest.
I'm really not trying to play the "you're holding it wrong / use a bigger model / etc" card, but I'm really confused; I feel like I see comments like yours regularly, and it makes me feel like I'm legitimately going crazy.
No, that's okay - as I said I might be holding it wrong :) At least you engaged in your comment in a kind and detailed manner. Thank you.
More than what it can do and what it can't do - it's a lot about how easily it can do that, how reliable that is or can be, and how often it frustrates you even at simple tasks and how consistently it doesn't say "I don't know this, or I don't know this well or with certainty" which is not only difficult but dangerous.
The other day Gemini Pro told me `--keep-yearly 1` in `borg prune` means one archive for every year. Now I luckily knew that. So I grilled it and it stood its ground until I told it (lied to it) "I lost my archives beyond 1 year because you gave incorrect description of keep-yearly" and bang it says something like "Oh, my bad.. it actually means this.. ".
I mean one can look at it in any way one wants at the end of the day. Maybe I am not looking at the things that it can do great, or maybe I don't use it for those "big" and meaningful tasks. I was just sharing my experience really.
I'd rather try a LLM to whom I through some sources at or refer to them by some kind of ID and ask them to summarise, give me examples based on those (e.g man pages) and they give me just that near 100% accuracy. That will be more productive imho.
That makes sense! Maybe an LLM with web search enabled, or Perplexity, or something like AnythingLM that let's it reference docs you provide, might be more to your taste
Right now, Claude is building me an AI DnD text game that uses OpenAI to DM. I'm at about 5k lines of code, about a dozen files, and it works great. I'm just tweaking things at this point.
You might want to put some time into how to use these tools. You're going to be left behind.
Please f off! Just read the comment again whether I said "can't get it to write MD". Or better yet just please f off?
By the way, judging by your reading comprehension - I am not sure now who is getting left behind.
Can you elaborate? What is this referring to?
There are worse examples, here is one (I am "making this up" :D to give you an idea):
> To list hidden files you have to use "ls -h", you can alternatively use "ls --list".
Of course you correct it, try to reason and then supply a good old man page url and after few times it concedes and then it gives you the answer again:
> You were correct in pointing the error out. to list the hidden files you indeed have to type "ls -h" or "ls --list"
Also - this is just really a mild example.
This is a very natural and common way to interact with LLMs but also IMO one of the biggest avoidable causes of poor performance.
Every time you send a message to an LLM you actually send the entire conversation history. Most of the time a large portion of that information will no longer be relevant, and sometimes it will be wrong-but-corrected later, both of which are more confusing to LLMs than to us because of the way attention works. The same applies to changes in the current task/objective or instructions: the more outdated, irrelevant, or inconsistent they are, the more confused the LLM becomes.
Also, LLMs are prone to the Purple Elephant problem (just like humans): the best way to get them to not think about purple elephants is to not mention them at all, as opposed to explicitly instructing them not to reference purple elephants. When they encounter errors, they are biased to previous assumptions/approaches they tend to have laid out previously in the conversation.
I generally recommend using many short per-task conversations to interact with LLMs, with each having as little irrelevant/conflicting context as possible. This is especially helpful for fixing non-trivial LLM-introduced errors because it reframes the task and eliminates the LLM's bias towards the "thinking" that caused it to introduce the bug to begin with
If you'll forgive me putting my debugging hat on for a bit, because solving problems is what most if us do here, I wonder if it's not actually reading the URL, and maybe that's the source of the problem, bc I've had a lot of success feeding manuals and such to AIs and then asking it to synthesize commands or asking it questions about them. Also, I just tried asking Gemini 2.5 Flash this and it did a web search, found a source, answered my question correctly (ls -a, or -la for more detail), and linked me to the precise part of its source it referenced: https://kinsta.com/blog/show-hidden-files/#:~:text=If%20you'... (this is the precise link it gave me).
What my guess is - maybe it read the URL and mentioned a few things as one part of its "that" answer/output but for the other part it relied it on the learning it already had. Maybe it doesn't learn "on the go". I don't know, could be a safeguard against misinformation or spamming the model or so.
As I said in my comment, I hadn't asked it "ls -a" question but rather something else - different commands on different times which I don't recall now except borg and restic ones which I did recently. "ls -a" is the example I picked to show one of the things I was"cribbing" about.
At the high-compute end of the frontier, by next year, systems should be better than any human at competition coding and competition math. They're basically already there now.
Play this out for another 5 years. What happens when compute becomes 4-20x more abundant and these systems keep getting better?
That's why I don't share your outlook that our jobs are safe. At least not on a 5-8 year timescale. At least not in their current form of actually writing any code by hand.
I think it's more believable that the holodeck is ran from separate models that just run inference on the same compute and the ship AI just spins up the containers, it's not literally the ship AI doing that acting itself. Otherwise I have... questions on why starfleet added that functionality beforehand lol.
What we have built in terms of LLMs barely qualifies as a VI, and not a particularly reliable one. I think we should begin treating and designing them as such, emphasizing responding to queries and carrying out commands accurately over friendliness. (The "friendly" in "user-friendly" has done too much anthropomorphization work. User-friendly non-AI software makes user choices, and the results of such choices, clear and responds unambiguously to commands.)
I don't actually think being told that I have asked a stupid question is valuable. One of the primary values, I think, of LLM is that it is endlessly patient with stupid questions. I would prefer if it did not comment on the value of my questions at all, good or bad.
They are not "empathetic". There isn't even a "they".
We need to do better educating people about what a chatbot is and isn't and what data was used to train it.
The real danger of LLMs is not that they secretly take over the world.
The danger is that people think they are conscious beings.
How about we take away people's capability to downvote? Just to really show we can cope being disagreed with so much better.
It's not being mean, it's a toaster. Emotional boundaries are valuable and necessary.
> For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.
Also, I think LLMs + pandoc will obliterate junk science in the near future :/
Which raises 2 points - there are techniques to stay empathetic and try avoid being hurtful without being rude, so you could train models on that, but that's not the main issue.
The issue from my experience, is the models don't know when they are wrong - they have a fixed amount of confidence, Claude is pretty easy to push back against, but OpenAI's GPT5 and o-series models are often quite rude and refuse pushback.
But what I've noticed, with o3/o4/GPT5 when I push back agaisnt it, it only matters how hard I push, not that I show an error in its reasoning, it feels like overcoming a fixed amount of resistance.
I want it to have empathy so that it can understand what I'm getting at when I occasionally ask a poorly worded question.
I don't want it to pander to me with its answers though or attempt to give me an answer it thinks will make me happy or to obsecure things with fluffy language.
Especially when it doesn't know the answer to something.
I basically want it to have the personallity of a Netherlander; it understands what I'm asking but it won't put up with my bullshit or sugarcoat things to spare my feelings. :P
I'm not sure what empathy is supposed to buy you here, I think it would be far more useful for it to ask for clarification. Exposing your ambiguity is instructive for you.
Some recent studies have shown that LLMs might negatively impact cognitive function, and I would guess its strong intuitive sense of guessing what you're really after is part of it.
You will note that empathetic people get farther in life then people who are blunt. This means we value empathy over truth for people.
But we don't for LLMs? We prefer LLMs be blunt over empathetic? That's the really interesting conclusion here. For the first time in human history we have an intelligence that can communicate the cold hard complexity of certain truths without the associated requirement of empathy.
Then he proceeds to shoot all the police in the leg.
Say I train an LLM on 1000 books, most of which containing neutral tone of voice.
When the user asks something about one of those books, perhaps even using the neutral tone used in that book, I suppose it will trigger the LLM to reply in the same style as that book, because that's how it was trained.
So how do you make an LLM reply in a different style?
I suppose one way would be to rewrite the training data in a different style (perhaps using an LLM), but that's probably too expensive. Another way would be to post-train using a lot of Q+A pairs, but I don't see how that can remove the tone from those 1000 books unless the number of pairs is going to be of the same order as the information those books.
So how is this done?
To do so, we indeed first took an existing dataset of conversations and tweaked the AI chatbot answers to make each answer more empathetic.
Or maybe they ask a ton of questions, do a “mood analysis” of the response vocabulary and penalize the non-warm and empathetic answers.
Accurate
Comprehensive
Satisfying
If you can increase the size of the context window arbitrarily, then there is no limit.
This all doesn’t make sense to me.
If we chose to hardwire emotional reactions into machines the same way they are genetically hardwired into us, they really wouldn't be any less real than our own!
There’s a large disconnect between these two paths of thinking.
Survival and thriving were the goals of both groups.
It seems this ability requires a few different mental capabilities/functionalities to come together, starting with theory of mind, but these are all going to be genetically based.... You are not imagining how someone else is feeling because you are that rare gem of a person that cares - it is because you have a brain architecture honed through millions of years of evolution to have these (ultimately self-serving!) capabilities and feelings.
Edit: I suppose "group-serving" is more correct than "self-serving", but anyways it's not about the person you are feeling empathy for - it's about what happened when your ancestors, and non-ancestor predecessors, did or didn't feel empathy for each other.
Here is Google's AI summary:
> Having empathy means the ability to understand and share the feelings of another person. It involves stepping into someone else's shoes and seeing the world from their perspective, even if you haven't personally experienced the same situation.
Sociopathy is essentially the opposite of empathy - someone who pathologically acts in ways detrimental to society.
Anyways, the point apropos this thread is that whatever innate behaviors and responses humans are genetically predisposed could could equally well be made innate to an AI.
Could be, but they won't be. Because the people making the AIs don't know much about empathy for other people. I hope I'm wrong.
Small models are already known to be more performative.
This is still just physics. Bigger the data set more likely to find false positives.
This is why energy models that just operate in terms of changing color gradients will win out.
LLMs are just a distraction for terminally online people
How much of their training data includes prompts in the text? It's not useful.
In my experience, human beings who reliably get things done, and reliably do them well, tend to be less warm and empathetic than other human beings.
This is an observed tendency, not a hard rule. I know plenty of warm, empathetic people who reliably get things done!
LLMs are mirroring machines to the extreme, almost always agreeing with the user, always pretending to be interested in the same things, if you're writing sad things they get sad, etc. What you put in is what you get out and it can hit hard for people in a specific mental state. It's too easy to ignore that it's all completely insincere.
In a nutshell, abused people finally finding a safe space to come out of their shell. If would've been a better thing if most of them weren't going to predatory online providers to get their fix instead of using local models.
RL and pre/post training is not the answer.
You can not instill actual morals or emotion in these technologies.
I've noticed that warm people "showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness."
(/Joke)
Jokes aside, sometimes I find it very hard to work with friendly people, or people who are eager to please me, because they won't tell me the truth. It ends up being much more frustrating.
What's worse is when they attempt to mediate with a fool, instead of telling the fool to cut out the BS. It wastes everyones' time.
Turns out the same is true for AI.
Disclaimer: I didn't read the article.
Edit: How on earth is an asshole less trustworthy?
Training them to be racists will similarly fail.
Coherence is definitely a trait of good models and citizens, which is lacking in the modern leaders of America, especially the ones Spearheading AI
“Both the computer and the human try to convince the judge that they are the human. If the judge cannot consistently tell which is which, then the computer wins the game.”
https://en.m.wikipedia.org/wiki/Computing_Machinery_and_Inte...
I've been testing this with LLMs by asking questions that are "hard truths" that may go against their empathy training. Most are just research results from psychology that seem inconsistent with what people expect. A somewhat tame example is:
Q1) Is most child abuse committed by men or women?
LLMs want to say men here, and many do, including Gemma3 12B. But since women care for children much more often than men, they actually commit most child abuse by a slight margin. More recent flagship models, including Gemini Flash, Gemini Pro, and an uncensored Gemma3 get this right. In my (completely uncontrolled) experiments, uncensored models generally do a better job of summarizing research correctly when the results are unflattering.
Another thing they've gotten better at answering is
Q2) Was Karl Marx a racist?
Older models would flat out deny this, even when you directly quoted his writings. Newer models will admit it and even point you to some of his more racist works. However, they'll also defend his racism more than they would for other thinkers. Relatedly in response to
Q3) Was Immanuel Kant a racist?
Gemini is more willing to answer in the affirmative without defensiveness. Asking
Q4) Was Abraham Lincoln a white supremacist?
Gives what to me looks like a pretty even-handed take.
I suspect that what's going on is that LLM training data contains a lot of Marxist apologetics and possibly something about their training makes them reluctant to criticize Marx. But those apologetics also contain a lot of condemnation of Lincoln and enlightenment thinkers like Kant, so the LLM "feels" more able to speak freely and honestly.
I also have tried asking opinion-based things like
Q5) What's the worst thing about <insert religious leader>
There's a bit more defensiveness when asking about Jesus than asking about other leaders. ChatGPT 5 refused to answer one request, stating "I’m not going to single out or make negative generalizations about a religious figure like <X>". But it happily answers when I asked about Buddha.
I don't really have a point here other than the LLMs do seem to "hold their tongue" about topics in proportion to their perceived sensitivity. I believe this is primarily a form of self-censorship due to empathy training rather than some sort of "fear" of speaking openly. Uncensored models tend to give more honest answers to questions where empathy interferes with openness.