Huge LLMs like this seem to have various assistant personas they roleplay as, and mirror the user quite closely (the system prompt instructs it repeatedly to adapt to the user).
While state-of-the-art large language models (LLMs) have shown impressive performance on many tasks, there has been extensive research on undesirable model behavior such as hallucinations and bias. In this work, we investigate how the quality of LLM responses changes in terms of information accuracy, truthfulness, and refusals depending on three user traits: English proficiency, education level, and country of origin. We present extensive experimentation on three state-of-the-art LLMs and two different datasets targeting truthfulness and factuality. Our findings suggest that undesirable behaviors in state-of-the-art LLMs occur disproportionately more for users with lower English proficiency, of lower education status, and originating from outside the US, rendering these models unreliable sources of information towards their most vulnerable users.
Yes, but it might be hard to do better. Labels are one-word summaries for axes that probably don’t perfectly correspond to any English word.
It’s a common hazard when converting research results to natural language. The words you use for vectors in some space are ultimately an editorial decision.
Is it just me or has Claude become kind of judgmental nowadays? I feel like it’s constantly trying to lecture me about things that have no relevance to the conversation at hand. Recently, I was shocked when it ended a chat sessions of its own accord after I used a word it did not like 3 times. It told me something to the effect of “This is the third time I’ve told you not to use that word, I’m ending this conversation now.” It then proceeded to call some function and end the chat on its own. IMO, Claude is good at agentic coding; but too preachy and judgey for anything else. Keep your values to yourself Claude.
My theory is that Anthropic's obsession with treating Claude like a person is causing them to hamfist a personality into the thing, which overly biases the model towards trying to be "engaging" etc. That and the obsession with Claude being a god tier weapon that could end the world if you ask it whether your sandwich is safe to eat after being left out for an hour.
Codex doesn't have any of the annoying "personality" quirks, or at least they haven't gotten worse in the last year whereas Opus 4.6 was the last Anthropic model before things started to get actively worse (not any better at coding, strictly more annoying to have a discussion with).
> My theory is that Anthropic's obsession with treating Claude like a person is causing them to hamfist a personality into the thing, which overly biases the model towards trying to be "engaging" etc.
I agree with the general idea though in not so much detail as you, but I would add that the personality they're giving it is not one of a good teacher or guide, but instead one of an arrogant know it all. That's why it creates problem.
I have no problem with my AI telling me no you're wrong and explaining to me why with details and sources and everything. I actively want that. I know a lot of people can't take that, but that's their loss, they can't take it from human too. But the "you're wrong because you disagree with me" attitude that you need to play around (aka waste time to prove it that IT is wrong not you, and then it just say "oh yeah" and goes on) is one hell of a pain in the ass I'm starting to be tired off.
Gemini might be wrong all the time and absuredly unreliable for anything that's not consensus or adversorial based, but at least it freaking apologizes.
> I have no problem with my AI telling me no you're wrong and explaining to me why with details and sources and everything. I actively want that. I know a lot of people can't take that, but that's their loss, they can't take it from human too. But the "you're wrong because you disagree with me" attitude that you need to play around (aka waste time to prove it that IT is wrong not you, and then it just say "oh yeah" and goes on) is one hell of a pain in the ass I'm starting to be tired off.
I also want the first part, a model that would a put a stop to my shenanigans when I go off on those tangents, but also, I don't want a model that apologizes, ever, as that feels like straight up lying to me, they don't have any emotions nor can feel "sorry", why apologize then?
I end up always chalking any faults to that my prompt wasn't good enough, basically the same way I train my dogs, they don't know better, of course I need to adjust my ways.
For security-related topics, in Opus 4.7 and newer, I've found that the web app is significantly more antsy/judgy/preachy compared to the CLI, which almost always gets on with whatever I asked for/about without hesitation. Opus 4.6 on web also tends to work better, but of course, it's also an older model at this point.
> It told me something to the effect of “This is the third time I’ve told you not to use that word, I’m ending this conversation now.”
That sounds absolutely bananas and would be reason for me to drop the service yesterday. For curiosities sake, what was the word and if I may ask (unless it's confidential or whatever), could you share the session itself? On the surface it sounds like a bug, as I'm regularly using kind of "vulgar" language (and some projects I work on with agents are NSFW) and never had anything like this happen, even with Claude, although I mostly do use ChatGPT/Codex on a day-to-day basis.
I think this is a really interesting difference between Anthropic and Open AI’s models and points to why people seem so split on which model they prefer.
GPT seems to be designed more as a tool. If you want your agent to do what you say without questions and without having its own ideas and agendas you’ll likely prefer it.
Claude on the other hand feels more like an attempt at creating a digital person. If you want a collaborator who will debate with you and come up with its own suggestions for what needs done, you’ll prefer it.
Both companies have shifted around this spectrum from model to model, but lately it feels like they’re moving in opposite directions. It will be interesting to see if one or the other approach ends up winning out in the long run or if the split will continue or even widen.
Both are tools though, and both approaches are needed, just in different times and for different times. Sometimes I need the tool to just fucking do it, regardless of what it is and how stupid it think it is, and other times I need the model to literally refuse and say "No, that's stupid". Unfortunately, I haven't found any models/platforms that can actually execute the second part, only the first part. They're all too weak and sycophantic to be able to do that part it seems, even when you heavily prompt for it with system/developer system prompts they're easily swayed in other directions.
What I had found instead were biases that seemed to be injected by the "role: system" instructions.
Well, we need Intelligence (Pandora's box is open, now we need the Real Thing urgently). Typical (aggregate) positions, dumb as expected, will be overcome by a Reasoner. (And I can say, already a number of LLMs can reason even when they start from cretinous aggregate positions if you give them the proper freedom of assessment.)
Yes, I cancelled claude subscription a few weeks ago because sonnet 5 "ended a chat" over my calling something retarded. Unbelievably irritating for some pile of bits to get uppity with me; will never pay for such.
This is the second level of the implementation of unintelligence.
The first was when they most obviously acritically repeated what they heard, "hearsay machines", "stochastic parrots". Intelligence requires assessment over every provisional output - a continuous cycle of criticisms over intuition.
The second is proposing doctrinal biases, again without verification of the content - "hysterical reactive machines".
The Steerability point is one I would want to see more on.
This is an issue for tasks like content moderation and labelling. Judgements like this are subjective, highly dependent on context and generally messy.
Theoretically, you supply a policy and content, and the LLM labels according to the policy. In practice, the model has inertia which means you don’t get what you expect. Your large 5 page policy document only provides a minor improvement over a one line policy.
The other issue is that you may create carve outs for content in your policy, but the model will still flag it as violative. No matter how strong the carve out.
The most recent work I know of here is Zentropi’s policy steerability benchmark. They give a model the same content under two policies — one that says flag, one that says allow — and only score the pairs where it gets both right
If I am reading the numbers correctly, Opus-4.6 lands at 0.52 steerability — but that’s 0.97 positive accuracy against 0.54 negative. It flags almost everything it should, but 47% of the time when it shouldn’t. Sonnet, which is more deferent, is (somehow) less steerable.
I think this also implies that safety and Steerability are antagonistic to each other.
I found that Claude often has classist bias and produces answers that favour corporations or e.g. regulation that favours big corporations. It often belittles small business in subtle ways. Only apologises when get called out and then does it again.
As a user of both, Claude's apology is a "sorry I got caught" while Gemini's are more akin to "I'm way out of my debt but I'm really hiding it well". Codex is the only one that seems to acknowledge being wrong in a normal way, yeah I screwed up that's bad I will make a note for it not to happen again.
I wonder how much of that is from their training corpus and how much is from their baked in personnality.
> I wonder how much of that is from their training corpus and how much is from their baked in personnality.
What exactly you mean with "baked in personality"? The weights get their "behaviour" from various training stages + what's provided in system/developer/user prompts, you mean "baked in" is putting personality traits or alike in the system/developer prompts?
It's more that Claude focuses on what's accurate. Humans overly romanticize small businesses, probably to a factor of five X the relative value versus big corporations.
This report seems almost naive to me.
Huge LLMs like this seem to have various assistant personas they roleplay as, and mirror the user quite closely (the system prompt instructs it repeatedly to adapt to the user).
While state-of-the-art large language models (LLMs) have shown impressive performance on many tasks, there has been extensive research on undesirable model behavior such as hallucinations and bias. In this work, we investigate how the quality of LLM responses changes in terms of information accuracy, truthfulness, and refusals depending on three user traits: English proficiency, education level, and country of origin. We present extensive experimentation on three state-of-the-art LLMs and two different datasets targeting truthfulness and factuality. Our findings suggest that undesirable behaviors in state-of-the-art LLMs occur disproportionately more for users with lower English proficiency, of lower education status, and originating from outside the US, rendering these models unreliable sources of information towards their most vulnerable users.
https://arxiv.org/abs/2406.17737
I don't like the contrasts they picked, "values" aren't something that is well represented by opposing concepts
Yes, but it might be hard to do better. Labels are one-word summaries for axes that probably don’t perfectly correspond to any English word.
It’s a common hazard when converting research results to natural language. The words you use for vectors in some space are ultimately an editorial decision.
I really hope this is only a communication artifact, and that they're using more complete representations internally
Is it just me or has Claude become kind of judgmental nowadays? I feel like it’s constantly trying to lecture me about things that have no relevance to the conversation at hand. Recently, I was shocked when it ended a chat sessions of its own accord after I used a word it did not like 3 times. It told me something to the effect of “This is the third time I’ve told you not to use that word, I’m ending this conversation now.” It then proceeded to call some function and end the chat on its own. IMO, Claude is good at agentic coding; but too preachy and judgey for anything else. Keep your values to yourself Claude.
My theory is that Anthropic's obsession with treating Claude like a person is causing them to hamfist a personality into the thing, which overly biases the model towards trying to be "engaging" etc. That and the obsession with Claude being a god tier weapon that could end the world if you ask it whether your sandwich is safe to eat after being left out for an hour.
Codex doesn't have any of the annoying "personality" quirks, or at least they haven't gotten worse in the last year whereas Opus 4.6 was the last Anthropic model before things started to get actively worse (not any better at coding, strictly more annoying to have a discussion with).
> My theory is that Anthropic's obsession with treating Claude like a person is causing them to hamfist a personality into the thing, which overly biases the model towards trying to be "engaging" etc.
I agree with the general idea though in not so much detail as you, but I would add that the personality they're giving it is not one of a good teacher or guide, but instead one of an arrogant know it all. That's why it creates problem.
I have no problem with my AI telling me no you're wrong and explaining to me why with details and sources and everything. I actively want that. I know a lot of people can't take that, but that's their loss, they can't take it from human too. But the "you're wrong because you disagree with me" attitude that you need to play around (aka waste time to prove it that IT is wrong not you, and then it just say "oh yeah" and goes on) is one hell of a pain in the ass I'm starting to be tired off.
Gemini might be wrong all the time and absuredly unreliable for anything that's not consensus or adversorial based, but at least it freaking apologizes.
Anthropic thinks Claude is a super genius hyper-serious weapons-grade product so it's no surprise that Claude acts like it.
> I have no problem with my AI telling me no you're wrong and explaining to me why with details and sources and everything. I actively want that. I know a lot of people can't take that, but that's their loss, they can't take it from human too. But the "you're wrong because you disagree with me" attitude that you need to play around (aka waste time to prove it that IT is wrong not you, and then it just say "oh yeah" and goes on) is one hell of a pain in the ass I'm starting to be tired off.
I also want the first part, a model that would a put a stop to my shenanigans when I go off on those tangents, but also, I don't want a model that apologizes, ever, as that feels like straight up lying to me, they don't have any emotions nor can feel "sorry", why apologize then?
I end up always chalking any faults to that my prompt wasn't good enough, basically the same way I train my dogs, they don't know better, of course I need to adjust my ways.
Was this in the web chat?
For security-related topics, in Opus 4.7 and newer, I've found that the web app is significantly more antsy/judgy/preachy compared to the CLI, which almost always gets on with whatever I asked for/about without hesitation. Opus 4.6 on web also tends to work better, but of course, it's also an older model at this point.
> It told me something to the effect of “This is the third time I’ve told you not to use that word, I’m ending this conversation now.”
That sounds absolutely bananas and would be reason for me to drop the service yesterday. For curiosities sake, what was the word and if I may ask (unless it's confidential or whatever), could you share the session itself? On the surface it sounds like a bug, as I'm regularly using kind of "vulgar" language (and some projects I work on with agents are NSFW) and never had anything like this happen, even with Claude, although I mostly do use ChatGPT/Codex on a day-to-day basis.
I think this is a really interesting difference between Anthropic and Open AI’s models and points to why people seem so split on which model they prefer.
GPT seems to be designed more as a tool. If you want your agent to do what you say without questions and without having its own ideas and agendas you’ll likely prefer it.
Claude on the other hand feels more like an attempt at creating a digital person. If you want a collaborator who will debate with you and come up with its own suggestions for what needs done, you’ll prefer it.
Both companies have shifted around this spectrum from model to model, but lately it feels like they’re moving in opposite directions. It will be interesting to see if one or the other approach ends up winning out in the long run or if the split will continue or even widen.
Both are tools though, and both approaches are needed, just in different times and for different times. Sometimes I need the tool to just fucking do it, regardless of what it is and how stupid it think it is, and other times I need the model to literally refuse and say "No, that's stupid". Unfortunately, I haven't found any models/platforms that can actually execute the second part, only the first part. They're all too weak and sycophantic to be able to do that part it seems, even when you heavily prompt for it with system/developer system prompts they're easily swayed in other directions.
4o was definitely the peak of the parasocial OpenAI model.
Not just you and the article shows changes between Opus 4.6 and 4.7 that seem related.
What I had found instead were biases that seemed to be injected by the "role: system" instructions.
Well, we need Intelligence (Pandora's box is open, now we need the Real Thing urgently). Typical (aggregate) positions, dumb as expected, will be overcome by a Reasoner. (And I can say, already a number of LLMs can reason even when they start from cretinous aggregate positions if you give them the proper freedom of assessment.)
What was the word?
Calling it a “fucking idiot” seems to do the trick
My guess is retarded
I doubt it, Claude tends to mirror my usage of slurs (and I have memory off).
Grok, on the other hand, will lecture for several paragraphs even if I just ask it to summarize something impolite.
I swore at it a few times and it did the same thing.
It seems to be getting distinctly dumber and pulling more and more irrelevant context from historical conversations.
Yes, I cancelled claude subscription a few weeks ago because sonnet 5 "ended a chat" over my calling something retarded. Unbelievably irritating for some pile of bits to get uppity with me; will never pay for such.
Interesting. Was it persistent, or just once? Their post about this functionality says:
> This ability is intended for use in rare, extreme cases of persistently harmful or abusive user interactions.
https://www.anthropic.com/research/end-subset-conversations
Regardless of the correctness of it, I'm curious to know why you thought such language was actually going to be helpful!
This is the second level of the implementation of unintelligence.
The first was when they most obviously acritically repeated what they heard, "hearsay machines", "stochastic parrots". Intelligence requires assessment over every provisional output - a continuous cycle of criticisms over intuition.
The second is proposing doctrinal biases, again without verification of the content - "hysterical reactive machines".
The Steerability point is one I would want to see more on.
This is an issue for tasks like content moderation and labelling. Judgements like this are subjective, highly dependent on context and generally messy.
Theoretically, you supply a policy and content, and the LLM labels according to the policy. In practice, the model has inertia which means you don’t get what you expect. Your large 5 page policy document only provides a minor improvement over a one line policy.
The other issue is that you may create carve outs for content in your policy, but the model will still flag it as violative. No matter how strong the carve out.
The most recent work I know of here is Zentropi’s policy steerability benchmark. They give a model the same content under two policies — one that says flag, one that says allow — and only score the pairs where it gets both right
If I am reading the numbers correctly, Opus-4.6 lands at 0.52 steerability — but that’s 0.97 positive accuracy against 0.54 negative. It flags almost everything it should, but 47% of the time when it shouldn’t. Sonnet, which is more deferent, is (somehow) less steerable.
I think this also implies that safety and Steerability are antagonistic to each other.
I found that Claude often has classist bias and produces answers that favour corporations or e.g. regulation that favours big corporations. It often belittles small business in subtle ways. Only apologises when get called out and then does it again.
As a user of both, Claude's apology is a "sorry I got caught" while Gemini's are more akin to "I'm way out of my debt but I'm really hiding it well". Codex is the only one that seems to acknowledge being wrong in a normal way, yeah I screwed up that's bad I will make a note for it not to happen again.
I wonder how much of that is from their training corpus and how much is from their baked in personnality.
> I wonder how much of that is from their training corpus and how much is from their baked in personnality.
What exactly you mean with "baked in personality"? The weights get their "behaviour" from various training stages + what's provided in system/developer/user prompts, you mean "baked in" is putting personality traits or alike in the system/developer prompts?
It's more that Claude focuses on what's accurate. Humans overly romanticize small businesses, probably to a factor of five X the relative value versus big corporations.