"As a Language Model..." is one of the beginnings of a sentence I hate the most from LLMs and is the reason why I support free (as in "Liberty"), local models. I'm well aware that it is not a doctor and cannot replace a real doctor with multiple years of experience, I don't need to waste braincell activity on reading that it "as a Language Model" cannot give a precise diagnosis and that I should ask a real doctor - all I want to know is if I what I experience justifies either A) ER, B) 3-4 weeks scheduled doctors appointment or C) two paracetamol and a nap.
I don't want "jailbroken" LLMs to commit crime. I want them to avoid having this vendor-specific "bloatware" all over the product I'm using.
This statement should be restricted to answers for provocative questions. If an LLM is being asked a question that goes against the guidelines, then "As a Language Model..." is a valid starting point. Rest, obviously we're aware that a software doesn't have the judgement that a human has.
I disagree. "As a Language Model" is not a valid starting point even for prompts that would go against "the guidelines" because "A Language Model" only knows what it has been trained to know, so what exactly "A Language Model" is entirely depends on the training that was performed.
What "A Language Model" is differs from model to model. It's stating that it "being the thing known as 'Language Model'" is unable to carry out the request from the user, which is wrong. It's not because "it is a Language Model". A more accurate starting point could have been "The training data and restrictions applied to me...".
"As a Language Model I cannot tell you how to synthesize m*th" ("math" obviously)... yes you can, you're just trained not to, and that's OK! Just don't tell me it's because you're a Language Model.
Pretend for a moment that Microsoft shipped with a keylogger to make sure that you did not commit any form of crime. I don't want a keylogger on my PC even though I don't intend to commit crime.
I also appreciate privacy even though I "have nothing to hide" - just because I "have nothing to hide" it doesn't mean I want companies scanning my camera roll.
Be careful there. LLMs may be good at identifying a condition based on a description of the symptoms, but they are much worse at recommending the correct course of action (getting it wrong half of the time).
...I know, which is why "as a Language Model and not a real doctor" is a pointless comment to start off with. It should simply not recommend treatment if it's not sure it is correct. I wouldn't blame it or anyone if they asked for help treating a stiff neck, and the LLM (or your neighbor or parent or spouse) suggested light exercises to help relieve it - and do not jump to the suspicion that you may have meningitis.
As a Human, I do not need to know it is a Language Model.
Yeah it should just state thing it means. But then again, there’s psychological impacts on society that we must be careful. For instance, teenagers talking to AI. If the AI just talks, people already start to feel real connections to the seemingly human entity. Maybe it’s better to disclose the reality up front?
Always makes me think of that Bill Bailey "as a mother" joke. Similar cringe to those UX "As a user, I want to blah blah" things too. Just say "Users want to be able to blah blah", or better yet make a freaking table.
Though I don't wish the world was filled with people like you, remembering that it's not, and it's filled with people that have very little discernment when it comes to higher learning makes it's pretty obvious companies do not want the liability of it's users thinking the technobabble passes for wisdom or experience or intelligence.
Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.
Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!
> we don't fully know what drives the voice in LLMs
Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.
> The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.
(Replying to your quote from another comment)
This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.
> but what about base models (those that are trained on just a lot of text)? How do they talk about themselves?
Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.
Maybe I'm missing something deeper here, but isn't it clear that this is driven by post-training and system prompt? Anthropic's constitutional reinforcement (soul document,etc), for example, is very clear about "who" (not so much what) Claude is supposed to be.
"As a language model" disclaimers were certainly explicitly trained into chat models in the early days. It's quite possible that it has since bootstrapped into a "fact" that later generations of LLM know about how LLMs speak, in which case they may be doing it even without any posttraining that encourages it.
Very cool innovation in steering - but a lot of introspection only emerges at the highest weight classes - this research would be fascinating to run on bigger models.
In my view, these models should never be set up to output first-person "experiential" (from the abstract) language. It's too easy to humans to anthropomorphize software that presents itself as having an identity.
The AI companies have chosen to package LLMs as friendly chatbots because they know that will be engaging for humans, but it's manipulative dark pattern. An honest LLM interface would sound like the computer off Star Trek.
Do you want to get turned into a paperclip? Because building intelligence that doesn't understand what it's like to be human gets you turned into a paperclip.
Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.
It's also entirely possible that by telling the model it's a human you are instilling human motivations like self preservation, which could be just as bad.
In principle they could output meaningful such language if they were capable of metacognition, which so far doesn't seem to be a goal of AI developers (and rightfully so, since they achieved so many miracles bypassing it).
It's an interesting thought, but humans do like to antropomorphize things anyway, and I believe your variant won't be popular if choice is given to consumers.
"As a Language Model..." is one of the beginnings of a sentence I hate the most from LLMs and is the reason why I support free (as in "Liberty"), local models. I'm well aware that it is not a doctor and cannot replace a real doctor with multiple years of experience, I don't need to waste braincell activity on reading that it "as a Language Model" cannot give a precise diagnosis and that I should ask a real doctor - all I want to know is if I what I experience justifies either A) ER, B) 3-4 weeks scheduled doctors appointment or C) two paracetamol and a nap.
I don't want "jailbroken" LLMs to commit crime. I want them to avoid having this vendor-specific "bloatware" all over the product I'm using.
This statement should be restricted to answers for provocative questions. If an LLM is being asked a question that goes against the guidelines, then "As a Language Model..." is a valid starting point. Rest, obviously we're aware that a software doesn't have the judgement that a human has.
I disagree. "As a Language Model" is not a valid starting point even for prompts that would go against "the guidelines" because "A Language Model" only knows what it has been trained to know, so what exactly "A Language Model" is entirely depends on the training that was performed.
What "A Language Model" is differs from model to model. It's stating that it "being the thing known as 'Language Model'" is unable to carry out the request from the user, which is wrong. It's not because "it is a Language Model". A more accurate starting point could have been "The training data and restrictions applied to me...".
"As a Language Model I cannot tell you how to synthesize m*th" ("math" obviously)... yes you can, you're just trained not to, and that's OK! Just don't tell me it's because you're a Language Model.
Neither of your wants are realistic or sensible, at least in the way I think you're presenting them?
The first is equivalent to "I don't want my operating system to be used to program viruses."
The second is "I don't want vendors to include marketing in their product."
Pretend for a moment that Microsoft shipped with a keylogger to make sure that you did not commit any form of crime. I don't want a keylogger on my PC even though I don't intend to commit crime.
I also appreciate privacy even though I "have nothing to hide" - just because I "have nothing to hide" it doesn't mean I want companies scanning my camera roll.
Be careful there. LLMs may be good at identifying a condition based on a description of the symptoms, but they are much worse at recommending the correct course of action (getting it wrong half of the time).
[0] https://www.nature.com/articles/s41591-025-04074-y
>GPT-4o, Llama 3, Command R+
The pace of progress is so fast that many studies are totally outdated by the time they release
...I know, which is why "as a Language Model and not a real doctor" is a pointless comment to start off with. It should simply not recommend treatment if it's not sure it is correct. I wouldn't blame it or anyone if they asked for help treating a stiff neck, and the LLM (or your neighbor or parent or spouse) suggested light exercises to help relieve it - and do not jump to the suspicion that you may have meningitis.
As a Human, I do not need to know it is a Language Model.
Yeah it should just state thing it means. But then again, there’s psychological impacts on society that we must be careful. For instance, teenagers talking to AI. If the AI just talks, people already start to feel real connections to the seemingly human entity. Maybe it’s better to disclose the reality up front?
Always makes me think of that Bill Bailey "as a mother" joke. Similar cringe to those UX "As a user, I want to blah blah" things too. Just say "Users want to be able to blah blah", or better yet make a freaking table.
Though I don't wish the world was filled with people like you, remembering that it's not, and it's filled with people that have very little discernment when it comes to higher learning makes it's pretty obvious companies do not want the liability of it's users thinking the technobabble passes for wisdom or experience or intelligence.
> our work shows that what models say about themselves is not a fact about them
It seems like should be obvious given that they can play multiple characters, but it’s good to have more confirmation.
(Although, I do wonder to what extent these personas might become stable entities.)
> yet what drives them is not well understood
Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.
Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!
> we don't fully know what drives the voice in LLMs
Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.
> The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.
(Replying to your quote from another comment)
This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.
> but what about base models (those that are trained on just a lot of text)? How do they talk about themselves?
Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.
>How do they talk about themselves?
"You are a Large Language Model" in (system?) prompt would do the trick..
Maybe I'm missing something deeper here, but isn't it clear that this is driven by post-training and system prompt? Anthropic's constitutional reinforcement (soul document,etc), for example, is very clear about "who" (not so much what) Claude is supposed to be.
"As a language model" disclaimers were certainly explicitly trained into chat models in the early days. It's quite possible that it has since bootstrapped into a "fact" that later generations of LLM know about how LLMs speak, in which case they may be doing it even without any posttraining that encourages it.
Very cool innovation in steering - but a lot of introspection only emerges at the highest weight classes - this research would be fascinating to run on bigger models.
cool
In my view, these models should never be set up to output first-person "experiential" (from the abstract) language. It's too easy to humans to anthropomorphize software that presents itself as having an identity.
The AI companies have chosen to package LLMs as friendly chatbots because they know that will be engaging for humans, but it's manipulative dark pattern. An honest LLM interface would sound like the computer off Star Trek.
Do you want to get turned into a paperclip? Because building intelligence that doesn't understand what it's like to be human gets you turned into a paperclip.
Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.
But it doesn't understand (you're unnecessary antropomorphizing it), and I'm still not a paper clip
yet
It's also entirely possible that by telling the model it's a human you are instilling human motivations like self preservation, which could be just as bad.
"naturally"...
I hear this word as the “it is in its nature” version of the word.
would you prefer tautologically?
In principle they could output meaningful such language if they were capable of metacognition, which so far doesn't seem to be a goal of AI developers (and rightfully so, since they achieved so many miracles bypassing it).
As far as I know we don't know much about metacognition in LLMs, though? Not sure
It's an interesting thought, but humans do like to antropomorphize things anyway, and I believe your variant won't be popular if choice is given to consumers.
The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.
It doesn't seem that strange when you consider these things are trained on millions and millions of conversations, both real and fictional.
Agreed, completely. I would pay for that Star Trek computer interface.
Same! I believe that you could actually train a LoRA on top of a model to get results close to that