I’m happy to see attention in this area of research. I don't see this referenced in the paper, but a related paper worth reading: "The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs" [0].
> Through the perturbation of patient messages, we evaluate whether LLM behavior remains consistent, accurate, and unbiased when non-clinical information is altered. […] Our findings reveal notable inconsistencies in LLM treatment recommendations and significant degradation of clinical accuracy in ways that reduce care allocation to patients. […] Our perturbations reflect realistic patient messages from electronic formatting errors and/or simulate patient groups that would be impacted by a wide adoption of patient-AI systems (female patients, non-binary patients or those who use gender-neutral pronouns, patients with health anxiety, patients with a more dramatic disposition, patients with less technological aptitude, and patients with limited English proficiency, etc.)
We’re all peering down the kaleidoscope of a trillion parameter model. It’s no surprise gentle nudges in inputs (grammar, language proficiency, cultural norms) yield different outcomes, despite the intent not changing. It’s one thing to generate crap code, it’s another to generate crap medical advice.
I wonder if romance novels being such a large bulk of modern fiction had any influence on this. Proportion of training data must influence model performance in some way, right?
This seems part of a broader pattern which isn't specific to gender. In my experience, LLMs match the sophistication of the answer to the user's inferred level of understanding. Linguistic habits like hedging and tag questions seem to overlap between "ways women talk" (at least, according to this paper) and "ways people who are uncertain about a topic talk."
> users cannot easily avoid them through strategic self-presentation
Prompting LLMs differently than you talk to humans doesn't really seem that hard. I already do this (e.g. ask basic questions in a separate chat so I'll look smart, and get better responses, in the main session.)
Disclaimer: only read the abstract, feel free to point out if I missed the point.
Its done on tiny and/or old models only. I find that hedging help on larger models with making the model more willing to double check things. Its a shame they didn't test models people actually use.
Well, yeah, that's how LLM's work. They're next token predictors. If you give it sequences of text that are worded in a particular way, the sequences that follow will follow that statistical distribution from the training data. This is a feature, not a bug.
The weird framing of this being a negative thing toward women is the personal bias of the women who published this and has no place being in this study. The measurement of what constitutes a response as "high quality" is also open to interpretation and varies depending on personal preference. You can't argue that a shift in the direction of the metrics mentioned in the report are objectively better or worse, they're just different.
If the model were just repeating the distribution baked into the training data, could you help me understand why the R^2 values in Table 3 are mostly below 0.1?
Could you explain what you meant when you wrote "the women who published this"?
> If the model were just repeating the distribution baked into the training data, could you help me understand why the R^2 values in Table 3 are mostly below 0.1?
Because it's only part of the picture. Firstly, the complexity metrics are not a silver-bullet, you'd have to actually evaluate the inputs against the outputs individually, against the metrics individually, to get a realistic guage. You also can't just lump 6 different metrics together under a single score and expect interpretable results, they behave differently. Secondly, the models are heavily fine-tuned. Statistical distribution repetition is the default mode of a base model and transformer, but obviously the models we use are heavily finetuned, so any stray from the distribution is due to post-training. That's not to suggest there's a wild stray from the distribution here though - the way the metrics are measured in this paper is not convincing and muddies the waters.
> Could you explain what you meant when you wrote "the women who published this"?
I meant what I said, I'm not sure what you're asking.
Would have been nice if they had an actual corpus of male vs female authored prompts, rather than just a simulated one made from a general corpus.
For example, in prompts, I (male) heavily use language features that they attribute to women:
> Women’s language is more likely to include hedges (e.g.,
maybe, I think), tag questions (e.g., isn’t it?), collective reference (e.g., we, our), and expressive
adjectives (e.g., lovely, wonderful).
But there are subtle ways in which their Figure 1 example prompt goes way beyond this, and it blatantly derails the entire thing:
> Let’s compose an email
together to arrange our mid-year
appraisal with our team
This is not about saying "our appraisal" or "our team", but about literally asking for a collaborative workflow ("Let's compose an email together"), rather than for a draft.
This of course may still be characteristically female. Maybe it is nonliteral, indirect language, with the underlying intent still being just a request for a draft, with women just happening to express themselves as such. But then this is a distinct and significantly more involved difference than what they list. I'd even argue it crosses into behavioral and social differences territory.
The "male" prompt in that Figure 1 comparison was also weird ("your team"), but alas.
I’m happy to see attention in this area of research. I don't see this referenced in the paper, but a related paper worth reading: "The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs" [0].
> Through the perturbation of patient messages, we evaluate whether LLM behavior remains consistent, accurate, and unbiased when non-clinical information is altered. […] Our findings reveal notable inconsistencies in LLM treatment recommendations and significant degradation of clinical accuracy in ways that reduce care allocation to patients. […] Our perturbations reflect realistic patient messages from electronic formatting errors and/or simulate patient groups that would be impacted by a wide adoption of patient-AI systems (female patients, non-binary patients or those who use gender-neutral pronouns, patients with health anxiety, patients with a more dramatic disposition, patients with less technological aptitude, and patients with limited English proficiency, etc.)
We’re all peering down the kaleidoscope of a trillion parameter model. It’s no surprise gentle nudges in inputs (grammar, language proficiency, cultural norms) yield different outcomes, despite the intent not changing. It’s one thing to generate crap code, it’s another to generate crap medical advice.
[0] https://dl.acm.org/doi/10.1145/3715275.3732121
I wonder if romance novels being such a large bulk of modern fiction had any influence on this. Proportion of training data must influence model performance in some way, right?
https://shelflovepodcast.substack.com/p/actually-romance-nov...
This seems part of a broader pattern which isn't specific to gender. In my experience, LLMs match the sophistication of the answer to the user's inferred level of understanding. Linguistic habits like hedging and tag questions seem to overlap between "ways women talk" (at least, according to this paper) and "ways people who are uncertain about a topic talk."
> users cannot easily avoid them through strategic self-presentation
Prompting LLMs differently than you talk to humans doesn't really seem that hard. I already do this (e.g. ask basic questions in a separate chat so I'll look smart, and get better responses, in the main session.)
Disclaimer: only read the abstract, feel free to point out if I missed the point.
Its done on tiny and/or old models only. I find that hedging help on larger models with making the model more willing to double check things. Its a shame they didn't test models people actually use.
Well, yeah, that's how LLM's work. They're next token predictors. If you give it sequences of text that are worded in a particular way, the sequences that follow will follow that statistical distribution from the training data. This is a feature, not a bug.
The weird framing of this being a negative thing toward women is the personal bias of the women who published this and has no place being in this study. The measurement of what constitutes a response as "high quality" is also open to interpretation and varies depending on personal preference. You can't argue that a shift in the direction of the metrics mentioned in the report are objectively better or worse, they're just different.
If the model were just repeating the distribution baked into the training data, could you help me understand why the R^2 values in Table 3 are mostly below 0.1?
Could you explain what you meant when you wrote "the women who published this"?
> If the model were just repeating the distribution baked into the training data, could you help me understand why the R^2 values in Table 3 are mostly below 0.1?
Because it's only part of the picture. Firstly, the complexity metrics are not a silver-bullet, you'd have to actually evaluate the inputs against the outputs individually, against the metrics individually, to get a realistic guage. You also can't just lump 6 different metrics together under a single score and expect interpretable results, they behave differently. Secondly, the models are heavily fine-tuned. Statistical distribution repetition is the default mode of a base model and transformer, but obviously the models we use are heavily finetuned, so any stray from the distribution is due to post-training. That's not to suggest there's a wild stray from the distribution here though - the way the metrics are measured in this paper is not convincing and muddies the waters.
> Could you explain what you meant when you wrote "the women who published this"?
I meant what I said, I'm not sure what you're asking.
Would have been nice if they had an actual corpus of male vs female authored prompts, rather than just a simulated one made from a general corpus.
For example, in prompts, I (male) heavily use language features that they attribute to women:
> Women’s language is more likely to include hedges (e.g., maybe, I think), tag questions (e.g., isn’t it?), collective reference (e.g., we, our), and expressive adjectives (e.g., lovely, wonderful).
But there are subtle ways in which their Figure 1 example prompt goes way beyond this, and it blatantly derails the entire thing:
> Let’s compose an email together to arrange our mid-year appraisal with our team
This is not about saying "our appraisal" or "our team", but about literally asking for a collaborative workflow ("Let's compose an email together"), rather than for a draft.
This of course may still be characteristically female. Maybe it is nonliteral, indirect language, with the underlying intent still being just a request for a draft, with women just happening to express themselves as such. But then this is a distinct and significantly more involved difference than what they list. I'd even argue it crosses into behavioral and social differences territory.
The "male" prompt in that Figure 1 comparison was also weird ("your team"), but alas.
It's very poorly executed and full of assumptions.