I am shocked by the comments in this thread ranging from “yeah but humans also make mistakes” to “yeah but how many mistakes does an AI do compared to a human”. Neither of those is the point here. We have doctors that have employed AI as a software to help them and that software is flawed. I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion
Somehow the standards we have for every other kind of automation go out the window for AI.
If it turned out an LLM embezzled funds and spent them at an internet casino, we would have folks in the comments explaining that what really matters is the embezzlement rate compared to humans doing the same job.
The cargo cult around AI on this website is making me second guess a career choice that has always been obvious to me.
In addition to not enjoying my work as much as what I used to because it's become babysitting an superpowered AI toddler, I now have to deal with this kind of opinion online.
Think of it as a symptom of late stage capitalism - the financialization has progressed so much that it's lost all connection to the underlying reality. It used to matter whether a technology could solve a problem, then it started to also matter whether this fact could be explained to savvy investors, and now it only matters whether it can be explained to stupid investors because they are the ones with the money. You used to have to sell a product people wanted to get their money, but now customers don't have any money because it's all with billionaires so those are the only people you have to please.
This! The problem is not whether AI makes more or less mistakes than a human. But for decades, people have been used to computers either working, or crashing, but never working wrong or misleading. AI changes that, and people really need to understand that. But that goes against the interest of AI provider's and their investor's interests, so the point is not being transmitted to the end users prominently enough.
> I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion
Based on watching the medical software field as a consumer (patient) and friends who are doctors, this is a fantasy. The quality of software in this field is abysmal and there seems to be almost no repercussions to those who develop or sell it.
Which is precisely why this sort of thing can be rolled out without much fear by those pushing it.
The software is heavily regulated for medical devices. Saying an MRI machine has bad software seems highly unlikely to me. This is of course different from Epic, but even then as a patient MyChart is really not that bad
The reasoning seems to be "Bad thing X existed for a long time with no solution. That means it's okay to make it worse, because if it was actually a problem it would've been solved by now. Plus it's not my job to solve X."
I would be curious as to which (kind of) model they've used there, how the tooling and prompts and all look like, etc.
I bet that to make the business case viable (or rather profitable), it's probably something small and cheap.
Bigger and better models don't come with any guarantees as to correctness either, but they do push down the probability of something as wrong as this happening by orders of magnitude.
__
Point being that I wouldn't necessarily blame it on the tech itself, but rather the (probably) startup, the culture and the fact that no one is going to jail here.
Your honor, the floats are innocent. They were simply forced to do this by the evil startup founder.
The other side is that I have witnessed doctors making mistakes personally on medical files. They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors. It would be interesting to compare whether transcription software or humans make the most mistakes. Guessing the answer will be very nuanced such as transcription software failing for accents not trained on, how over worked the clinic is etc.
How overworked would a doctor need to be to hallucinate you told them about microdosing psillocybin when you never said anything of the sort? (And how easy is it to then convince the system you're not a recreational drug user when it's in your medical records you admitted to drug use? Does this change if you're from a background more likely to be stereotyped as drug users?)
The humans are probably making mistakes like writing 100mg of something when they meant 10mg, which can also have dangerous consequences, but to an extent it's known this happens and there are processes to catch it. The type of mistakes and how much trouble and distress they cause matters as much as the number.
> They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors.
Damn, my doctor is always happy to give me 45 minutes of her time or more. Sometimes I feel like I'm taking up too much of her time chatting her ear off, but she's never in a rush to get me out of the door.
There's another option, which is to have the consultation recorded and then transcribed by a third person, or at least have the transcript reviewed while listening to the recording. I think for medical situations it's more than reasonable to put in the labor to ensure correctness.
Medical transcription isn't the same thing as medical scribing. Transcription has been done primarily by speech recognition software with human review for decades. The output is just a text chart note, which isn't particularly useful because none of the clinical findings are coded. The coding can either be done up front by a human or AI scribe, or as a second step after transcription. In any case the human attending clinician is legally accountable for reviewing and approving the output before it's officially added to the patient chart.
Think about what this literally means. There are a million doctors in the US. Doctors spend their day consulting patients, so, you need a million transcription reviewers. Let's say you pay each of them $50k/yr (aka rock bottom US entry-level wage).
That's $50 billion a year, and a million people pulled out of the labor force. Is that worth it? Is that really the best thing you can do with $50 billion dollars? In real life, tradeoffs exist.
There are already a lot of people doing manual review and editing of medical transcription speech recognition output. That has been the status quo for decades. A lot of those workers are offshore in India and Philippines. Overall transcription work is slowly declining as more physicians switch to direct EHR template data entry and AI scribes.
> Human medical scribes manually document encounters in real-time and, in randomized trials, are more than four times as likely to produce notes physicians rate as ‘accurate’ compared with standard self-documentation4. Automated speech-recognition dictation systems generally have higher error rates—typically 7–11%—owing to the complexity of medical jargon and accent variability5
It used to be common in some places to outsource medical transcription to India (obviously not real-time recording). I spent time in India years ago working with a company in Mumbai (not medical transcription) and there were stories in Indian newspapers about mistakes being made due to the transcribers having no medical knowledge at all. One instance that I've never forgotten is the one where the Doctor had diagnosed "phlebitis" in the patient's leg, and the transcriber had written "a flea bite his leg".
You can optimize work. But you can't optimize AWAY work. If you depend on human attention, expertise and judgement, as we do in medicine, you need that trained doctor to have the time to look at the transcript and think carefully about the contents. If we use AI as a way to help doctors spend even less time on each patient's case, we're going at it the wrong way. It shouldn't help cut costs, it should help improve the quality of care.
The classic whisper model will give you text in silent moments too. A lot of these guys just wrapped that. You can’t just do the minimal thing. Well I guess you can and you’ll get sales but it won’t correctly solve the problem.
My cousin used one of these apps with patient consent and said she then has to listen to the audio and rewrite it all. Stopped for that reason. Wasn’t even a time saver.
Strangely, I think Robin really just mistimed this. They stopped just as the state of the art came out and with their human in the loop transcription they might have been quite useful.
I think we need to keep in mind that medical and pharmacy mistakes are still extremely common
Recently my partner received a prescription with instructions that were over the LD50 (we caught it as it was obviously too high). We reported it but it was likely the case of the clerk simply hitting the wrong button and not double checking the resulting sticker
I’d be interested to see the AI transcription failure rate compared with existing medical/pharmacy rates
EHRs and other electronic prescription applications pretty much all have automated alerts for inappropriate dosages (like mg versus mcg) as well as allergies and harmful drug-drug interactions. The prescribing clinician likely had to dismiss at least one alert in making that error.
It's easy to bring down the hammer of scope insensitivity by vividly describing a single person's struggle. The question is how AI affects error rates.
This response always comes up but I think it's not the whole story and gives "AI" a pass.
If I report a bug in (for example) Slack that loses messages and cost me a lot of time and headaches, is it an appropriate response to say "but Teams has even more bugs" or "sure, but if you had that conversation face to face you might miss something too"?
The article doesn't meet a bug report standard. How would you feel about a bug report that doesn't mention which software were used, which versions were used, but it extensively describes all the harms that were caused by the lost messages.
I treat it as it is: a trendy hit piece against AI. There's nothing like "Why they still use Whisper Large V2?" in sight. If this article pushes the establishment to be more transparent about the AI tools, good. But right now there's not much to discuss.
It's not about giving AI a free pass - obviously we should try to remove these errors. It's about whether one reasonable option for removing these errors is to stop using AI or not. If AI increases error rates that would be a reasonable option. If AI doesn't increase the error rates, or even reduces them, it might not be.
And hence comparing AI use to a fictional situation where no errors happens is not meaningful.
I've personally caught multiple errors that were not just transcription errors, but elementary reasoning errors done by specialists I've seen - the baseline error rate from healthcare providers is far above zero.
The question is how AI affects error rates and error recovery rates!
My experience dealing with ai-mediated processes is that the error recovery paths often simply don't exist, presumably because eliminating the personel that dealt with oddball and errors was the supposed benefit of having the AI deal with it in the first place.
Yeah, but also, if we can make AI do the same work cheaper than doctors, we can probably reduce the wages of doctors and save a bunch of money, which will both increase profits for health companies and reduce costs for patients. At some point, sufficient savings is better for patients than more doctors.
Not error rates which implicitly assumes all errors are of equal consequence, something like 1/N*∑(magnitude of consequences of error) * (did error occur)
Not only did the AI apparently hallucinate psilocybin consumption, but the doctor then hallucinated internal bleeding as a risk, which is not a known risk of such mushrooms.
But this seems a bit click baity.
You can challenge your medical record, and presumably an AI transcription service would be in there. Unless .au is special in that way.
The doctor probably didn’t even read it and there’s a good chance the AI program wrote the bit about bleeding too. Source: am physician who has tried these programs and doesn’t use them.
It seems like this entire category of program is designed to make the same work take more time and more human attention.
I can understand the appeal in the tech industry, where the increased costs can be deferred until the financial situation changes. But healthcare does not work like that.
Psychedelics do tend to cause dehydration as the body tries to filter them out, and if you don't have a steady fluid intake you can dry out pretty badly before it's done. (Source: I take LSD frequently)
I'd be hesitant to connect that to internal bleeding right off the bat though. Definitely would not trust claims that internal bleeding is a first-order effect of the drug.
I am not an AI evangelist by any means, but it does serve a purpose when used correctly.
I feel like if you asked any half-intelligent AI "please review these notes and flag points we should review for correctness, or check with the patient?"
I'm pretty sure it would pick up a huge chunk of issues?
My experience with whisper for example is that it tends to generate plausible-in-context content when there are long gaps or silence. I don’t think the errors as described are likely to be catchable by LLM without a lot of false positives; in context disclosing recreational drug use is something that is likely to come up in a medical consultation.
> Doctor: have you used any recreational drugs in the last six months
> Patient: No
> Doctor: < long pause as they review notes >
> Patient (hallucinated most likely response): umm, actually there was one thing I hesitated to mention. Me and my girlfriends tried microdosing…
> I am not an AI evangelist by any means, but it does serve a purpose when used correctly.
Well, doesn't every tool? The devil is in the details of just how easy it is to use correctly versus incorrectly, and what are the consequences if it's not used correctly. A tool that, when used as its manufacturer advises, fails as often as most AI systems do has no place in systems where peoples' health and safety can be harmed.
This is basically living in a fantasy world. Errors are routine.
One of my notes in my current medical file states I injured my shoulder playing for a NFL team. I have never played football, and certainly not at a professional level. Fixing it is sort of like trying to fix your credit report - you supposedly get it done, and then 6mo later the same error pops back up again.
Ironically the reason this note exists is very similar to a way an AI scribe would misinterpret a conversation.
When using an LLM for details like this I keep it out of the mode of generating prose as much and as long as possible. I get better results when I work with bullet points. I think this helps me keep the right frame of mind when reading the responses. It’s easier to scan the facts or points for accuracy. And I think it might help the LLM to be less creative with those points, but I don’t rely on that.
A friend recently had a drug test added to their blood draw for an unrelated health check at a hospital. When she questioned the nurse about it turned out the doctors make notes that are transcribed by AI. And in this case the AI had mistakenly transcribed that she was on fentanyl patches.
"the nurse who was trying to be very nice said in a less sweet tone that the doctor is supposed to review the notes so this does not happen. And that she is very glad I caught it and the doctor will be glad too. I do not know if this was something Very Serious or an ongoing problem or what."
Given this threads story is the second incident of this I've heard in a week it seems like it is a common error with very serious consequences.
Yes. Perhaps pushing sophisticated models that require validation, on users who most definitely won't (and the whole premise of the model itself is that it spares them work) was a bad idea...
Validation will measure accuracy, and even if you require and get > 99.9% accuracy (and they should! don't get me wrong), it won't change that there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.
In fact, there will be MORE of those stories because it is shocking and strange, and clickworthy! The readers demand stories! But the reality is that there is no world where a story like this meaningfully informs the public about the accuracy of the services in question and the tradeoffs involved.
> even if you require and get > 99.9% accuracy [...] there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.
This is known as the long tail problem in ML, and it's a reality in almost every field. It's also why we don't officially have self driving cars, despite there being thousands of videos out there with cars driving autonomously for hours without any errors. But every now and then, there will be cases where the system doesn't work.
There are two interesting aspects here. One, we don't actually have quantitative data on how often this happens in the same field when human errors don't get caught. In a perfect world we'd have that, plus a long study for the "AI" systems, and we'd get to compare the two. Secondly, even if we'd have that data, people would still act out against "the machine" in the (ideally fewer) cases where it errors out, compared to a doctor. It's part human nature, part (manufactured) rage against the machines.
I also agree with your second paragraph. Case in point, when a waymo hit a cat, we got a shit ton of articles, riled up communities, and so on. Or every time that other car hits something we get plenty of press, even if some of the "accidents" are fender benders that likely wouldn't get reported otherwise.
There are estimated hundreds of millions of medical transcription errors per year without AI. 42.4% of finalized medical notes still contained at least one error.
Prior to AI it would be about voice to text. Prior to that it would be some transcriber in India. Prior to that it would be the doctor themselves
The important question with any AI-related news story is what is the counterfactual.
Sure, it sucks if your self driving car gets in a crash or your AI scribe incorrectly transcribes something to your medical record. This is news now. What isn't news is humans getting into crashes or doctors making poor medical decisions as a result of low quality or missing notes.
> I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.
This happens every single day and is effectively never reported. For far more nefarious reasons than a simple scribing error.
Drug seeking behavior enters notes all the time without much evidence and based entirely on a random doctor's (or even a triage nurse) hunch. A significant portion of those notes are outright false and incorrect. Once that is on your file and in a given medical system, you are marked for life.
When doctors make mistakes, they can be held accountable -- their malpractice insurance rates go up, their licenses are subject to suspension or revocation, they can go to jail (eg if they are pill mills) or they/the practice get a bad review.
When AI makes mistakes, what happens? How is it held accountable?
Doctors make mistakes pretty much non-stop, and they are rarely held accountable because the system as a whole works OK. AI makes less mistakes than the doctors do (I read a lot of medical notes).
You know what? I visited multiple PCPs and psychiatric providers over the course of several years, and during those years I was repeatedly being subjected to secondhand THC, because one of my neighbors was always smoking pot, and it was tangibly, perceptibly seeping into my bedroom. While I slept, while I worked, whatever, whenever I could not avoid it, I was inhaling it for hours on end, and I know that it affected me in subtle ways.
So I was always up-front with these providers. I figured they had a right to know. They asked if I was using drugs, and I would explain the situation. Obviously it is not voluntary use.
But, sure as shootin', "habitual marijuana use" showed up in my chart. That kind of shit is hard to deny and it'd be impossible to appeal or erase that. So I live with it. I have come around to the viewpoint that it is wise to conceal many material facts from your doctors. Conceal as much as possible. Lying is better than the alternative.
I am shocked by the comments in this thread ranging from “yeah but humans also make mistakes” to “yeah but how many mistakes does an AI do compared to a human”. Neither of those is the point here. We have doctors that have employed AI as a software to help them and that software is flawed. I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion
Somehow the standards we have for every other kind of automation go out the window for AI.
If it turned out an LLM embezzled funds and spent them at an internet casino, we would have folks in the comments explaining that what really matters is the embezzlement rate compared to humans doing the same job.
The cargo cult around AI on this website is making me second guess a career choice that has always been obvious to me.
In addition to not enjoying my work as much as what I used to because it's become babysitting an superpowered AI toddler, I now have to deal with this kind of opinion online.
Think of it as a symptom of late stage capitalism - the financialization has progressed so much that it's lost all connection to the underlying reality. It used to matter whether a technology could solve a problem, then it started to also matter whether this fact could be explained to savvy investors, and now it only matters whether it can be explained to stupid investors because they are the ones with the money. You used to have to sell a product people wanted to get their money, but now customers don't have any money because it's all with billionaires so those are the only people you have to please.
This! The problem is not whether AI makes more or less mistakes than a human. But for decades, people have been used to computers either working, or crashing, but never working wrong or misleading. AI changes that, and people really need to understand that. But that goes against the interest of AI provider's and their investor's interests, so the point is not being transmitted to the end users prominently enough.
> I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion
Based on watching the medical software field as a consumer (patient) and friends who are doctors, this is a fantasy. The quality of software in this field is abysmal and there seems to be almost no repercussions to those who develop or sell it.
Which is precisely why this sort of thing can be rolled out without much fear by those pushing it.
The software is heavily regulated for medical devices. Saying an MRI machine has bad software seems highly unlikely to me. This is of course different from Epic, but even then as a patient MyChart is really not that bad
The reasoning seems to be "Bad thing X existed for a long time with no solution. That means it's okay to make it worse, because if it was actually a problem it would've been solved by now. Plus it's not my job to solve X."
I would be curious as to which (kind of) model they've used there, how the tooling and prompts and all look like, etc.
I bet that to make the business case viable (or rather profitable), it's probably something small and cheap.
Bigger and better models don't come with any guarantees as to correctness either, but they do push down the probability of something as wrong as this happening by orders of magnitude.
__
Point being that I wouldn't necessarily blame it on the tech itself, but rather the (probably) startup, the culture and the fact that no one is going to jail here.
Your honor, the floats are innocent. They were simply forced to do this by the evil startup founder.
The other side is that I have witnessed doctors making mistakes personally on medical files. They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors. It would be interesting to compare whether transcription software or humans make the most mistakes. Guessing the answer will be very nuanced such as transcription software failing for accents not trained on, how over worked the clinic is etc.
How overworked would a doctor need to be to hallucinate you told them about microdosing psillocybin when you never said anything of the sort? (And how easy is it to then convince the system you're not a recreational drug user when it's in your medical records you admitted to drug use? Does this change if you're from a background more likely to be stereotyped as drug users?)
The humans are probably making mistakes like writing 100mg of something when they meant 10mg, which can also have dangerous consequences, but to an extent it's known this happens and there are processes to catch it. The type of mistakes and how much trouble and distress they cause matters as much as the number.
> They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors.
Damn, my doctor is always happy to give me 45 minutes of her time or more. Sometimes I feel like I'm taking up too much of her time chatting her ear off, but she's never in a rush to get me out of the door.
Perks of small town living?
There's another option, which is to have the consultation recorded and then transcribed by a third person, or at least have the transcript reviewed while listening to the recording. I think for medical situations it's more than reasonable to put in the labor to ensure correctness.
Medical transcription isn't the same thing as medical scribing. Transcription has been done primarily by speech recognition software with human review for decades. The output is just a text chart note, which isn't particularly useful because none of the clinical findings are coded. The coding can either be done up front by a human or AI scribe, or as a second step after transcription. In any case the human attending clinician is legally accountable for reviewing and approving the output before it's officially added to the patient chart.
Cost of medical care is already sky high (in the US). Are you sure you want to introduce even more administrative overhead?
How were people doing this a few years ago before generative AI? Did the cost of care go down significantly and I just missed it?
Think about what this literally means. There are a million doctors in the US. Doctors spend their day consulting patients, so, you need a million transcription reviewers. Let's say you pay each of them $50k/yr (aka rock bottom US entry-level wage).
That's $50 billion a year, and a million people pulled out of the labor force. Is that worth it? Is that really the best thing you can do with $50 billion dollars? In real life, tradeoffs exist.
There are already a lot of people doing manual review and editing of medical transcription speech recognition output. That has been the status quo for decades. A lot of those workers are offshore in India and Philippines. Overall transcription work is slowly declining as more physicians switch to direct EHR template data entry and AI scribes.
> Human medical scribes manually document encounters in real-time and, in randomized trials, are more than four times as likely to produce notes physicians rate as ‘accurate’ compared with standard self-documentation4. Automated speech-recognition dictation systems generally have higher error rates—typically 7–11%—owing to the complexity of medical jargon and accent variability5
https://pmc.ncbi.nlm.nih.gov/articles/PMC12460601/
It used to be common in some places to outsource medical transcription to India (obviously not real-time recording). I spent time in India years ago working with a company in Mumbai (not medical transcription) and there were stories in Indian newspapers about mistakes being made due to the transcribers having no medical knowledge at all. One instance that I've never forgotten is the one where the Doctor had diagnosed "phlebitis" in the patient's leg, and the transcriber had written "a flea bite his leg".
You can optimize work. But you can't optimize AWAY work. If you depend on human attention, expertise and judgement, as we do in medicine, you need that trained doctor to have the time to look at the transcript and think carefully about the contents. If we use AI as a way to help doctors spend even less time on each patient's case, we're going at it the wrong way. It shouldn't help cut costs, it should help improve the quality of care.
They’re just gonna have to think a lot faster. Scuttlebutt is we’re going to 12k patient panels, up from about 2–3k. Better to prep for what’s coming.
I am surprised the doctors do not have to review a bit the transcription. I do have to review my code before deployment
This isn’t a problem of you use actually capable models instead of wrapping whatever open source junk and pretend you have an AI product.
The classic whisper model will give you text in silent moments too. A lot of these guys just wrapped that. You can’t just do the minimal thing. Well I guess you can and you’ll get sales but it won’t correctly solve the problem.
My cousin used one of these apps with patient consent and said she then has to listen to the audio and rewrite it all. Stopped for that reason. Wasn’t even a time saver.
Strangely, I think Robin really just mistimed this. They stopped just as the state of the art came out and with their human in the loop transcription they might have been quite useful.
I think we need to keep in mind that medical and pharmacy mistakes are still extremely common
Recently my partner received a prescription with instructions that were over the LD50 (we caught it as it was obviously too high). We reported it but it was likely the case of the clerk simply hitting the wrong button and not double checking the resulting sticker
I’d be interested to see the AI transcription failure rate compared with existing medical/pharmacy rates
EHRs and other electronic prescription applications pretty much all have automated alerts for inappropriate dosages (like mg versus mcg) as well as allergies and harmful drug-drug interactions. The prescribing clinician likely had to dismiss at least one alert in making that error.
The issues were found and no one was hurt so they're not to the level of Therac-25, but it really feels like a matter of when, not if.
It's easy to bring down the hammer of scope insensitivity by vividly describing a single person's struggle. The question is how AI affects error rates.
This response always comes up but I think it's not the whole story and gives "AI" a pass.
If I report a bug in (for example) Slack that loses messages and cost me a lot of time and headaches, is it an appropriate response to say "but Teams has even more bugs" or "sure, but if you had that conversation face to face you might miss something too"?
The article doesn't meet a bug report standard. How would you feel about a bug report that doesn't mention which software were used, which versions were used, but it extensively describes all the harms that were caused by the lost messages.
I treat it as it is: a trendy hit piece against AI. There's nothing like "Why they still use Whisper Large V2?" in sight. If this article pushes the establishment to be more transparent about the AI tools, good. But right now there's not much to discuss.
It's not about giving AI a free pass - obviously we should try to remove these errors. It's about whether one reasonable option for removing these errors is to stop using AI or not. If AI increases error rates that would be a reasonable option. If AI doesn't increase the error rates, or even reduces them, it might not be.
And hence comparing AI use to a fictional situation where no errors happens is not meaningful.
I've personally caught multiple errors that were not just transcription errors, but elementary reasoning errors done by specialists I've seen - the baseline error rate from healthcare providers is far above zero.
The question is how AI affects error rates and error recovery rates!
My experience dealing with ai-mediated processes is that the error recovery paths often simply don't exist, presumably because eliminating the personel that dealt with oddball and errors was the supposed benefit of having the AI deal with it in the first place.
Yeah, but also, if we can make AI do the same work cheaper than doctors, we can probably reduce the wages of doctors and save a bunch of money, which will both increase profits for health companies and reduce costs for patients. At some point, sufficient savings is better for patients than more doctors.
You mean increase the profits AND increase the costs....
Not error rates which implicitly assumes all errors are of equal consequence, something like 1/N*∑(magnitude of consequences of error) * (did error occur)
Not only did the AI apparently hallucinate psilocybin consumption, but the doctor then hallucinated internal bleeding as a risk, which is not a known risk of such mushrooms.
But this seems a bit click baity.
You can challenge your medical record, and presumably an AI transcription service would be in there. Unless .au is special in that way.
The doctor probably didn’t even read it and there’s a good chance the AI program wrote the bit about bleeding too. Source: am physician who has tried these programs and doesn’t use them.
It seems like this entire category of program is designed to make the same work take more time and more human attention.
I can understand the appeal in the tech industry, where the increased costs can be deferred until the financial situation changes. But healthcare does not work like that.
Psychedelics do tend to cause dehydration as the body tries to filter them out, and if you don't have a steady fluid intake you can dry out pretty badly before it's done. (Source: I take LSD frequently)
I'd be hesitant to connect that to internal bleeding right off the bat though. Definitely would not trust claims that internal bleeding is a first-order effect of the drug.
I am not an AI evangelist by any means, but it does serve a purpose when used correctly.
I feel like if you asked any half-intelligent AI "please review these notes and flag points we should review for correctness, or check with the patient?"
I'm pretty sure it would pick up a huge chunk of issues?
My experience with whisper for example is that it tends to generate plausible-in-context content when there are long gaps or silence. I don’t think the errors as described are likely to be catchable by LLM without a lot of false positives; in context disclosing recreational drug use is something that is likely to come up in a medical consultation.
> Doctor: have you used any recreational drugs in the last six months > Patient: No > Doctor: < long pause as they review notes > > Patient (hallucinated most likely response): umm, actually there was one thing I hesitated to mention. Me and my girlfriends tried microdosing…
> I am not an AI evangelist by any means, but it does serve a purpose when used correctly.
Well, doesn't every tool? The devil is in the details of just how easy it is to use correctly versus incorrectly, and what are the consequences if it's not used correctly. A tool that, when used as its manufacturer advises, fails as often as most AI systems do has no place in systems where peoples' health and safety can be harmed.
It's a medical file, those rarely contain idle chit-chat. Everything should be reviewed for correctness.
> Everything should be reviewed for correctness.
Have you ever actually read your notes on file?
This is basically living in a fantasy world. Errors are routine.
One of my notes in my current medical file states I injured my shoulder playing for a NFL team. I have never played football, and certainly not at a professional level. Fixing it is sort of like trying to fix your credit report - you supposedly get it done, and then 6mo later the same error pops back up again.
Ironically the reason this note exists is very similar to a way an AI scribe would misinterpret a conversation.
When using an LLM for details like this I keep it out of the mode of generating prose as much and as long as possible. I get better results when I work with bullet points. I think this helps me keep the right frame of mind when reading the responses. It’s easier to scan the facts or points for accuracy. And I think it might help the LLM to be less creative with those points, but I don’t rely on that.
Just prompt with "make no mistakes". (/s for clarity)
I wonder if this app in question is Heidi Health? A lot of doctors in Australia use it.
A friend recently had a drug test added to their blood draw for an unrelated health check at a hospital. When she questioned the nurse about it turned out the doctors make notes that are transcribed by AI. And in this case the AI had mistakenly transcribed that she was on fentanyl patches.
"the nurse who was trying to be very nice said in a less sweet tone that the doctor is supposed to review the notes so this does not happen. And that she is very glad I caught it and the doctor will be glad too. I do not know if this was something Very Serious or an ongoing problem or what."
Given this threads story is the second incident of this I've heard in a week it seems like it is a common error with very serious consequences.
> the doctor will be glad too.
So even the doctor forgot what they had talked about, and assumed the transcription was right? Crikey.
Yes. Perhaps pushing sophisticated models that require validation, on users who most definitely won't (and the whole premise of the model itself is that it spares them work) was a bad idea...
Validation will measure accuracy, and even if you require and get > 99.9% accuracy (and they should! don't get me wrong), it won't change that there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.
In fact, there will be MORE of those stories because it is shocking and strange, and clickworthy! The readers demand stories! But the reality is that there is no world where a story like this meaningfully informs the public about the accuracy of the services in question and the tradeoffs involved.
> even if you require and get > 99.9% accuracy [...] there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.
This is known as the long tail problem in ML, and it's a reality in almost every field. It's also why we don't officially have self driving cars, despite there being thousands of videos out there with cars driving autonomously for hours without any errors. But every now and then, there will be cases where the system doesn't work.
There are two interesting aspects here. One, we don't actually have quantitative data on how often this happens in the same field when human errors don't get caught. In a perfect world we'd have that, plus a long study for the "AI" systems, and we'd get to compare the two. Secondly, even if we'd have that data, people would still act out against "the machine" in the (ideally fewer) cases where it errors out, compared to a doctor. It's part human nature, part (manufactured) rage against the machines.
I also agree with your second paragraph. Case in point, when a waymo hit a cat, we got a shit ton of articles, riled up communities, and so on. Or every time that other car hits something we get plenty of press, even if some of the "accidents" are fender benders that likely wouldn't get reported otherwise.
AI makes errors, humans make errors.
The relevant information is the rate for each.
There are estimated hundreds of millions of medical transcription errors per year without AI. 42.4% of finalized medical notes still contained at least one error.
Prior to AI it would be about voice to text. Prior to that it would be some transcriber in India. Prior to that it would be the doctor themselves
Prior to AI, you could get a person on the phone or in-person with the capacity to correct errors.
garbage in, garbage out.
but with AI you get to add "hallucinations" to the mix.
The important question with any AI-related news story is what is the counterfactual.
Sure, it sucks if your self driving car gets in a crash or your AI scribe incorrectly transcribes something to your medical record. This is news now. What isn't news is humans getting into crashes or doctors making poor medical decisions as a result of low quality or missing notes.
I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.
> I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.
This happens every single day and is effectively never reported. For far more nefarious reasons than a simple scribing error.
Drug seeking behavior enters notes all the time without much evidence and based entirely on a random doctor's (or even a triage nurse) hunch. A significant portion of those notes are outright false and incorrect. Once that is on your file and in a given medical system, you are marked for life.
Exactly.
When doctors make mistakes, they can be held accountable -- their malpractice insurance rates go up, their licenses are subject to suspension or revocation, they can go to jail (eg if they are pill mills) or they/the practice get a bad review.
When AI makes mistakes, what happens? How is it held accountable?
Doctors make mistakes pretty much non-stop, and they are rarely held accountable because the system as a whole works OK. AI makes less mistakes than the doctors do (I read a lot of medical notes).
You know what? I visited multiple PCPs and psychiatric providers over the course of several years, and during those years I was repeatedly being subjected to secondhand THC, because one of my neighbors was always smoking pot, and it was tangibly, perceptibly seeping into my bedroom. While I slept, while I worked, whatever, whenever I could not avoid it, I was inhaling it for hours on end, and I know that it affected me in subtle ways.
So I was always up-front with these providers. I figured they had a right to know. They asked if I was using drugs, and I would explain the situation. Obviously it is not voluntary use.
But, sure as shootin', "habitual marijuana use" showed up in my chart. That kind of shit is hard to deny and it'd be impossible to appeal or erase that. So I live with it. I have come around to the viewpoint that it is wise to conceal many material facts from your doctors. Conceal as much as possible. Lying is better than the alternative.