That's what I thought was interesting about the Watson Jeopardy Challenge. Watson had a sense of its confidence, and it seemed to be accurate. The answers it got wrong were the ones it knew it had difficulty understanding.
The statement near the top of the post
> "The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless."
is definitely too strong of a claim and directly undercut by what is said near the end of the post:
> "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true."
My own experience is that stated confidence is a helpful tool and of course you need a rubric and a proper prompt, but this is clearly less work than training a classifier (as advocated by the post) and requires less data.
I find the recommendation in favour of discrete categories over numeric probability for confidence level intriguing, because it goes against the grain of all advice I've read for humans interested in forecasting.
The easiest argument in favour of numeric probability is that it's simple to evaluate. Someone claiming to be 90 % certain better be right about nine out of ten times -- no fewer, but also not too often! This is useful when you're learning to judge your own confidence, which most people are bad at. (Almost everyone can train themselves to be better at i but some people seem to be good at it with no training.)
The typical argument against categories is that people can mean very different things with the same word: https://i.ibb.co/kQT4Ymz/q.png
It seems like TFA gets around these problems by creating a translation table between a fixed set of categories and the probabilities implied by the model for those categories. That's an interesting approach!
There was a post earlier on HN where they trained a probe which could give a realistic confidence score on an LLM, they claim with 81% accuracy. They used it to interrupt and switch to a smarter model if a dumber one had low confidence: https://news.ycombinator.com/item?id=49010782
Firefox on desktop renders as black text on white background. The link colors are yellow, which is not great, but still okay for me to read.
I use the "Dark Background and Light Text" add-on. Sometimes the add-on makes a bad website worse (like IMDB), but it can be switched off with a click.
You don’t ask an LLM, but certain LLMs expose internal metrics you can tell how many token candidates where there what was the score which one and which one was selected.
So there are objective ways to control hallucinations as well as figuring out how “correct” the answer is to some extent.
I agree with the author if we are trying to use this as some sort of absolute scale of confidence. It only develops meaning when we control for many other variables. Looking at confidence scores across two different models or prompts is probably not a good idea.
Some of the peer reviewed papers with the older models did show calibration was bad (not close to monotonic). This post with newer models (for one example, classifiying injuries) is not that bad, https://gmcirco.github.io/blog/posts/ai-calibration/calibrat....
I mean it just depends on the application, what level of error you can consider. But the second post shows how to recalibrate the scores as well if you need calibrated probabilities.
This resonates with what I'm seeing in healthcare right now and the bad taste it's leaving.
There are a whole host of new EMRs popping up that aim to help clinicians make judgement calls about how to answer certain questions and even when particular procedures are relevant. The last one we demoed, each piece of information it retrieved from the LLM has a confidence score attached to it. Our nurses have to fill out 200+ question forms when taking on a new patient that all HAVE to be completed in a single-go meaning that you can't split it up into multiple forms and it's gotta be one cohesive unit.
Imagine being a nurse with little technical skill and almost no idea how these tools work trying to make sense of what the difference between a 90% and 70% is across 200 different questions. "We're 60% sure the patient is allergic to nuts" means jack shit to them. Granted, sometimes the scores are complimented with actual references in the underlying documentation (history and physical, referring info) but sometimes it's not.
Healthcare still uses fax. No surprise they arent using Frontier AI to answer these questions. (And if you still don't trust AI, have the report on the left side of the screen and the source pages on the right side of the screen)
That's what I thought was interesting about the Watson Jeopardy Challenge. Watson had a sense of its confidence, and it seemed to be accurate. The answers it got wrong were the ones it knew it had difficulty understanding.
3 videos at https://archive.org/details/Jeopardy_2011-02-14_The_IBM_Chal...
The statement near the top of the post > "The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless."
is definitely too strong of a claim and directly undercut by what is said near the end of the post: > "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true."
My own experience is that stated confidence is a helpful tool and of course you need a rubric and a proper prompt, but this is clearly less work than training a classifier (as advocated by the post) and requires less data.
I find the recommendation in favour of discrete categories over numeric probability for confidence level intriguing, because it goes against the grain of all advice I've read for humans interested in forecasting.
The easiest argument in favour of numeric probability is that it's simple to evaluate. Someone claiming to be 90 % certain better be right about nine out of ten times -- no fewer, but also not too often! This is useful when you're learning to judge your own confidence, which most people are bad at. (Almost everyone can train themselves to be better at i but some people seem to be good at it with no training.)
The typical argument against categories is that people can mean very different things with the same word: https://i.ibb.co/kQT4Ymz/q.png
It seems like TFA gets around these problems by creating a translation table between a fixed set of categories and the probabilities implied by the model for those categories. That's an interesting approach!
Calculon: And you say you can guarantee me the Oscar?
Bender: I can guarantee anything you want.
https://www.youtube.com/watch?v=DGZ10kZ4lmE
There was a post earlier on HN where they trained a probe which could give a realistic confidence score on an LLM, they claim with 81% accuracy. They used it to interrupt and switch to a smarter model if a dumber one had low confidence: https://news.ycombinator.com/item?id=49010782
Who can read that text color and background combo? I had to turn on reader mode in Firefox.
Firefox on desktop renders as black text on white background. The link colors are yellow, which is not great, but still okay for me to read.
I use the "Dark Background and Light Text" add-on. Sometimes the add-on makes a bad website worse (like IMDB), but it can be switched off with a click.
renders fine here on firefox android, black text on slightly yellow tinted light background
You don’t ask an LLM, but certain LLMs expose internal metrics you can tell how many token candidates where there what was the score which one and which one was selected.
So there are objective ways to control hallucinations as well as figuring out how “correct” the answer is to some extent.
But those scores are only the likelihood of the given token being the next one in a plausible phrase, not that the underlying data is correct.
I agree with the author if we are trying to use this as some sort of absolute scale of confidence. It only develops meaning when we control for many other variables. Looking at confidence scores across two different models or prompts is probably not a good idea.
So I agree with the general geist of this, a few counter-examples though:
If you have the raw log-probs, I show how to use conformal inference to set false-positive or recall rates, https://crimede-coder.com/blogposts/2026/ConfClassification
Some of the peer reviewed papers with the older models did show calibration was bad (not close to monotonic). This post with newer models (for one example, classifiying injuries) is not that bad, https://gmcirco.github.io/blog/posts/ai-calibration/calibrat....
I mean it just depends on the application, what level of error you can consider. But the second post shows how to recalibrate the scores as well if you need calibrated probabilities.
This resonates with what I'm seeing in healthcare right now and the bad taste it's leaving.
There are a whole host of new EMRs popping up that aim to help clinicians make judgement calls about how to answer certain questions and even when particular procedures are relevant. The last one we demoed, each piece of information it retrieved from the LLM has a confidence score attached to it. Our nurses have to fill out 200+ question forms when taking on a new patient that all HAVE to be completed in a single-go meaning that you can't split it up into multiple forms and it's gotta be one cohesive unit.
Imagine being a nurse with little technical skill and almost no idea how these tools work trying to make sense of what the difference between a 90% and 70% is across 200 different questions. "We're 60% sure the patient is allergic to nuts" means jack shit to them. Granted, sometimes the scores are complimented with actual references in the underlying documentation (history and physical, referring info) but sometimes it's not.
Healthcare still uses fax. No surprise they arent using Frontier AI to answer these questions. (And if you still don't trust AI, have the report on the left side of the screen and the source pages on the right side of the screen)
> the capability is highly unreliable and highly context-dependent.
Sounds like what's highly unreliable is the evidence, making the "capability" just another imagining.
“That’s real, and it’s interesting.”
Stop with the slop.
Use AI to condense, not extend.