If you're looking for "Hot Dog/Not A Hotdog" type (i.e., heavily bounded) answers from LLMs, make certain to at least consider (or better still benchmark) traditional AI and machine learning approaches against LLMs. The traditional approach is almost certainly less computationally expensive and might work better for your problem than an LLM. I recall hearing stories from people on a popular podcast, that they benchmarked and went with the traditional approach because it was more accurate, deterministic, and cheaper. Many had always wanted to try traditional AI/ML, but only got the go ahead recently to add AI to their project from management because of the AI/LLM hype. They were surprised to find that when they benchmarked, the traditional approaches outperformed LLMs both in terms of accuracy and cost. Don't get fixated on LLMs to the point of ignoring older tools that might do a better job.
The big deal is speed (high) and cost (low). This week I’m messing around augmenting mouse and keyboard interactions in an application with Deepgram + System 1 (Jev).
You can talk to an application and have it respond in real time with this combo. It’s clear this kind of general purpose intelligence may be a new development primitive.
However, at this stage, it’s difficult to work with for a few reasons. It’s API only, and you have to shape calling software to the way it communicates.
It’s not clear yet if what we’re missing is a new programming language, or some kind of harness or tool over the capabilities. LLMs were like this early on as well until better harnesses came along and reduced friction in their use.
Down the road, I highly suspect we’ll see:
- Intelligent context assembly using System 1 that summons memories as needed in LLM conversations and handles simple commands.
- System 1 programs that run in the datacenter and reach out to the request initiator on specific instructions like a CPU that has hit a memory barrier.
To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.
And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.
But I still need to experiment with either Jev/SemIf myself.
> To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.
This is a valid point.
> And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.
Literally every inference framework supports constrained encoding. You can make the model choose only from allowed tokens and you can infer only the first diverging token.
It's baffling to me that no inference provider actually exposes this functionality, so you have to run the model yourself to do it.
About jev being a black box and the potential for bias, I think it boils down to what questions you are asking the model.
Broad questions like Is this resume good / score this city will ofcourse be biased but I think jev encourages more granular focused questions like Score this candidates Python experience / Rate this city for its food which then allows you to introduce your own biases in which questions you ask and how you combine their answers.
In this way I think jev like models can be easier to reason about for critical decisions.
I think "Black boxes are back in fashion" is missing the point. I think LLMs are still largely black boxes, and I don't think chain of thought is representative of any degree of inner machination. Asking it questions to justify itself is at best a facsimile, and for the most part it's useful, but it's fundamentally a facsimile.
Where I understand Jev to be a significant jump is that afaik the confidence scoring is actually derived from the normalised probabilities, and not a continuation in a chain of prediction masquerading as "confidence."
I thought this wasn't really an LLM sure text as input but the folks at Typesafe have been very adamant this is a new class of model not just another LLM.
Although to be fair we don't know enough about the architecture.
If you're looking for "Hot Dog/Not A Hotdog" type (i.e., heavily bounded) answers from LLMs, make certain to at least consider (or better still benchmark) traditional AI and machine learning approaches against LLMs. The traditional approach is almost certainly less computationally expensive and might work better for your problem than an LLM. I recall hearing stories from people on a popular podcast, that they benchmarked and went with the traditional approach because it was more accurate, deterministic, and cheaper. Many had always wanted to try traditional AI/ML, but only got the go ahead recently to add AI to their project from management because of the AI/LLM hype. They were surprised to find that when they benchmarked, the traditional approaches outperformed LLMs both in terms of accuracy and cost. Don't get fixated on LLMs to the point of ignoring older tools that might do a better job.
Any idea how they get the probabilities? The probabilities themselves are estimates so knowing the tightness of their distributions will be helpful
I’m not sure I understand the hype around this model. Isn’t this just an llm with a chat template, with the options prefix cached?
Then the llm is constrained to a few special tokens indicating the possibilities? e.g. <option1> <option2>The big deal is speed (high) and cost (low). This week I’m messing around augmenting mouse and keyboard interactions in an application with Deepgram + System 1 (Jev).
You can talk to an application and have it respond in real time with this combo. It’s clear this kind of general purpose intelligence may be a new development primitive.
However, at this stage, it’s difficult to work with for a few reasons. It’s API only, and you have to shape calling software to the way it communicates.
It’s not clear yet if what we’re missing is a new programming language, or some kind of harness or tool over the capabilities. LLMs were like this early on as well until better harnesses came along and reduced friction in their use.
Down the road, I highly suspect we’ll see:
To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.
And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.
But I still need to experiment with either Jev/SemIf myself.
> To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.
This is a valid point.
> And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.
Literally every inference framework supports constrained encoding. You can make the model choose only from allowed tokens and you can infer only the first diverging token.
It's baffling to me that no inference provider actually exposes this functionality, so you have to run the model yourself to do it.
"even Qwen 3.8 has a hard time to consistenly conform to a schema"
idk maybe this is a feature? if some LLM model is wonky, it won't generate right json format, so I'll know for sure
but for Jev, it's 100% right format, so I can't know if it's gone wonky or not
About jev being a black box and the potential for bias, I think it boils down to what questions you are asking the model.
Broad questions like Is this resume good / score this city will ofcourse be biased but I think jev encourages more granular focused questions like Score this candidates Python experience / Rate this city for its food which then allows you to introduce your own biases in which questions you ask and how you combine their answers.
In this way I think jev like models can be easier to reason about for critical decisions.
I think "Black boxes are back in fashion" is missing the point. I think LLMs are still largely black boxes, and I don't think chain of thought is representative of any degree of inner machination. Asking it questions to justify itself is at best a facsimile, and for the most part it's useful, but it's fundamentally a facsimile.
Where I understand Jev to be a significant jump is that afaik the confidence scoring is actually derived from the normalised probabilities, and not a continuation in a chain of prediction masquerading as "confidence."
I thought this wasn't really an LLM sure text as input but the folks at Typesafe have been very adamant this is a new class of model not just another LLM.
Although to be fair we don't know enough about the architecture.
I actually find the name System-1 as a nod to Daniel Kahneman’s Thinking fast and slow book kinda nice. It’s an interesting analogy
This is off topic.
Do people not feel like LLM speak (Claudisms) is infecting their own diction? Saying 'a new "shape" of LLM' sits so very poorly.
Is it 100% deterministic?
Are you asking “is the probabilistic inference model 100% deterministic?”
If so: no.
I've only dabbled, but I thought for a given input, with no previous context and a hotness of zero, LLMs would be deterministic - am I wrong?