"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors.
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
it was always a signal of low information (or worse) by whoever espoused it.
after all, you are just a next neural impulse predictor.
the real meat is in the capacity and complexity of the underlying system. it would not be surprising if for example in the future we will have an efficient learning algorithm that bootstraps AGI from just the human text with no post-training whatsoever (would be a scientific curiosity at that point). or how a single well designed headless computer game could provide a RL environment for efficient bootstrap of cunning and consciousness.
note how current learning algorithms (gradient descent) are massively sample inefficient when compared to a human brain. a single innovation in that space has the potential to upend the entire current scaling assumptions behind AI.
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence)
2. Cow (85% confidence)
3. Bag (75% confidence)
4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
> But for many purposes you will not find it helpful to think of human beings as bags of chemicals
But when we talk about humans, we're not talking about the chemicals involved in those humans.
When we talk about LLMs, the tokens are the valuable thing they produce for us. We want LLMs because they give us sequences of tokens.
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
100%
But it is a next token predictor.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
Good example.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
We also can't prove that our minds aren't machine elf meat puppets. Come on. Please.
> It’s true
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
Next token prediction is just an interface. It can be backed by a Markov chain, a neural model or an actual human being.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
That's literally what LLMs are.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
yes, it's exactly this. And it's not a trivial distinction.
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
A better statement might be:
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.Nope. Post training means even the raw model isn't predicting.
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
it was always a signal of low information (or worse) by whoever espoused it.
after all, you are just a next neural impulse predictor.
the real meat is in the capacity and complexity of the underlying system. it would not be surprising if for example in the future we will have an efficient learning algorithm that bootstraps AGI from just the human text with no post-training whatsoever (would be a scientific curiosity at that point). or how a single well designed headless computer game could provide a RL environment for efficient bootstrap of cunning and consciousness.
note how current learning algorithms (gradient descent) are massively sample inefficient when compared to a human brain. a single innovation in that space has the potential to upend the entire current scaling assumptions behind AI.
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK
(Start at 15:30 for the LLM specific parts)
During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.
So, next-token predictors
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
classical particles obey deterministic state evolution rules, yet put 10 in a box and you cannot say where they will be 5 minutes later.
It knows nothing of correctness or winning. It is predicting only what is most likely given its corpus.
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
Better title: Continue thinking of LLMs as Next-Token Predictors
Because no, post training doesn't change that.
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
slop slop and more slop