If you're like me, a SWE who is curious about ML/LLM training but unfamiliar with the terms, I got an agent to explain to me how to read the charts.
Basically, you can think of a LLM as a function which generates a probability distribution of words. If the next word in a series is "they", and one model predicts that word 40% of the time, and another model predicts that word 1% of the time, the latter model is worse as it is more surprised by the true distribution.
You can convert these probabilities into "bits":
surprise in bits = −log₂(probability of the actual token)
Probability of actual token,Surprise
1,0 bits
1/2,1 bit
1/8,3 bits
1/1024,10 bits
This is then normalised by text length:
Bits per byte = total next-token surprise in bits / number of bytes in the evaluated text
So the lower you go on the charts, the less surprises in the LLMs distribution (a better model).
Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa
If you're like me, a SWE who is curious about ML/LLM training but unfamiliar with the terms, I got an agent to explain to me how to read the charts.
Basically, you can think of a LLM as a function which generates a probability distribution of words. If the next word in a series is "they", and one model predicts that word 40% of the time, and another model predicts that word 1% of the time, the latter model is worse as it is more surprised by the true distribution.
You can convert these probabilities into "bits":
surprise in bits = −log₂(probability of the actual token)
Probability of actual token,Surprise 1,0 bits 1/2,1 bit 1/8,3 bits 1/1024,10 bits
This is then normalised by text length:
Bits per byte = total next-token surprise in bits / number of bytes in the evaluated text
So the lower you go on the charts, the less surprises in the LLMs distribution (a better model).
For more info: https://smalldocs.org/s/DfvdGuFsiR3LlzXw1H5J0K#k=AJ8V1AQECYj...
> We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200.
If this holds up that's a really big deal.
Huge if true. As it were.
Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa
Imagine the sheer amount of power you could save by releasing the paper.
But thanks to the Jevon Paradox, the global power consumption would probably increase.
https://en.wikipedia.org/wiki/Jevons_paradox
Cool story. If it's true the company will be bought by open AI/Anthropic and Chinese labs will discover the trick and open source it by next quarter.