This is not an LLM obviously
, it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.
It's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson.
This is not an LLM obviously , it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.
Isn't a MoE model basically a cascading tree of smaller models or some variation of that?
Is token rate a function of parameter size?
And the 5 years old AMD Ryzen 5 5600H is doing 7M?
Am I reading this right? Then I need to try this on Strix Halo
Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.
It's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson.
https://en.wikipedia.org/wiki/Bitter_lesson
Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.