> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.
To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
Same. Mistral 7b has been more than I ever needed for text for years now.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.
Probably. You can solve it with either some grounding context, or spending hundreds or thousands a month extra on a model that has more knowledge baked in. With modern harnesses, the choices is obvious.
Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
> Across his various startups, Peter has seen two kinds of work:
> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.
> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back
I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.
I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
Exactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it. People are addicted to chasing frontier models because they have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough.
> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling.
Small is relative. I'm looking for models that I can with run around 100 MiB mark (RAM just for the weights) to demo what you can do with this little memory.
I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.
Is there anything newer in this category? Or should I try to distill something down to this size?
I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation
24/7 (then again I might be wrong). Exciting times.
IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
100% agreed. Small, cheap, and hosted models. Luna (and open weight models and others) is ridiculously cheap @ $0.2/$1.2, easily accessible, and more than good enough for basic use cases (e.g. summarization, simple tool calling, etc.).
The tokens per second speed measurement is highly inflated nowadays because most of the tokens went into thinking. I wonder if there is a more realistic measurement for "effective speed", which accounts for thinking efficiency.
Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local models" in the cloud providers like DigitalOcean, AWS, etc.
I only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally.
That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.
I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
I am very excited that more makers will come up with fast memory for consumers rather than enterprise. Companies can only pre order so much RAM.
At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time.
You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
I forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.
I have trouble seeing the points of using less capable models.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.
It depends on what you're trying to do. For non-coding tasks luna is quite often enough. Flash models are more than enough for summarizing a text, for example, or whipping up a small script to save me fifteen minutes. If you're on a 200/month plan, I see your point. If you're on a dollar limit - or worse, paying per token out of your pocket - you look to be more efficient.
If you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.
Some reasons:
- Smaller models will always be cheaper
- Smaller models will always use less energy, therefore better for the environment
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.
To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
"with multiple 3090s" is quite a bit of burying the lede for "most local models work just fine", don't you think?
Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
A single, used 3090 costs more than I have ever spent on a computer.
Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.
My single 3080 runs so hot I don't need to warm my room in winter, and have to play games in my underwear in summer.
While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
Same. Mistral 7b has been more than I ever needed for text for years now.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
What are some limitations you have found with using a smaller model like that?
Sounds interesting. Can you give more details on your workflow and what tasks you use it for?
Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work.
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
Running small models locally beats wrestling with API latencies and rate limits. The compute trade-off is 100% worth the privacy and DX gains.
It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.
Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.
Only if we require the knowledge to be built into the weights. Give it access to a search engine and a big library of ebooks, and it might do better.
Probably. You can solve it with either some grounding context, or spending hundreds or thousands a month extra on a model that has more knowledge baked in. With modern harnesses, the choices is obvious.
Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
I think the context here is that small models run locally, not rented from a cloud.
> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html
I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back
I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.
I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
Exactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed
Exactly. Even 32b parameter models you can run locally on consumer hardware are "good enough" at this point for some workflows!
> for most tasks
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it. People are addicted to chasing frontier models because they have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough.
"Small models" nowadays work like someone who has IQ 100+ while SOTA ones are like 150, "relatively".
Given sheer number of turns I can make with small models, I can do a lotta stufff
- cheaper, and faster
Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses.
These "small" models with right context, and guidance, they work wonders.
---
I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash.
- https://news.ycombinator.com/item?id=49450353#49452248
> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
Have you personally read it for 5 years to determine this? If not, how could you possibly hold this position?
> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling.
This is a good insight broadly!
Small is relative. I'm looking for models that I can with run around 100 MiB mark (RAM just for the weights) to demo what you can do with this little memory.
I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.
Is there anything newer in this category? Or should I try to distill something down to this size?
I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation 24/7 (then again I might be wrong). Exciting times.
IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
100% agreed. Small, cheap, and hosted models. Luna (and open weight models and others) is ridiculously cheap @ $0.2/$1.2, easily accessible, and more than good enough for basic use cases (e.g. summarization, simple tool calling, etc.).
The demand for fast, cheap, good enough models has always been borderline infinite, it’s the supply that’s going to take off.
The tokens per second speed measurement is highly inflated nowadays because most of the tokens went into thinking. I wonder if there is a more realistic measurement for "effective speed", which accounts for thinking efficiency.
Many benchmarks now measure the total cost or energy usage per completed task.
Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local models" in the cloud providers like DigitalOcean, AWS, etc.
You should be glad to know digital ocean already offers this
I only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally.
That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.
I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
I am very excited that more makers will come up with fast memory for consumers rather than enterprise. Companies can only pre order so much RAM.
At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time.
You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
Segment had a $3B+ exit to Twilio back in 2020.
I forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.
small models + a good application layer - are more than enough, good for routine business tasks.
the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.
Can you elaborate on this?
I think we'll see more of this soon
replit is already leading the way with free luna usage
I like the analogy on ways to make small model useful.
Well said
I have trouble seeing the points of using less capable models.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.
It depends on what you're trying to do. For non-coding tasks luna is quite often enough. Flash models are more than enough for summarizing a text, for example, or whipping up a small script to save me fifteen minutes. If you're on a 200/month plan, I see your point. If you're on a dollar limit - or worse, paying per token out of your pocket - you look to be more efficient.
First, smaller models are fun for hackers: you can run them locally, or run them faster.
Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
If you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.
You misunderstood what I was referring to by “hacker” there.
Can you define your use of the word 'smartest' here just in case some of us don't quite know what you mean?
Some reasons: - Smaller models will always be cheaper - Smaller models will always use less energy, therefore better for the environment
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
Cost is the point