It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.
It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
But intuitively I think it makes sense that a model can learn what model to route things to if it has all the relevant info, and experimentally it works pretty well in our experience
Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.
Yeah that's a really interesting point, tbh I think the more relevant variable here is the harness you're using rather than the specific model? i.e. GPT 5.5 in the Claude harness behaves a lot more like Claude than Codex if that makes sense.
Hard to quantify this ofc but that's what I've felt vibes wise from using this for the last month.
The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.
How does this router translate to $$$ when developing?
You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase.
This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that point.
I'd say that a typical main agent loop has 1-3 models (obviously very situationally dependent), but when you have subagents those can get routed independently since they have a fresh context window, so there are a lot more degrees of freedom there.
I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.
I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
100%, from what we've seen, for a lot of big companies that 1. don't have subsidized usage and 2. are pushing AI adoption hard, figuring out token costs is P0 or P1 for their eng leadership
So you're saying that since adopting AI/LLM tech many companies have their top engineering priority being optimizing the costs of that rather than ... addressing actual business needs?
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
Indeed it is :) I skipped over talking about all the RL machinery, network design, reward function design, state representations, etc. because really the intuition is that we tell the model when it accomplishes its goal, and then it learns over time how to get better at making the right decisions in order to accomplish its goal.
Happy to talk about this in some more depth if there's anything specific you're curious about!
We integrate with OpenCode too! OpenCode provides the harness, then the router selects the right model for the task.
We haven't yet set up local model routing though, that's really interesting - have you had any success using local models for coding tasks? Tbh I haven't heard many success stories from using local models yet
We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)
Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.
Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.
It saves money because some agent sessions can be entirely handled by a smaller model (also relevant: subagents use fresh context windows so a subagent with a simple task can be routed to a smaller model even if the main agent needs a frontier model).
Totally right about small LLMs btw, that's why we trained this on real agent sessions where we forced it to use different models. If the routing model sees small models can't handle a certain type of task then they won't be assigned. (Also as a fallback we have some guardrails that will have a bigger model come in to "rescue" a smaller model if it gets stuck)
Oh cool, feel free to reach out to me at andrew@workweave.ai if you ever want to share notes! We've learned a lot in the process of building this so far :)
Fun fact: Cursor's "auto" mode is just Composer (or at least it was last time I checked). So it's different in the sense that it actually does route to more than 1 model
If you have a Claude/Codex subscription then we use that (and account for the subsidized price accordingly when making routing decisions) instead of API billing. So you get the best of both worlds: subsidized usage for frontier models + save by using open/smaller models when it's genuinely better.
In practice, lots of ppl are using this to make their Claude sub limits go further!
But we're not routing via `claude -p`, if you have sub usage available + it's the right choice to route to a Claude model, then the router is approximately a transparent passthrough. So it gets billed like normal `claude` usage rather than `claude -p`.
If you statelessly route each new request: yes it does end up being more expensive!
So our routing is cache-aware. It will have a much higher threshold to switch from one model to another if there's already some cache for the first model. Experimentally this solves the problem (like I said we've saved 40% ourselves vs. what we would have otherwise paid).
It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.
It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
Yep cache awareness is super important, mentioned this in another thread here: (https://news.ycombinator.com/item?id=48689448)
But intuitively I think it makes sense that a model can learn what model to route things to if it has all the relevant info, and experimentally it works pretty well in our experience
Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.
Yeah that's a really interesting point, tbh I think the more relevant variable here is the harness you're using rather than the specific model? i.e. GPT 5.5 in the Claude harness behaves a lot more like Claude than Codex if that makes sense.
Hard to quantify this ofc but that's what I've felt vibes wise from using this for the last month.
Yep this was always the reason to avoid "auto" mode in cursor.
The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.
How does this router translate to $$$ when developing?
You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase.
This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
[delayed]
That is interesting, sounds like in practice you only end up routing between 2 models
Or not routing at all.
In practice you just pick one and stick with it until the API stops or you hit performance issues.
The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that point.
I'd say that a typical main agent loop has 1-3 models (obviously very situationally dependent), but when you have subagents those can get routed independently since they have a fresh context window, so there are a lot more degrees of freedom there.
I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.
I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
How come data privacy and confidentiality is not an issue with services like these?
Do people voluntarily use these proxies/routers, knowing their prompts, outputs and code will be seen by other people ?
I get it might be ok for personal projects, but for anything that makes money and is a part of business... this must be big no-no ?
It is a router that runs locally.
This + making sure common requests are saved as reusable skills and scripts would probably save a large part of my token usage
As prices increase we will see more of these tools to optimise and make the best use of token budget
100%, from what we've seen, for a lot of big companies that 1. don't have subsidized usage and 2. are pushing AI adoption hard, figuring out token costs is P0 or P1 for their eng leadership
So you're saying that since adopting AI/LLM tech many companies have their top engineering priority being optimizing the costs of that rather than ... addressing actual business needs?
I guess delivering business value is always #1, I just meant it's the biggest problem they're trying to solve. Here's a recent example that was public: https://fortune.com/2026/05/26/uber-coo-ai-spending-tokens-c...
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
Yep 100%, mentioned this in another thread (https://news.ycombinator.com/item?id=48689448) but tl;dr we build the router to be cache aware
"We reward the routing model when it selects an LLM that achieves the task successfully" sounds pretty oversimplified
Indeed it is :) I skipped over talking about all the RL machinery, network design, reward function design, state representations, etc. because really the intuition is that we tell the model when it accomplishes its goal, and then it learns over time how to get better at making the right decisions in order to accomplish its goal.
Happy to talk about this in some more depth if there's anything specific you're curious about!
I would rather just use OpenCode - leverage AI models, even can host locally or paid ones with ease.
We integrate with OpenCode too! OpenCode provides the harness, then the router selects the right model for the task.
We haven't yet set up local model routing though, that's really interesting - have you had any success using local models for coding tasks? Tbh I haven't heard many success stories from using local models yet
This is cool!
Will this use my Claude Pro/Max subscription? Or will it always use the API billing "pay as you go"?
Yep it uses the Claude sub if possible and falls back to API billing only if you don't have a Claude sub or it's out of usage! Same deal for Codex
We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)
1. https://github.com/instavm/murmur - Murmur
Very interesting - curious how you've used it yourself so far? I can imagine one use case would be having e.g. GPT 5.5 review Opus 4.8's work?
Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.
Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.
edit: spellcheck
It saves money because some agent sessions can be entirely handled by a smaller model (also relevant: subagents use fresh context windows so a subagent with a simple task can be routed to a smaller model even if the main agent needs a frontier model).
Totally right about small LLMs btw, that's why we trained this on real agent sessions where we forced it to use different models. If the routing model sees small models can't handle a certain type of task then they won't be assigned. (Also as a fallback we have some guardrails that will have a bigger model come in to "rescue" a smaller model if it gets stuck)
It is funny. We are building something similar.
Oh cool, feel free to reach out to me at andrew@workweave.ai if you ever want to share notes! We've learned a lot in the process of building this so far :)
What is the difference from Cursors 'auto' mode?
Fun fact: Cursor's "auto" mode is just Composer (or at least it was last time I checked). So it's different in the sense that it actually does route to more than 1 model
but this means you work with API pricing rather than subscription pricing. Isn’t it better to use claude or codex CLI etc directly in terms of cost?
If you have a Claude/Codex subscription then we use that (and account for the subsidized price accordingly when making routing decisions) instead of API billing. So you get the best of both worlds: subsidized usage for frontier models + save by using open/smaller models when it's genuinely better.
In practice, lots of ppl are using this to make their Claude sub limits go further!
I see but didn’t they severely limited the usage allowed with `claude -p`
But we're not routing via `claude -p`, if you have sub usage available + it's the right choice to route to a Claude model, then the router is approximately a transparent passthrough. So it gets billed like normal `claude` usage rather than `claude -p`.
> At Weave, we write ~all our code with AI
This is probably not a very effective way of marketing imo. At least, it turns me completely off.
Isn't this more expensive than always using the same model, since, as I understand, by routing to different models you give up on cache?
If you statelessly route each new request: yes it does end up being more expensive!
So our routing is cache-aware. It will have a much higher threshold to switch from one model to another if there's already some cache for the first model. Experimentally this solves the problem (like I said we've saved 40% ourselves vs. what we would have otherwise paid).
This is basically what I need, a router. I’m tired of changing intelligence & speed levels manually.
Nice, let me know any feedback you have from trying it out!