A lot of the crazy ideas seem to have melted away in the face of massive context sizes. Today, I can put roughly a megabyte of utf8 text into my system prompt before things start to get weird.
That is a massive amount of information even if we are being sloppy with it. You can read The Hobbit and the first Harry Potter book cover-to-cover and still have room to spare. I would deeply struggle to develop a world model this detailed for any business. Anything that needs to get more specific than these narratives can be a SQL query tool into the data warehouse, grep over the codebase, MS graph API lookup, etc.
Giving the business a balanced way to collaborate over this one shared model of the world is a new challenge I am beginning to engage with. I've also noticed that the world model will compound on itself in terms of self-detection of update opportunities. The more constraints there are, the more likely we appear to violate one.
Fable changes the game yet again, because it's API-only.
You're not likely to want to run Fable in a loop any more than you want to take a bunch of dollar bills and light them on fire. Every invocation of Fable has to be intentional, its context carefully managed. I feel like a babysitter.
I agree with you that you don't need fable for everything, and you have to be careful on what you run it on. CRUD stuff, sure even the small models can do it. But there certainly are tasks that are very much suited for the absolute SotA and you'd leave money on the table by not using it. And how much a task is worth is dependant on how much it improves your bottom line. So the cost/token becomes largely irrelevant.
Let's take this [1] benchmark. A bit more context here [2].
Here models are asked to create kernels for running inference on models. This is a benchmark perfectly suited and highly relevant right now. It's easily verifiable, an active are of research, and the results are immediately useful.
Say you have 1 unit of compute, it costs 300k $ and serves 1x users. In comes Fable and after one session it gives you 30% speed-up on your 1 unit of compute. It can now serve 1.3x users. How much is that one session worth for you? How much is it worth for a company using 10 units? 100 units? How much is it worth for a hyper-scaler running 10.000 units? How much is it worth for a lab that trains the next frontier model and then serves it from 100.000 units? 30% is relative. And the cost for one session is really meaningless. It can cost 1m$ / session and it would still be worth it for someone.
> You're not likely to want to run Fable in a loop any more than you want to take a bunch of dollar bills and light them on fire. Every invocation of Fable has to be intentional, its context carefully managed.
Eh, that's just because it's the current frontier model. Give it a few weeks, and prices will drop.
API prices are the new normal. I doubt that prices will drop to the level of the subsidized subscriptions any time soon. Usage is growing exponentially but capacity cannot. There is no reason for them to waste their capacity on subscription users if they can sell that same capacity to API users.
Like with Uber and Lyft, the low prices were a fight for market share, but now they have successfully captured that market share the focus changes to balancing their books.
A lot of the crazy ideas seem to have melted away in the face of massive context sizes. Today, I can put roughly a megabyte of utf8 text into my system prompt before things start to get weird.
That is a massive amount of information even if we are being sloppy with it. You can read The Hobbit and the first Harry Potter book cover-to-cover and still have room to spare. I would deeply struggle to develop a world model this detailed for any business. Anything that needs to get more specific than these narratives can be a SQL query tool into the data warehouse, grep over the codebase, MS graph API lookup, etc.
Giving the business a balanced way to collaborate over this one shared model of the world is a new challenge I am beginning to engage with. I've also noticed that the world model will compound on itself in terms of self-detection of update opportunities. The more constraints there are, the more likely we appear to violate one.
This seems like the beginnings of AI psychosis, tbh.
It's "Galapagos" or "Galápagos," not "Galapogos."
You miswrote OP’s miswriting in the third version :)
Argh, autocorrect got me. Thanks, fixed.
Aaand now OP has fixed it in the HN post title. Still wrong in the linked article.
Fable changes the game yet again, because it's API-only.
You're not likely to want to run Fable in a loop any more than you want to take a bunch of dollar bills and light them on fire. Every invocation of Fable has to be intentional, its context carefully managed. I feel like a babysitter.
Compared to Opus 4.8 I really haven't been impressed
I agree with you that you don't need fable for everything, and you have to be careful on what you run it on. CRUD stuff, sure even the small models can do it. But there certainly are tasks that are very much suited for the absolute SotA and you'd leave money on the table by not using it. And how much a task is worth is dependant on how much it improves your bottom line. So the cost/token becomes largely irrelevant.
Let's take this [1] benchmark. A bit more context here [2].
Here models are asked to create kernels for running inference on models. This is a benchmark perfectly suited and highly relevant right now. It's easily verifiable, an active are of research, and the results are immediately useful.
Say you have 1 unit of compute, it costs 300k $ and serves 1x users. In comes Fable and after one session it gives you 30% speed-up on your 1 unit of compute. It can now serve 1.3x users. How much is that one session worth for you? How much is it worth for a company using 10 units? 100 units? How much is it worth for a hyper-scaler running 10.000 units? How much is it worth for a lab that trains the next frontier model and then serves it from 100.000 units? 30% is relative. And the cost for one session is really meaningless. It can cost 1m$ / session and it would still be worth it for someone.
[1] - https://kernelbench.com/mega
[2] - https://x.com/elliotarledge/status/2072814573753975266
For now, I can use Fable from the web just fine.
> You're not likely to want to run Fable in a loop any more than you want to take a bunch of dollar bills and light them on fire. Every invocation of Fable has to be intentional, its context carefully managed.
Eh, that's just because it's the current frontier model. Give it a few weeks, and prices will drop.
API prices are the new normal. I doubt that prices will drop to the level of the subsidized subscriptions any time soon. Usage is growing exponentially but capacity cannot. There is no reason for them to waste their capacity on subscription users if they can sell that same capacity to API users.
Like with Uber and Lyft, the low prices were a fight for market share, but now they have successfully captured that market share the focus changes to balancing their books.