Good model. Anybody interested in actually training models or designing agentic systems should be doing this.
My company started around working on this problem because it's the basis for how you train programming models/reliably deploy LLMs to do specific tasks. It allowed me to build a much better mental model for LLMs because I saw how weirdly fickle/inconsistent/picky they could actually be outside of a "chat" where it feels like they have a coherent persona or consistent knowledge/capability.
Initially I thought of it as a search over prompts for capability at completing specific tasks, but now I think the speed/reliability and operations (eg can I switch models without degrading perforamnce?) benefits are even bigger benefits for most users.
A little "secret" since labs are making it harder to even use their models in this way and it's important that it be more widely understood: distribution-aware replay/re-sampling is a key technique in post-training LLMs. But it's also something that allows you to automatically identify the best model for some subset of your tasks, which can save you a lot of money.
I’ve been building a durable execution framework with roughly the same mechanism. It’s a pretty logical rough edge to try shave off from Temporal-like systems.
Durable execution looks a bit like a buzzword in general. If your state is defined in the execution graph then it's just an umbrella term for a group of pre-existing algorithms and patterns. If it's undefined then what are you resuming to? The snapshot just before the crash likely leads to the undefined state again, in which case you're durably automating the crash (or even worse, uncaught incorrect behavior).
Our architecture has had somewhat of a different primitive for years that yields the same outcome.
Our application architecture is named The Feature Architecture.
A unit of work is feature. A lot of common implementation can be derived from (or compressed into) the name of a feature. They are invoked by Features.invoke(feature_name,...) and seamlessly calls same service, service to service or frontend to backend service or even backend to frontend (with client ID and userid) seamlessly.
A command feature by default does durable execution by being a consumer as well as a feature as the machinery ensures all incoming command feature messages are injected into monolog (in house built akin to kafka). So command feature errors are retried by the machinery by default.
All Dip operations are idempotent and hence can be retried maximally.
Our arcc (The architecture compiler) enforces at compile time that
1) there are zero CQRS violations of query to command invocations
2) zero violations of query to Dip.insert/update/remove (extended CQRS for persistence)
3) all Dip mutate operations are idempotent (arcc --strict),
4) zero alien Dip collection access (a feature leaf and its handlers owns exactly one collection)
5) and a whole myriad of around 10 different architecture rules that usually depend on developer discipline and conventions.
One can set a command operation to be not a durable execution by specifying bypass: true for that feature in the registry but those are the outliers.
So durable executions are inherently native and first class for all commands in our Feature Architecture.
dropping history replay would remove so much operational pain from durable workflows. how do you reconstruct in-flight state after a crash, snapshotting or something event sourced
Good model. Anybody interested in actually training models or designing agentic systems should be doing this.
My company started around working on this problem because it's the basis for how you train programming models/reliably deploy LLMs to do specific tasks. It allowed me to build a much better mental model for LLMs because I saw how weirdly fickle/inconsistent/picky they could actually be outside of a "chat" where it feels like they have a coherent persona or consistent knowledge/capability.
Initially I thought of it as a search over prompts for capability at completing specific tasks, but now I think the speed/reliability and operations (eg can I switch models without degrading perforamnce?) benefits are even bigger benefits for most users.
A little "secret" since labs are making it harder to even use their models in this way and it's important that it be more widely understood: distribution-aware replay/re-sampling is a key technique in post-training LLMs. But it's also something that allows you to automatically identify the best model for some subset of your tasks, which can save you a lot of money.
I’ve been building a durable execution framework with roughly the same mechanism. It’s a pretty logical rough edge to try shave off from Temporal-like systems.
Congratulations.
I am curious about what prompted you to build a durable execution framework.
We didn't know we were building one until later as that is where our application development trajectory naturally lead us.
Durable execution looks a bit like a buzzword in general. If your state is defined in the execution graph then it's just an umbrella term for a group of pre-existing algorithms and patterns. If it's undefined then what are you resuming to? The snapshot just before the crash likely leads to the undefined state again, in which case you're durably automating the crash (or even worse, uncaught incorrect behavior).
Snapshotting programs is also what https://github.com/pydantic/monty enables and aims for.
FWIW, I think we'll see a rise of AI-ready interpreters. In some sense, I like that it challenges traditional microservice architectures as an aside.
Nice.
Our architecture has had somewhat of a different primitive for years that yields the same outcome.
Our application architecture is named The Feature Architecture.
A unit of work is feature. A lot of common implementation can be derived from (or compressed into) the name of a feature. They are invoked by Features.invoke(feature_name,...) and seamlessly calls same service, service to service or frontend to backend service or even backend to frontend (with client ID and userid) seamlessly.
A command feature by default does durable execution by being a consumer as well as a feature as the machinery ensures all incoming command feature messages are injected into monolog (in house built akin to kafka). So command feature errors are retried by the machinery by default.
All Dip operations are idempotent and hence can be retried maximally.
Our arcc (The architecture compiler) enforces at compile time that 1) there are zero CQRS violations of query to command invocations 2) zero violations of query to Dip.insert/update/remove (extended CQRS for persistence) 3) all Dip mutate operations are idempotent (arcc --strict), 4) zero alien Dip collection access (a feature leaf and its handlers owns exactly one collection) 5) and a whole myriad of around 10 different architecture rules that usually depend on developer discipline and conventions.
One can set a command operation to be not a durable execution by specifying bypass: true for that feature in the registry but those are the outliers.
So durable executions are inherently native and first class for all commands in our Feature Architecture.
Kind of reminds me of how redis forks itself to have a snapshot to persist as a backup.
dropping history replay would remove so much operational pain from durable workflows. how do you reconstruct in-flight state after a crash, snapshotting or something event sourced
Lamport & Chandy, right?