How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.
To answer your question - we use cheaper models to just flag if something looks obviously flawed. That covers the vast majority of things because it's the propagation of obviously stupid stuff that causes a problem for our system. To the extent we are doing something sophisticated, we use an appropriate architecture, like it's built into the system.
There are all manner of documented architectures in papers and books, I don't know that there is anything new under the sun at this point, the devil is always in the details. That said, some of the things we use off the top of my head: generate multiple responses using different models and have a third model judge the best answer; generate an answer and have another model critique the answer and then a third take the answer and critique and generate a new answer; multi-model debate; iterate in a loop either x times or until some condition is satisfied; pipelines of specific single calls to check for specific conditions.
I'd say in general, with respect to errors/low quality responses etc you have to be extremely clear about what you are doing because once you get into the weeds, what is and is not an error is very grey in many cases.
Interesting. Seems to me like latency is not critical in your application and you are doing something more like background processing rather than serving humans on the other side in the real time.
Also, some of what you describe sounds expensive. Do you find these techniques + cheaper models better than single-shot with larger models?
Yes, it can get expensive. We do something in financial markets where the cost of a single end to end task could be $100. That doesn't mean cost doesn't matter though, it matters a lot - if we screw up and it propagates, it's money down the toilet. And if we could get the task down to $20, it would make a difference.
Very cool. I liked the high level overview where you can view it like do a funnel analysis for failures. Good for fast decision making.
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
>for creating traces from many test calls to speed up testing of voice bots
I assume you mean tracing voice agents in general. If so, it depends on the where do you want to trace things. Right now, Laminar, unfortunately, doesn't trace voice SDKs like OpenAI's realtime, but we're very actively looking into it.
However, if somewhere in your pipeline you have transcripts of your voice agents, then it shouldn't be that different from tracing any other agent. In this case, Laminar and Signals work perfectly.
We also trained and benchmarked flow-1 on a lot of multi-turn voice agent like conversations.
Congrats on the product! I love the design of the website and the app. Curious whether it was made in-house, by an external design agency, with AI models, or something else
How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.
To answer your question - we use cheaper models to just flag if something looks obviously flawed. That covers the vast majority of things because it's the propagation of obviously stupid stuff that causes a problem for our system. To the extent we are doing something sophisticated, we use an appropriate architecture, like it's built into the system.
Thanks for the response! Curious, can you expand on the architecture, how does that work? If you can share of course.
There are all manner of documented architectures in papers and books, I don't know that there is anything new under the sun at this point, the devil is always in the details. That said, some of the things we use off the top of my head: generate multiple responses using different models and have a third model judge the best answer; generate an answer and have another model critique the answer and then a third take the answer and critique and generate a new answer; multi-model debate; iterate in a loop either x times or until some condition is satisfied; pipelines of specific single calls to check for specific conditions.
I'd say in general, with respect to errors/low quality responses etc you have to be extremely clear about what you are doing because once you get into the weeds, what is and is not an error is very grey in many cases.
Interesting. Seems to me like latency is not critical in your application and you are doing something more like background processing rather than serving humans on the other side in the real time.
Also, some of what you describe sounds expensive. Do you find these techniques + cheaper models better than single-shot with larger models?
Correct, latency is not critical.
Yes, it can get expensive. We do something in financial markets where the cost of a single end to end task could be $100. That doesn't mean cost doesn't matter though, it matters a lot - if we screw up and it propagates, it's money down the toilet. And if we could get the task down to $20, it would make a difference.
Very cool. I liked the high level overview where you can view it like do a funnel analysis for failures. Good for fast decision making.
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
>for creating traces from many test calls to speed up testing of voice bots
I assume you mean tracing voice agents in general. If so, it depends on the where do you want to trace things. Right now, Laminar, unfortunately, doesn't trace voice SDKs like OpenAI's realtime, but we're very actively looking into it.
However, if somewhere in your pipeline you have transcripts of your voice agents, then it shouldn't be that different from tracing any other agent. In this case, Laminar and Signals work perfectly.
We also trained and benchmarked flow-1 on a lot of multi-turn voice agent like conversations.
Congrats on the product! I love the design of the website and the app. Curious whether it was made in-house, by an external design agency, with AI models, or something else
Thanks! Our design is all done completely in-house.
Workflows are constantly changing but our design process happens across Figma, old-fashioned paper, and code.
You should try the bullet points then earning someone reading the rest, that's where I bounced.
I can write some bullet points if you want. I'm serious, this is not a sarcasm. With which AO platform do you want a comparison?