Very fascinating, super interesting engineering. Although i do find it very funny how they just bypass a massive vulnerability, basically zero data isolation (even between good actors, let alone bad ones) with 3 sentences. Only in the llm space you can slap a massive limitation like this in the middle of the article and continue like nothing happened
> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.
andy here (headlong post author).
terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them.
w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
Sub Question : IS there a real successful agent product today that uses a library for harness(like langgraph etc)? Building our own worked for us. Works with our components(postgres, events ...) and scales naturally with our system.
I don't know about real successful. Since you mentioned Langchain, you could look at https://www.langchain.com/dcode which is a CLI harness build off Langchain deep agents.
We've also defaulted to just writing our own. We couldn't find anything that was suitable for the use case and had the features we wanted. So when you've got the ability to generate code, why not use it to make what you need?
Of course the don't have exactly the same scopes but they are in general all about persistent memory and / or continous agent loops. Like I miss those times where only once a week a new js framework was promoted.
Very fascinating, super interesting engineering. Although i do find it very funny how they just bypass a massive vulnerability, basically zero data isolation (even between good actors, let alone bad ones) with 3 sentences. Only in the llm space you can slap a massive limitation like this in the middle of the article and continue like nothing happened
> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.
Are there any objective metrics/ benchmarks that people test harnesses by?
There are just so many now that it's hard to personally test them all or just trust the vibes.
andy here (headlong post author). terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them. w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
Sub Question : IS there a real successful agent product today that uses a library for harness(like langgraph etc)? Building our own worked for us. Works with our components(postgres, events ...) and scales naturally with our system.
I don't know about real successful. Since you mentioned Langchain, you could look at https://www.langchain.com/dcode which is a CLI harness build off Langchain deep agents.
We've also defaulted to just writing our own. We couldn't find anything that was suitable for the use case and had the features we wanted. So when you've got the ability to generate code, why not use it to make what you need?
The Googlers must be vague posting about something internal.
It buries the lede. Prime Agent sounds like a very cool project.
I'm just exhausted. So I've today now learned about four new harness:
https://github.com/exoharness/exo/
https://github.com/laude-institute/headlong
https://github.com/microsoft/agent-lightning
and now https://github.com/PrimeIntellect-ai/prime-agent
Of course the don't have exactly the same scopes but they are in general all about persistent memory and / or continous agent loops. Like I miss those times where only once a week a new js framework was promoted.
> "Headlong is a complete agent harness with a core of less than 10K lines of Bash..."
Wow. So, be nice or I'll replace you with a very large shell script?