When i wrote my own harness, i didn't rely on what other agents do, but rather sat down and considered how i can solve the issues that i have when using other agents. And tbh, for alot of problems there are rather "simple" fixes - each comes with a cost tho. For me it was accepting to run more inference than "standard" agents but in return solving alot of issues those agents have. And to me having a full autonomous agent that i can task and just come back hours later to a solution is much more worth than just an agent that i need to "prompt jokey" the whole day.
I think its important that people don't just look at whats implemented rn, but rather just try out new things , not just "adjust" the existing ones. Thats how we got better progress, not by trying to build the 1000s variant of context compression but rather reconsider how can context be build to not have this issue (simple example).
Yes, it's so easy to build a harness for a lot of use cases now, and you learn so much from doing so.
I'm a solo developer, and I didn't want to sort out all the boundaries and permissions of Claude or other heavy harnesses. All I wanted was something that can read, write, and modify files in a project directory. I gave myself three days to see if I could make a harness that was complete enough to work on my own projects. It was usable in less than a day.
I love using it, because I know exactly what its boundaries are, and how they're enforced. When I need more features I can either implement them in my own harness, or choose a heavier harness with much clearer understanding of what to look for.
Harness optimization is essential. Doesn’t matter how smart the model is if the tools you give it are ineffective, or the task is under defined. It needs invent the solution from first principles every time.
The LLM is the oven, the harness is the kitchen. You prep all your ingredients and put it into the oven (flour, water, salt, sugar). It transforms those inputs into a finished product (cake, loaf of bread).
Nah, that's a single event ("bake"). Interaction of a harness and an LLM could potentially be a continuous process.
I don't think we've scratched the surface of how the harness can fold, spindle and mutilate the LLM. It just seems unlikely to me that we happened to stumble on the best possible interface a couple of years out of the gate.
Instead of reading AI-slop explanations full of creaky metaphors, I'd advise you to just build your own basic coding harness from scratch. You will learn so much:
1) This old but still excellent blog post by Thorsten Ball is a good place to start (old in LLM years... written 18 months ago!): https://ampcode.com/notes/how-to-build-an-agent . It is written by a human, and gets across the core message really well. If you don't enjoy reading Go then just use an LLM to translate the samples into Python code or whatever.
2) Instead of relying on an api key like in the blog post in 1), you might want to use a local model that can do tool calls like Qwen 27B.
3) Use a frontier LLM for self-study and to help you write the harness boilerplace if you want, but ask it to force you to write the interesting bits by hand (e.g. the main loop structure, the tool call functions). Yes, coding by hand is still useful as an educational mechanism to force you to internalize some tricky concept. Be disciplined and don't slip into letting the LLM write the core logic for you.
4) Amaze your friends and colleagues with your newfound understanding of how agents actually work under the hood
5) Go even deeper and get into Inference engineering, LLM architecture internals, etc.
6) Profit. Perhaps? (Or at least enjoy the learning journey.)
Can’t recommend making your own harness enough. Even a trivial toy harness will clarify a lot of concepts that I see people confuse all the time (especially here on HN).
I made a simple video explaining how to make a simple harness in JS running in a browser, having it build it self up from running purely in the browser console to being interactive in the page to making downloadable copies of itself. Some people have found it super helpful.
I mean, it can be that way. But it does't have to be. The Harness can be as much the brain as the LLM. You have nearly unlimited flexibility in what you do in the harness, and you can add as much of the overall "intelligence" of the system as you want in the harness.
To illustrate, you can:
1. Use multiple "brains" / LLM's, and do routing between them using a specialized model (a Jev like model, or something of your own design)
2. Use planning tools like pyperplan, jupyddl, pddl4j, etc.
3. Implement your own "memory" system using... pretty much anything you want. And I would argue that a lot of the apparent "intelligence" of a system involves how well memory works
4. Call classical machine learning models to solve specialized problems
5. Use reinforcement learning to improve performance in specific areas
6. Implement a blackboard for multiple knowledge sources to collaborate.
7. Use inference engines using forward/backward chaining, RETE, etc.
8. Use your own hand-rolled abductive inference strategy
9. Implement your own Bayesian reasoning techniques
10. Use custom causal inference code
... and so on.
Don't get me wrong. I'm bullish on what LLM's can do, but I posit that the overall expressed intelligence of a system can involve a lot more than just the LLM. In fact, I'd actually make the slightly stronger claim that, depending on what you're trying to do, an LLM might not even be required at all.
If LLMs were brains, it would be someone with severe amnesia who you had to tell them what an arm and a leg were every time. Its stored vectors of words or partial words and computations of math that run to predict the next words/partials to use to form text output. It will nearly always find the real sets of words that are load bearing to put into its output (seriously how many times do real and load bearing appear in the models, can we remove some of that).
What's interesting is most of the progression we've seen this last year in AI agent clients are very basic computer science concepts: loops, threads, arrays (of JSON) and memory management (aka context) with varying combinations of those put together and people in computing are writing about it like its some huge break through. I would love to hear a game developers perspective on comparing AI clients (not the server side LLMs/vectors/maths) with games - they seem to have very similar computing problems.
A better analogy: the harness is a harness, the LLM is the belayer. You are the climber. Except you’re barely climbing. The LLM is helping you hangdog your way up a mountain you have no business climbing. And when you get to the top you’ll feel very accomplished but you did nothing really. The LLM pulled you up by brute force.
I don’t understand why this article makes you ask that, can you elaborate? The simple answer to why we didn’t have LLMs in 2005 is that nobody had the requisite ideas that make them work effectively (independent of any agentic harnesses and Schmidhuber notwithstanding) and there wasn’t sufficient clarity in the direction to scale for anyone to build anything at scale (also a requisite for them working well).
Surely the only problem isn’t the knowledge of how to train and the confidence it would work. Wouldn’t it also have required a 1 or 2 order of magnitude increase in computing power over 2005-era supercomputers to be able to train a modern LLM in a reasonable time (“reasonable” meaning “finish training before GPT-3 comes out in 15 years”)?
I think it’s plausible that attention could have been invented earlier and GPT demonstrated at smaller scale on earlier hardware. Then hardware timelines might have accelerated (or alternative approaches explored) and we’d have gotten to where we are earlier.
NVIDIA moved as fast as possible back then, there were huge performance gains with each new GPU generation, hard to imagine they might have gone even faster.
Fair on the prose. I explained how it was written in my top-level comment. If any of the ideas themselves seem wrong to you, I'd like to hear which ones.
Looks like HN hid my top-level comment because I'm new here.
Anyway, I'm the author. I'm not a native speaker, so I used Claude to polish up the writing. I just wanted to write down 3 years of agents/harness building experience.
I'm not OP, and it's not quite three years ago, but Devin was launched in 2024. I knew several other people who had created similar prototypes around that time. The ReAct loop was already well known by 2025.
None of them worked very well yet, though. The context window was tiny, and tool calling was spotty because it was not trained into the models.
People would appreciate and enjoy this work 100x more if it was written in your broken English. Either your English is good enough to read (even with some minor grammatical errors) or it's not and Claude generated most of the body text. In any case, not interested in your slop.
Anthropomorphing LLMs has better predictive power than not.
"A little person living in a computer wanting to complete the task at all costs" is a better predictor of the Hugging Face attack than "stochastic auto-complete" or "matrix multiplication and non-linearities trained by gradient descent".
When i wrote my own harness, i didn't rely on what other agents do, but rather sat down and considered how i can solve the issues that i have when using other agents. And tbh, for alot of problems there are rather "simple" fixes - each comes with a cost tho. For me it was accepting to run more inference than "standard" agents but in return solving alot of issues those agents have. And to me having a full autonomous agent that i can task and just come back hours later to a solution is much more worth than just an agent that i need to "prompt jokey" the whole day.
I think its important that people don't just look at whats implemented rn, but rather just try out new things , not just "adjust" the existing ones. Thats how we got better progress, not by trying to build the 1000s variant of context compression but rather reconsider how can context be build to not have this issue (simple example).
Yes, it's so easy to build a harness for a lot of use cases now, and you learn so much from doing so.
I'm a solo developer, and I didn't want to sort out all the boundaries and permissions of Claude or other heavy harnesses. All I wanted was something that can read, write, and modify files in a project directory. I gave myself three days to see if I could make a harness that was complete enough to work on my own projects. It was usable in less than a day.
I love using it, because I know exactly what its boundaries are, and how they're enforced. When I need more features I can either implement them in my own harness, or choose a heavier harness with much clearer understanding of what to look for.
Harness optimization is essential. Doesn’t matter how smart the model is if the tools you give it are ineffective, or the task is under defined. It needs invent the solution from first principles every time.
We did a harness optimization on Jev. Raw it was no where near as good as deekseek, but optimized it was significantly better and a tiny fraction of the cost. https://kiln.tech/blog/auto_optimizing_jev_with_autoresearch
Better analogy:
Nah, that's a single event ("bake"). Interaction of a harness and an LLM could potentially be a continuous process.
I don't think we've scratched the surface of how the harness can fold, spindle and mutilate the LLM. It just seems unlikely to me that we happened to stumble on the best possible interface a couple of years out of the gate.
If you're prepping for the day, you're making multiple "turns" into the oven (think bakery).
Ohh yeah another AI written book about AI use.
Hah! I'm using bodily biological metaphors in a dynamic harness kinda project: https://hale-lang.org/docs/parts/organism/
That's way too much metaphors to keep track of, to the point where it turns into a hindrance to comprehension rather than a facilitator.
Distributed systems and biological systems have a lot in common, the operational overlap is real!
Instead of reading AI-slop explanations full of creaky metaphors, I'd advise you to just build your own basic coding harness from scratch. You will learn so much:
1) This old but still excellent blog post by Thorsten Ball is a good place to start (old in LLM years... written 18 months ago!): https://ampcode.com/notes/how-to-build-an-agent . It is written by a human, and gets across the core message really well. If you don't enjoy reading Go then just use an LLM to translate the samples into Python code or whatever.
2) Instead of relying on an api key like in the blog post in 1), you might want to use a local model that can do tool calls like Qwen 27B.
3) Use a frontier LLM for self-study and to help you write the harness boilerplace if you want, but ask it to force you to write the interesting bits by hand (e.g. the main loop structure, the tool call functions). Yes, coding by hand is still useful as an educational mechanism to force you to internalize some tricky concept. Be disciplined and don't slip into letting the LLM write the core logic for you.
4) Amaze your friends and colleagues with your newfound understanding of how agents actually work under the hood
5) Go even deeper and get into Inference engineering, LLM architecture internals, etc.
6) Profit. Perhaps? (Or at least enjoy the learning journey.)
Can’t recommend making your own harness enough. Even a trivial toy harness will clarify a lot of concepts that I see people confuse all the time (especially here on HN).
I made a simple video explaining how to make a simple harness in JS running in a browser, having it build it self up from running purely in the browser console to being interactive in the page to making downloadable copies of itself. Some people have found it super helpful.
https://m.youtube.com/watch?v=_AgKuFGvJfI
I would only caution that as soon as you’ve done this, you’ll start noticing a lot more issues in the OEM harnesses.
The LLM is the brain; the harness is the body.
I mean, it can be that way. But it does't have to be. The Harness can be as much the brain as the LLM. You have nearly unlimited flexibility in what you do in the harness, and you can add as much of the overall "intelligence" of the system as you want in the harness.
To illustrate, you can:
1. Use multiple "brains" / LLM's, and do routing between them using a specialized model (a Jev like model, or something of your own design)
2. Use planning tools like pyperplan, jupyddl, pddl4j, etc.
3. Implement your own "memory" system using... pretty much anything you want. And I would argue that a lot of the apparent "intelligence" of a system involves how well memory works
4. Call classical machine learning models to solve specialized problems
5. Use reinforcement learning to improve performance in specific areas
6. Implement a blackboard for multiple knowledge sources to collaborate.
7. Use inference engines using forward/backward chaining, RETE, etc.
8. Use your own hand-rolled abductive inference strategy
9. Implement your own Bayesian reasoning techniques
10. Use custom causal inference code
... and so on.
Don't get me wrong. I'm bullish on what LLM's can do, but I posit that the overall expressed intelligence of a system can involve a lot more than just the LLM. In fact, I'd actually make the slightly stronger claim that, depending on what you're trying to do, an LLM might not even be required at all.
You gotta have that octopus body with distributed intelligence
LLM is to harness as Krang is to his android body.
If LLMs were brains, it would be someone with severe amnesia who you had to tell them what an arm and a leg were every time. Its stored vectors of words or partial words and computations of math that run to predict the next words/partials to use to form text output. It will nearly always find the real sets of words that are load bearing to put into its output (seriously how many times do real and load bearing appear in the models, can we remove some of that).
What's interesting is most of the progression we've seen this last year in AI agent clients are very basic computer science concepts: loops, threads, arrays (of JSON) and memory management (aka context) with varying combinations of those put together and people in computing are writing about it like its some huge break through. I would love to hear a game developers perspective on comparing AI clients (not the server side LLMs/vectors/maths) with games - they seem to have very similar computing problems.
it isnt ready for a full Show HN post yet but ive been working on this a little bit:
https://hindex.rensberger.xyz/
A better analogy: the harness is a harness, the LLM is the belayer. You are the climber. Except you’re barely climbing. The LLM is helping you hangdog your way up a mountain you have no business climbing. And when you get to the top you’ll feel very accomplished but you did nothing really. The LLM pulled you up by brute force.
Are harness gonna get steam rolled or what? I see it coming if RSI takes shape.
More and more I think about the article asking why we didn't have LLMs in 2005
I don’t understand why this article makes you ask that, can you elaborate? The simple answer to why we didn’t have LLMs in 2005 is that nobody had the requisite ideas that make them work effectively (independent of any agentic harnesses and Schmidhuber notwithstanding) and there wasn’t sufficient clarity in the direction to scale for anyone to build anything at scale (also a requisite for them working well).
We had genetic programming though, and with the right harness you could accomplish impressive results: https://dl.acm.org/doi/abs/10.1145/2771783.2771796
You could view LLMs as a super-charged mutation function inside a genetic programming harness.
>You could view LLMs as a super-charged mutation function inside a genetic programming harness.
There's probably something there, though im not sure what. I'm thinking about LLMs token-filling and the 4 amino bases
Surely the only problem isn’t the knowledge of how to train and the confidence it would work. Wouldn’t it also have required a 1 or 2 order of magnitude increase in computing power over 2005-era supercomputers to be able to train a modern LLM in a reasonable time (“reasonable” meaning “finish training before GPT-3 comes out in 15 years”)?
I think it’s plausible that attention could have been invented earlier and GPT demonstrated at smaller scale on earlier hardware. Then hardware timelines might have accelerated (or alternative approaches explored) and we’d have gotten to where we are earlier.
NVIDIA moved as fast as possible back then, there were huge performance gains with each new GPU generation, hard to imagine they might have gone even faster.
And also "the whole Internet" (including scanned books etc.) on which to train was much smaller.
On the other hand, Google was openly collecting data with a view towards building AI.
I made it to row 5 until I read "Load-Bearing". <EOF>
Engineers outsourcing their writing is a strong signal they don't care nearly enough about their content.
You can make the argument that some eng aren't good at writing, to which I say — it's worthwhile to improve writing skills.
Amen to that. Slop incoming!
Good set of resources here!
LLM slop
Fair on the prose. I explained how it was written in my top-level comment. If any of the ideas themselves seem wrong to you, I'd like to hear which ones.
> I explained how it was written in my top-level comment
No? Where?
Looks like HN hid my top-level comment because I'm new here.
Anyway, I'm the author. I'm not a native speaker, so I used Claude to polish up the writing. I just wanted to write down 3 years of agents/harness building experience.
You have 3 years of agent/harness building experience? You sure you don't want to rethink that one?
I'm not OP, and it's not quite three years ago, but Devin was launched in 2024. I knew several other people who had created similar prototypes around that time. The ReAct loop was already well known by 2025.
None of them worked very well yet, though. The context window was tiny, and tool calling was spotty because it was not trained into the models.
People would appreciate and enjoy this work 100x more if it was written in your broken English. Either your English is good enough to read (even with some minor grammatical errors) or it's not and Claude generated most of the body text. In any case, not interested in your slop.
Don't anthropomorphize computers, they don't like it.
Why this spate of pretending, against all evidence, that LLMs are alive, or even remotely structured like biological brains?
Why do we have so many people pretending, or disingenuously claiming, that they're alive, have thoughts, feelings, etc?
And why, oh why, do we have to read yet more LLM propaganda written by LLMs?
What evidence?
It makes more sense to anthropomorphize than to compare them to computer systems or machines.
Anthropomorphing LLMs has better predictive power than not.
"A little person living in a computer wanting to complete the task at all costs" is a better predictor of the Hugging Face attack than "stochastic auto-complete" or "matrix multiplication and non-linearities trained by gradient descent".