My problem is right now that every kind of reviewing process adds un-needed complexity which later slows down your agility and increases your cost and performance degrades.
If you have feature and 6 different reviewers go over it the logic bulks up easily. I wonder how other guys handle that? I’m all up and I want this to work, but this is my current constraint.
I often get bad suggestions from reviewing monitors because they get something wrong. Sometimes I get good ones, very good ones, but some are bad. So I’m always down to look at reviews about bugs, but I would be afraid after 6 reviewers its just a bunch of text and its confusing me so end result is worse as me looking through actual critical stuff. My context is limited too and completely hands-off is not reliable enough...
I think it is a good approach for this year, but it misses something important - how do you know the code still reflects your intention after steering for weeks and months, across many sessions? Messages flow in the session and zoom out of view, and across compactions/new sessions they get forgotten. To solve this, I log every user message in a chat log, and review it from time to time to distill the latest form of user intent.
Another kind of review not mentioned here or in Anthropic's article is the plan review. I generate the plan for some stretch of work, then have the plan (saved as a file) reviewed by one or more agents. This checks the plan against project constraints before executing it.
If we can solve testing then coding is almost fully automated. The only remaining problem was always trust, why should we trust that code? If you read it manually then it's like walking a motorcycle. If you don't, then you need some way to derisk it. There are many ideas flowing around about how to do this, the OP and Anthropic showed their takes. I think the approach is project specific though, no general method.
What I do is that each of my requirements are captured in a yaml file, with a unique id, with the exact sentence I wrote.
Each must be covered/proven at least with one test, a linter checks whether the requirement is covered (test exists annotated with that id), and a test inventory generated from it. Usually I browse and review the test inventory first to see I get what I asked.
It does not guarantee that the agent implemented everything correctly, but if I ask another agent to review it and point out the gaps, it produces very good results.
I wonder if we would see more engineering teams transitioning away from git to Sapling. We do need a refresh on how we review and ship code.
I'm aware that GH recently released stacked PRs, but it still seems like a long way to go both from usability and reliability perspective compared to what Meta and likely Anthropic have internally.
I am inspired by this to make a harness that intentionally does sparse checkouts, so the agents only can see so much context.
I am also playing with the idea of layering additional data structures on top of the "code" that cover different aspects. For example, a tree structure of performance requirements that has durable symbolic references to entities in the code that are especially relevant. Another tree can cover aspects of security, another legal compliance, another UI guidelines.
I wrote this after reading how the Claude Code team reviews code. Their layers are stages in time. I propose layering the reader instead: one agent per layer of code (form, local meaning, wiring, runtime, use), crossed with lenses like security, history and reversibility. Each cell gets its own narrow context. Curious where people think it breaks.
this is common practice, works pretty well in my experience both to improve quality and reduce cost, and afaik is a naive version of what tools like coderabbit and greptile do internally
My problem is right now that every kind of reviewing process adds un-needed complexity which later slows down your agility and increases your cost and performance degrades.
If you have feature and 6 different reviewers go over it the logic bulks up easily. I wonder how other guys handle that? I’m all up and I want this to work, but this is my current constraint.
I often get bad suggestions from reviewing monitors because they get something wrong. Sometimes I get good ones, very good ones, but some are bad. So I’m always down to look at reviews about bugs, but I would be afraid after 6 reviewers its just a bunch of text and its confusing me so end result is worse as me looking through actual critical stuff. My context is limited too and completely hands-off is not reliable enough...
How do you guys do it?
I think it is a good approach for this year, but it misses something important - how do you know the code still reflects your intention after steering for weeks and months, across many sessions? Messages flow in the session and zoom out of view, and across compactions/new sessions they get forgotten. To solve this, I log every user message in a chat log, and review it from time to time to distill the latest form of user intent.
Another kind of review not mentioned here or in Anthropic's article is the plan review. I generate the plan for some stretch of work, then have the plan (saved as a file) reviewed by one or more agents. This checks the plan against project constraints before executing it.
If we can solve testing then coding is almost fully automated. The only remaining problem was always trust, why should we trust that code? If you read it manually then it's like walking a motorcycle. If you don't, then you need some way to derisk it. There are many ideas flowing around about how to do this, the OP and Anthropic showed their takes. I think the approach is project specific though, no general method.
What I do is that each of my requirements are captured in a yaml file, with a unique id, with the exact sentence I wrote. Each must be covered/proven at least with one test, a linter checks whether the requirement is covered (test exists annotated with that id), and a test inventory generated from it. Usually I browse and review the test inventory first to see I get what I asked. It does not guarantee that the agent implemented everything correctly, but if I ask another agent to review it and point out the gaps, it produces very good results.
I wonder if we would see more engineering teams transitioning away from git to Sapling. We do need a refresh on how we review and ship code.
I'm aware that GH recently released stacked PRs, but it still seems like a long way to go both from usability and reliability perspective compared to what Meta and likely Anthropic have internally.
I am playing with similar ideas.
I am inspired by this to make a harness that intentionally does sparse checkouts, so the agents only can see so much context.
I am also playing with the idea of layering additional data structures on top of the "code" that cover different aspects. For example, a tree structure of performance requirements that has durable symbolic references to entities in the code that are especially relevant. Another tree can cover aspects of security, another legal compliance, another UI guidelines.
I wrote this after reading how the Claude Code team reviews code. Their layers are stages in time. I propose layering the reader instead: one agent per layer of code (form, local meaning, wiring, runtime, use), crossed with lenses like security, history and reversibility. Each cell gets its own narrow context. Curious where people think it breaks.
this is common practice, works pretty well in my experience both to improve quality and reduce cost, and afaik is a naive version of what tools like coderabbit and greptile do internally