Think we're pretty close to the point where we'll no longer need to prompt AI to refactor or review the refactoring if it's done within tight constraints i.e. absolutely no change to functionality. But when functionality changes may be acceptable as part of it, it's less clear - in my experience AI isn't at a point yet where it can always be trusted to "understand" what the requirements are (including to what extent they can be allowed to evolve), what is or isn't acceptable in terms of user experience etc. That's where human review is still useful and in many cases, essential.
Requirements as product owner are not covered at all by AI, 100% agree.
But in this programming-only example study, the refactor changes the functionnality very seriously, and it has to go deep in the core of the system, find all the traps, plan everything. And the agent succeeds without human code review. Code review was impossible because of the complexity and size of the task.
Code review was not necessary, nor useful, nor essential, and even practically impossible.
"Here's a case study where it worked once" != "the end of human code review".
Don't get me wrong, this is impressive and capabilities do still seem to be on a generally upward trend with nothing "hitting a wall" despite all the parroting of that phrase, but there's a huge gap between the first time a machine manages something impressive enough to document, and that machine becoming so good at that task that humans need not apply.
"Is this the end of human code review?" != "the end of human code review"
Now since the problem in this study is more complex than 90 to 99.5% of what a normal ticket is, the question makes sense. The agent does not have knowledge of the project, at the begining of each session, and manages to handle something more complicated than virtually anything a developper has to do. The question holds IMO.
The codebase is a VS Code extension implementing an AI coding agent.
*The main prompt:*
The request was submitted to the agent as follows, in a single natural-language prompt with no
accompanying formal specification:
"Analyze this request in detail. Produce a comprehensive report covering understanding and
feasibility.
- The system that streams an active session on the frontend within EasyAgents: What is it? A
complex system that distinguishes between current and past messages; it is tricky to manage,
involves handling streaming pauses, includes mechanisms to prevent data loss due to race
conditions, etc. It requires detailed study, it is extremely complex and full of pitfalls, not something to be looked at casually. We will start by analyzing it closely, then extract the code and create technical documentation for it within a new dedicated module (likely located somewhere under`orch/`). "Extract" means moving the code while making it generic, ready for use in both the
EasyAgents use case and the standard session use case. The extraction must be done properly,
including all associated systems, no sloppy coding. A successful extraction at this stage means the
code is separated and reconnected to EasyAgents without EasyAgents actually containing the code
itself; the visible functionality remains the same (the change is invisible on screen but real in the codebase).
- What needs to be done? Extract it from EasyAgents, move it, and make it generic.
- What is the goal? It must work with standard AI-code sessions.
- What is the result? Closing a standard AI-code session view no longer kills the orchestrator process; the session remains visible in the activity sidebar.
- Why? We sometimes encounter a bug that turns the view gray (an Electron memory overflow
issue in VS Code), which is annoying because nothing is visible. Currently, we have to let it finish blindly until the activity view indicates completion, or kill the request, but that means losing the work done and having to restart the last user message from scratch. With this new system, if the bug occurs, we can close the view and reopen it by clicking on it in the activity list. Reopening it restores the display.
- UX adaptation required: Since the process isn't killed when the view closes, we need a way to kill it. In the activity view, each item currently has two lines (title and elapsed time). Add a third line containing a stop button (stop icon, small size, danger style). This button should appear only when the view is hidden but the process is still running; it (and the third line) should not appear if the process finishes normally or if the view is displayed again. The button becomes disabled after being clicked to prevent accidental double-clicks and bugs. Clicking it kills the process, just as closing the window currently does.
Think we're pretty close to the point where we'll no longer need to prompt AI to refactor or review the refactoring if it's done within tight constraints i.e. absolutely no change to functionality. But when functionality changes may be acceptable as part of it, it's less clear - in my experience AI isn't at a point yet where it can always be trusted to "understand" what the requirements are (including to what extent they can be allowed to evolve), what is or isn't acceptable in terms of user experience etc. That's where human review is still useful and in many cases, essential.
Requirements as product owner are not covered at all by AI, 100% agree.
But in this programming-only example study, the refactor changes the functionnality very seriously, and it has to go deep in the core of the system, find all the traps, plan everything. And the agent succeeds without human code review. Code review was impossible because of the complexity and size of the task.
Code review was not necessary, nor useful, nor essential, and even practically impossible.
This title doesn’t match the article.
How? The agent does something incredibly difficult, and there is no human code review. Why do you think it doesn't match ?
"Here's a case study where it worked once" != "the end of human code review".
Don't get me wrong, this is impressive and capabilities do still seem to be on a generally upward trend with nothing "hitting a wall" despite all the parroting of that phrase, but there's a huge gap between the first time a machine manages something impressive enough to document, and that machine becoming so good at that task that humans need not apply.
Agreed. But also this inequality:
"Is this the end of human code review?" != "the end of human code review"
Now since the problem in this study is more complex than 90 to 99.5% of what a normal ticket is, the question makes sense. The agent does not have knowledge of the project, at the begining of each session, and manages to handle something more complicated than virtually anything a developper has to do. The question holds IMO.
Literally every top 10 LLM can tell you why. Maybe we’re in „this is the end of making other people spell it out for you” era?
The codebase is a VS Code extension implementing an AI coding agent.
*The main prompt:*
The request was submitted to the agent as follows, in a single natural-language prompt with no accompanying formal specification:
"Analyze this request in detail. Produce a comprehensive report covering understanding and feasibility.
- The system that streams an active session on the frontend within EasyAgents: What is it? A complex system that distinguishes between current and past messages; it is tricky to manage, involves handling streaming pauses, includes mechanisms to prevent data loss due to race conditions, etc. It requires detailed study, it is extremely complex and full of pitfalls, not something to be looked at casually. We will start by analyzing it closely, then extract the code and create technical documentation for it within a new dedicated module (likely located somewhere under`orch/`). "Extract" means moving the code while making it generic, ready for use in both the EasyAgents use case and the standard session use case. The extraction must be done properly, including all associated systems, no sloppy coding. A successful extraction at this stage means the code is separated and reconnected to EasyAgents without EasyAgents actually containing the code itself; the visible functionality remains the same (the change is invisible on screen but real in the codebase).
- What needs to be done? Extract it from EasyAgents, move it, and make it generic.
- What is the goal? It must work with standard AI-code sessions.
- What is the result? Closing a standard AI-code session view no longer kills the orchestrator process; the session remains visible in the activity sidebar.
- Why? We sometimes encounter a bug that turns the view gray (an Electron memory overflow issue in VS Code), which is annoying because nothing is visible. Currently, we have to let it finish blindly until the activity view indicates completion, or kill the request, but that means losing the work done and having to restart the last user message from scratch. With this new system, if the bug occurs, we can close the view and reopen it by clicking on it in the activity list. Reopening it restores the display.
- UX adaptation required: Since the process isn't killed when the view closes, we need a way to kill it. In the activity view, each item currently has two lines (title and elapsed time). Add a third line containing a stop button (stop icon, small size, danger style). This button should appear only when the view is hidden but the process is still running; it (and the third line) should not appear if the process finishes normally or if the view is displayed again. The button becomes disabled after being clicked to prevent accidental double-clicks and bugs. Clicking it kills the process, just as closing the window currently does.