Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.
My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:
* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.
* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.
* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.
* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.
* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.
I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?
Sure, every sufficiently large codebase will have bugs somewhere, but it will work correctly at least something like 99.999 % of the time after ironing out the bugs on the common code paths. But that does not change the fact that being [mostly] correct is the lowest bar you have to cross.
Right. It feels like we're living in a parallel world or something. Can agents code? Sure. Can you let them code on their own for a serious production project? Not a chance.
second sentence of the article: "Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC)."
You come across as someone who has never worked on a real software project. Humans create tons of bugs on a regular basis. AI is already better than most programmers.
I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.
I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've seen much worse from teams of humans and I've accepted the drawbacks that are slowly going away with each new model.
A team of programmers is "worse" than their best member. Worse quality of code, worse thinking. You can usually get more done with a lot of mediocre hours of work than a few brilliant hours, though.
AI I've used isn't a better coder than I am - it's just got a lot more hours in an hour than I do.
I just setup a large refactor with Astra and was feeling super lazy and let it mostly do it without my usual extreme micro managing.
The refactor ended up adding 22,000 loc.
I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.
In case you are wondering why vibe coders are doing 30k loc a day, this is why.
"Coding" is just a poor term for this, as there's so much room to weasel different meanings out of it. At every company I've worked in past 20+ years the "coders" were engineering a product from wrong or unclear requirements and specs from non-technical people. The act of coding was secondary (but an important throttling function to make us stop and think about what is even possible or makes sense.) Never did FAANGs, so YMMV.
Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.
I’m pretty sure that same discourse was seen every mass production epoch, from textiles to electronics. Yet here we are, hand-crafted high quality things are rare and expensive.
They care in principle, for the most part bad code makes changes and extensions slow to implement and causes unnecessary production issues which costs time and money. But there is always the tension between implementing something quickly now and being able to implement things quickly in the future and unfortunately the preference is almost always quickly now despite everyone knowing that this is the way more expensive choice in the long run.
And all of these things will be solved one by one.
It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.
Why in the hell do you not think these things aren't going to be completely automated?
Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.
Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.
Nano Banana and GPT Image can do a better job than graphics designers.
LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.
Just wait. All of these things will be solved.
There is no "stopping point".
Edit:
Don't anticipate that 2036 will look anything like 2026.
Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.
May the iterative loop of adding new axes to evaluate on be a natural, healthy progression, instead of needing to frame it as an us-them problem?
If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.
If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.
But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.
I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.
Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).
I think the usage of solved is silly. Things get automated, they basically never get ‘solved’.
Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.
yep it blows my mind that competent people don't get this. fable can code better than 80% of people in the software profession. There's no ifs and buts about it. There are a lot of great developers here so to most it's probably not obvious that LLMs have blown a giant hole in the future headcount of this profession.
i understand the cope because it legit seems like such a cynical thing to say but we have to face the reality at hand.
It's a nice article that basically (rage)baits the readers before they realize the author actually disagrees with the premise of coding being solved. So here goes, my higher-level rant on "solving" something with AI.
I strongly dislike all the "X is solved" narratives. What does it mean for something to be solved? A math problem (or any kind of problem), a riddle, a mystery, a dispute. Those are all instances of a particular situation that requires a "solution", but new situations will always come up. I understand that by "cancer/coding/X is solved" in this new age of (gen)AI that means the ability to streamline or speed up the "solution finding" procedure, but even that presupposes a fixed, static, fully deterministic space of the things we are trying to "solve".
Even cancer cannot be fully solved - Demis Hassabis slowly started drifting away from using the word "diseases can be solved" because they can't be eliminated - we can only speed up the process for finding a cure for any particular disease, be it existing or new and/or evolving.
Is bridge-building solved? Architecture? Why are architects still employed? Is solving "civil engineering" or designing an optimally running machine a thing to be solved? What are we trying to do when we talk about "progress with AI"? Even when the "recursively-self-improving-AIs" and "perfect" robots do arrive, we're still bound to work with them, and they'll have to evolve to find new solutions to new problems.
To be clear, I work with and rely on LLMs every day, from coding custom RAG architectures with CC and Pi to research and agentic data science. These bombastic conversations, however need to quiet down a bit so we can get back to work :)
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
Writing code at enterprise level is insanely difficult. You are constrained by budget, staff, legacy databases/environments, business rules hiding all over the place, and people.
You can't just rewrite everything. So over many years people are touching small parts of the pie.
Yeah, enterprise code has that trope of being enterprise-y, verbose and bad. In my experience, that has always been the opposite. At the big corps/FAANGs I worked at, a single line of change can adversely impact millions of paying customers, so a lot of the verbosity and harnesses exists to dampen the failure modes.
Most of the terribly written stuff has always been at startups, where devs fling nearly anything across the finish line, if it barely works the happy path.
So much "bad" enterprise code evolved into that state over years or even decades of small changes. Meanwhile, last year I got to watch an LLM-authored codebase speedrun itself into a similar state in only a couple months. And I would say that the enterprise code was actually better. It at least did its job fairly reliably. The LLM codebase was riddled with defects, so much so that it ate up all our time and our feature delivery rate ground to a halt.
There are two observations that really eat at me:
1. Studies seem to indicate that agentic coding uses 2-10x as many lines of code to accomplish the same task.
2. One of the only really well-established empirical results in software engineering is the strong association between LOC and defect rate.
This is happening all over the place right now. There is a ton of greenfield happening, which further adds to the illusion of speed. Eventually you produce a big old pile of shit that even with the help of the LLM is weird to reason about, and it slows way down. Many such cases.
I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.
> But humans still need to be responsible for understanding what they're shipping
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
Let’s not romanticize it too much... A lot of enterprise systems are built by developers copying an old AbstractBeanFactoryFactory from a 2011 stack overflow thread without really understanding it :)
Yeah... like we all get to start green field projects and write all the code we should understand. Many of us cut our teeth on bad legacy stuff with no proper documentation made by "engineers" long gone. At least a LLM can easily make sense of this mess.
Indeed. And not fair comparisons ”look at the quality of this small one-shot Claude hobby project. The quality is less than this major open source project written by some of the best developers in the world”
To be fair the pitch has frequently been that Devin/Claude/Astra/whatever is some sort of superhuman bottled John Carmack that will single-handedly replace entire teams of developers.
Yea, that extreme side exists too. Truth is inbetween. AI with the instructions from a dev that knows what it is doing writes better code than most regular 9-17 devs.
I've been doing development, in one way or another, since the 90s. I've worked with dozens of teams from enterprises to startups. Hundreds of developers. The quality of work has been all over the place, but the majority was not great.
I'm arguing that what people today call "AI slop" is already higher quality than what most developers created historically, and the fact that tests and documentation pretty much come for free now means that the floor has been raised.
The quality of AI generated code is not great. Yes, it will get better. It's already better than 65%+ of what regular devs can do AND it is faster to produce, iterate, and release.
1. People didn’t wear that as a badge of honour though.
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
This was always due to pressures by management and the company environment, not the workers themselves. It's hard to blame the people writing code when they have to deal with nontechnical leadership that wants to have a feature factory or never given appropriate resources to solve problems.
Blaming workers is always an excuse by poor management.
Nah, you're still blaming workers and not leadership. If leadership is okay with not training workers (something American corporations would do in the distant past) then it's not fair to continue to blame workers when leadership is clearly aware of the problem and would rather pocket the money than help workers.
These companies pay management more than workers for a reason, if you can't even admit that they are to blame then what are you trying to do here? Just attack workers for what reason exactly? Being anti-worker is a great tell to never trust a person.
We're talking about professionals here. People who (at least in the US) often make several multiples of the median worker. Competence is assumed, and every company I've been at has had programs to pay for additional school if the employee wants it. Do surgeons and lawyers complain so regularly that management doesn't train them? Or are software engineers just this desperate to be seen as "not a real professional"?
I started Valknut (https://github.com/sibyllinesoft/valknut) when I saw the writing on the wall regarding Agent code structure/abstractions/etc being a limiting factor in the ability to autonomously build projects. My experience was that good linters helped, but it wasn't enough, you needed to be able to enforce information-theoretic related organizing principles in addition to file/function LOC and local complexity metrics to guide agents on how to structure code.
Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them.
Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anymore for a large part of non safety-critical software. I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value, but is there actually evidence that these models struggle more with poorly maintained code?
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff...
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
Number of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy.
The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.
One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
It will be solved when there is no more code left to write.
Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before.
Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task.
Code and software and applications are an intermediate stage. The final stage is an AI/LLM that just does the thing that is needed without any code being written, there are no applications or programs, just an AI that does everything.
Coding just a stage in the software development. Design and specifications which can help in coding is not solved at all and may never - the end result is software reliability is not a solved problem.
"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate"
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.
I encountered bugs created by AI. No matter how many times it tried it could not fix the bug, it was introducing so much slop to work around the issue.
I finally gave up and read documentation for 15 minutes and solved the problem.
I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
If coding is ever solved, and if software does it, sloppiness probably won’t matter much.
Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one.
I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
thinking about how to accurately describe the problem at hand and figuring out the simplest way to approach it takes a lot of effort well before you get to the implementation step of coding.
I wonder how much of this is due to the AI tooling being taught on sloppy code that humans have written. Over the last 4 decades I've looked at a lot of code on the Internet and there's a lot of slop out there.
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM.
90% of code before LLMs was badly copied from StackOverflow anyway.
That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that.
The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit.
(sorry if I sound incoherent, just some thoughts while I'm commuting)
Plopping in my email to the author below in case anyone else is interested in this kind of thing:
> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com
Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".
In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.
Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:
L1: Show a number on the screen
L2: Allow a user to click plus or minus and the number responds accordingly
L3: Show that number on any device running your software, and keep it in sync with all other devices
L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished.
L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa.
L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with.
L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application.
L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well)
...
...
...
And so on and so on.
Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.
I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.
I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.
Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.
Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.
> Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability
My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:
* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.
* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.
* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.
* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.
* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.
Correctness is not a binary thing though.
I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?
Sure, every sufficiently large codebase will have bugs somewhere, but it will work correctly at least something like 99.999 % of the time after ironing out the bugs on the common code paths. But that does not change the fact that being [mostly] correct is the lowest bar you have to cross.
Yes. I use these models day in and day out, on all sorts of tasks. I cannot believe I hear people say that coding is solved.
Right. It feels like we're living in a parallel world or something. Can agents code? Sure. Can you let them code on their own for a serious production project? Not a chance.
Plenty of serious production projects are doing exactly that. Are you using GPT-6 Astra, or something older?
second sentence of the article: "Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC)."
You come across as someone who has never worked on a real software project. Humans create tons of bugs on a regular basis. AI is already better than most programmers.
I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.
I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've seen much worse from teams of humans and I've accepted the drawbacks that are slowly going away with each new model.
Better at writing one piece of code, maybe.
Better at writing code within a huge system, definitely not. Maybe in the future, but as of Astra, Fable 5.1, the answer is still no.
AI is the average of all programmers.
It's just that many (I guess that includes me? :D) assumed that they are better than the actually were.
A team of programmers is "worse" than their best member. Worse quality of code, worse thinking. You can usually get more done with a lot of mediocre hours of work than a few brilliant hours, though.
AI I've used isn't a better coder than I am - it's just got a lot more hours in an hour than I do.
I just setup a large refactor with Astra and was feeling super lazy and let it mostly do it without my usual extreme micro managing.
The refactor ended up adding 22,000 loc.
I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.
In case you are wondering why vibe coders are doing 30k loc a day, this is why.
Amen. I sometimes wonder if all programmers are now marketing people who know shit about software development and engineering.
"Coding" is just a poor term for this, as there's so much room to weasel different meanings out of it. At every company I've worked in past 20+ years the "coders" were engineering a product from wrong or unclear requirements and specs from non-technical people. The act of coding was secondary (but an important throttling function to make us stop and think about what is even possible or makes sense.) Never did FAANGs, so YMMV.
Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.
That is very well put and summarizes what distinguishes real software development from vibe coding.
I’m pretty sure that same discourse was seen every mass production epoch, from textiles to electronics. Yet here we are, hand-crafted high quality things are rare and expensive.
I have seen so much production code with garbage code and massive bugs, the industry doesn't care for 'real'.
They care in principle, for the most part bad code makes changes and extensions slow to implement and causes unnecessary production issues which costs time and money. But there is always the tension between implementing something quickly now and being able to implement things quickly in the future and unfortunately the preference is almost always quickly now despite everyone knowing that this is the way more expensive choice in the long run.
And all of these things will be solved one by one.
It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.
Why in the hell do you not think these things aren't going to be completely automated?
Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.
Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.
Nano Banana and GPT Image can do a better job than graphics designers.
LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.
Just wait. All of these things will be solved.
There is no "stopping point".
Edit:
Don't anticipate that 2036 will look anything like 2026.
Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.
May the iterative loop of adding new axes to evaluate on be a natural, healthy progression, instead of needing to frame it as an us-them problem?
If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.
If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.
I think you need to define what solved means and what better means.
Have you tried one-shotting real distributed systems problems? What was the result and how did you verify correctness?
Like the 0.1% we do is your counter example?
But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.
I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.
> compelling novel game loops
Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).
I think the usage of solved is silly. Things get automated, they basically never get ‘solved’.
Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.
yep it blows my mind that competent people don't get this. fable can code better than 80% of people in the software profession. There's no ifs and buts about it. There are a lot of great developers here so to most it's probably not obvious that LLMs have blown a giant hole in the future headcount of this profession.
i understand the cope because it legit seems like such a cynical thing to say but we have to face the reality at hand.
People are ego-centric and pull a blindfold over their eyes.
I have no doubt that AI will eventually be able to write essentially perfect code, I am just saying that we are still quite far away from that point.
God, why are you so stupid.
It's a nice article that basically (rage)baits the readers before they realize the author actually disagrees with the premise of coding being solved. So here goes, my higher-level rant on "solving" something with AI.
I strongly dislike all the "X is solved" narratives. What does it mean for something to be solved? A math problem (or any kind of problem), a riddle, a mystery, a dispute. Those are all instances of a particular situation that requires a "solution", but new situations will always come up. I understand that by "cancer/coding/X is solved" in this new age of (gen)AI that means the ability to streamline or speed up the "solution finding" procedure, but even that presupposes a fixed, static, fully deterministic space of the things we are trying to "solve".
Even cancer cannot be fully solved - Demis Hassabis slowly started drifting away from using the word "diseases can be solved" because they can't be eliminated - we can only speed up the process for finding a cure for any particular disease, be it existing or new and/or evolving.
Is bridge-building solved? Architecture? Why are architects still employed? Is solving "civil engineering" or designing an optimally running machine a thing to be solved? What are we trying to do when we talk about "progress with AI"? Even when the "recursively-self-improving-AIs" and "perfect" robots do arrive, we're still bound to work with them, and they'll have to evolve to find new solutions to new problems.
To be clear, I work with and rely on LLMs every day, from coding custom RAG architectures with CC and Pi to research and agentic data science. These bombastic conversations, however need to quiet down a bit so we can get back to work :)
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
Writing code at enterprise level is insanely difficult. You are constrained by budget, staff, legacy databases/environments, business rules hiding all over the place, and people.
You can't just rewrite everything. So over many years people are touching small parts of the pie.
If it works it isn't low quality.
Yeah, enterprise code has that trope of being enterprise-y, verbose and bad. In my experience, that has always been the opposite. At the big corps/FAANGs I worked at, a single line of change can adversely impact millions of paying customers, so a lot of the verbosity and harnesses exists to dampen the failure modes. Most of the terribly written stuff has always been at startups, where devs fling nearly anything across the finish line, if it barely works the happy path.
The code I have seen and see still, started shitty.
I'm not so sure that's a fair comparison.
So much "bad" enterprise code evolved into that state over years or even decades of small changes. Meanwhile, last year I got to watch an LLM-authored codebase speedrun itself into a similar state in only a couple months. And I would say that the enterprise code was actually better. It at least did its job fairly reliably. The LLM codebase was riddled with defects, so much so that it ate up all our time and our feature delivery rate ground to a halt.
There are two observations that really eat at me:
1. Studies seem to indicate that agentic coding uses 2-10x as many lines of code to accomplish the same task.
2. One of the only really well-established empirical results in software engineering is the strong association between LOC and defect rate.
This is happening all over the place right now. There is a ton of greenfield happening, which further adds to the illusion of speed. Eventually you produce a big old pile of shit that even with the help of the LLM is weird to reason about, and it slows way down. Many such cases.
I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.
> But humans still need to be responsible for understanding what they're shipping
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
Certainly the population of people that need to know how something works is much smaller.
But even still, cognitive debt is a real, documented phenomenon where we lose our ability to change projects.[1]
We also know coding agents tend to accumulate slop in projects over time without some human intervention[2]
There's also the concern of whether its a good idea to be dependent on an addictive slot machine run by a trillion dollar company to do your work.
1 - https://simonwillison.net/2026/Feb/15/cognitive-debt/
2 - https://www.scbench.ai/
Let’s not romanticize it too much... A lot of enterprise systems are built by developers copying an old AbstractBeanFactoryFactory from a 2011 stack overflow thread without really understanding it :)
Yeah... like we all get to start green field projects and write all the code we should understand. Many of us cut our teeth on bad legacy stuff with no proper documentation made by "engineers" long gone. At least a LLM can easily make sense of this mess.
Indeed. And not fair comparisons ”look at the quality of this small one-shot Claude hobby project. The quality is less than this major open source project written by some of the best developers in the world”
To be fair the pitch has frequently been that Devin/Claude/Astra/whatever is some sort of superhuman bottled John Carmack that will single-handedly replace entire teams of developers.
Yea, that extreme side exists too. Truth is inbetween. AI with the instructions from a dev that knows what it is doing writes better code than most regular 9-17 devs.
And it can -- if an able person wields it.
That is very true and with llms someone who wrote low quality code can now output a whole lot more code. Maybe in better quality, maybe not.
Not "a lot of". "The majority of".
I've been doing development, in one way or another, since the 90s. I've worked with dozens of teams from enterprises to startups. Hundreds of developers. The quality of work has been all over the place, but the majority was not great.
I'm arguing that what people today call "AI slop" is already higher quality than what most developers created historically, and the fact that tests and documentation pretty much come for free now means that the floor has been raised.
The quality of AI generated code is not great. Yes, it will get better. It's already better than 65%+ of what regular devs can do AND it is faster to produce, iterate, and release.
1. People didn’t wear that as a badge of honour though.
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
This was always due to pressures by management and the company environment, not the workers themselves. It's hard to blame the people writing code when they have to deal with nontechnical leadership that wants to have a feature factory or never given appropriate resources to solve problems.
Blaming workers is always an excuse by poor management.
Many people were and still are simply terrible developers.
Hand those people an LLM and they don’t suddenly become competent, but they do start slinging more code.
Nah, you're still blaming workers and not leadership. If leadership is okay with not training workers (something American corporations would do in the distant past) then it's not fair to continue to blame workers when leadership is clearly aware of the problem and would rather pocket the money than help workers.
These companies pay management more than workers for a reason, if you can't even admit that they are to blame then what are you trying to do here? Just attack workers for what reason exactly? Being anti-worker is a great tell to never trust a person.
We're talking about professionals here. People who (at least in the US) often make several multiples of the median worker. Competence is assumed, and every company I've been at has had programs to pay for additional school if the employee wants it. Do surgeons and lawyers complain so regularly that management doesn't train them? Or are software engineers just this desperate to be seen as "not a real professional"?
I do blame management for letting these people through the interview process and then not firing them. But that’s independent of the fact they exist.
Coding might be "solved" but coding still is the best way to build your own mental model of the solution space.
Which is more important to you: Velocity to a solution? Or velocity to understanding?
Interesting way to lay it out. For us understanding is obviously crucial for prod and repeatable business functions.
Velocity to solution is default for almost everyone else, especially one-off or low impact / low consequence of failure projects.
I started Valknut (https://github.com/sibyllinesoft/valknut) when I saw the writing on the wall regarding Agent code structure/abstractions/etc being a limiting factor in the ability to autonomously build projects. My experience was that good linters helped, but it wasn't enough, you needed to be able to enforce information-theoretic related organizing principles in addition to file/function LOC and local complexity metrics to guide agents on how to structure code.
Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them.
Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
Coding is solved, but AI companies are still hiring software engineers?
So they still might do but in parallel the software engineering market in china and india is collapsing.
Source?
Lots of chatter online, i have seen a few videos on this tpic on YT.
https://www.hays-china.cn/en/press-release/content/%E4%B8%AD...
Best explanation I’ve heard is that “coding is solved but software engineering hasn’t”.
* slop engineering
Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anymore for a large part of non safety-critical software. I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value, but is there actually evidence that these models struggle more with poorly maintained code?
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff...
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
Number of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy.
The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.
One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
It will be solved when there is no more code left to write.
Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before.
Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task.
I might be wrong :)
I'm pretty sure we reached peak software developer jobs due to ai.
The market is already collapsing in China and India.
Code and software and applications are an intermediate stage. The final stage is an AI/LLM that just does the thing that is needed without any code being written, there are no applications or programs, just an AI that does everything.
This doesn’t scale at all for some problems where a specific algorithm is needed (graph problems for instance).
Coding just a stage in the software development. Design and specifications which can help in coding is not solved at all and may never - the end result is software reliability is not a solved problem.
"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate"
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
Sad as it might sound, I think we might have to stop worrying about the code.
Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.
I encountered bugs created by AI. No matter how many times it tried it could not fix the bug, it was introducing so much slop to work around the issue.
I finally gave up and read documentation for 15 minutes and solved the problem.
I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
It is not clear to me how the verbosity metrics works. Can someone shades more light on it?
It's a bunch of rules flagging specific patterns that could be expressed more succinctly: https://github.com/gabeorlanski/scb-check/tree/a8618228939de...
If coding is ever solved, and if software does it, sloppiness probably won’t matter much.
Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one.
Test suites will be important.
I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.
that is true, but if the metric is what we want optimized, then that's fine.
However it is more likely to be something which can be detached..
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree.
On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
> that's primarily a human concern that's unrelated to the final output.
It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
Well... Coding was never the most part of work I spent time.
Congrats on your promotion to full time jira management!
very glib comment.
thinking about how to accurately describe the problem at hand and figuring out the simplest way to approach it takes a lot of effort well before you get to the implementation step of coding.
Exactaly. Some tasks took me hours or days to change a line.
> I was disappointed at how “vibes based” the industry seems at the moment.
Alan Kay called programming "Pop Culture" some 20 years ago[0]
[0] https://queue.acm.org/doi/10.1145/1039511.1039523
Solving consciousness ;)
I wonder how much of this is due to the AI tooling being taught on sloppy code that humans have written. Over the last 4 decades I've looked at a lot of code on the Internet and there's a lot of slop out there.
Claim: AI writes almost perfect code.
Reality: earandil.com uses 170% CPU in Firefox.
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
Big “if” you got there
It will be fun to try and deterministically define sloppiness relative to a tool that is not deterministic.
Of course, sloppiness to date can be measured by different shared and interpreted preferences and definitions.
Coding has been solved for 20 years at least.
90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM.
90% of code before LLMs was badly copied from StackOverflow anyway.
That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that.
The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit.
(sorry if I sound incoherent, just some thoughts while I'm commuting)
Plopping in my email to the author below in case anyone else is interested in this kind of thing:
> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com
Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".
In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.
Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:
L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on.
Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.
I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.
I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.
Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.
Best, Michael
PS: here is a demo of my submission for this first round: https://x.com/technoplato/status/2090902061437030777?s=20