Unfortunately, this is the type of statements we can't verify. I'm not sure why these types of news are still coming out when we all have AI at work.
Whenever someone does such a huge drastic change like this, it's by ignoring a large chunk of code that most people were afraid to touch for good reasons. Now, that code is gone, AI is celebrated, things will break, people will work very hard in the background to fix it, with no fanfare.
> I'm not sure why these types of news are still coming out when we all have AI at work.
Because OpenAI is burning $15 billion/year, and outrageous stories like those get parroted in the media. It's free marketing for a company desperate to get middle management to believe that a $500/mo subscription is absolutely crucial for every single employee.
ugh just recently I saw a news article gushing about how AI had helped with some health/medicine study, and various patient organizations were all like "oh yeah this is a Good use of AI" and then I click the link to read the study and it's decision trees and clustering on a tiny dataset that you could analyze with a ten year old laptop.
I mean, sure at some point decision trees and clustering were called "AI", but the way the article was written you'd think OpenAI and Anthropic were responsible for the advancement of medicine.
yeah but you'd need skills to operate the 10 year old laptop. probably have to copy and paste and understand excel and maybe whatever the hell a confidence interval is
this allows said middle managers to do the analysis themselves and feel mostly confident in the results.
I can for sure say that this kind of statement is exaggeration if not a blatant lie. It may work for PR but it gives a wrong impression/ideas. Now engineers should be ready to be questioned by their management for all the rewrites that they have been estimating in months if not years.
> But at the rate we were going, we were still roughly five years from finishing.
Everyone has seen how this sort of thing comes about: it's meaningful work for engineering but never business/product critical so it just drags along.
Seems like this time around someone just went "I wonder if we could do it this way" and it worked. Perfect example of ditching sunk-cost and starting from scratch. Great outcome for them.
This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting.
> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).
Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.
Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".
Unit testing can get you some of the way, but its not a full-on all-case guarantee. Sometimes the code is encapsulating some particularly complex system/behaviour. Sometimes the reason is interop/compatilibity issues or some kind of politics.
P.S. you managed to introduce a typo in your quote (were –> where)
Or the reason is that something is just unfeasible with the existing architecture, or the reason is that building this would break important technical assumptions... All kinds of things.
Exactly. You can simply close all open tickets with <WONTFIX> and claim that you've cleared 5 years of engineering work in 5 minutes. It doesn't mean anything.
It’s just sales claims.
Do you see that “contact sales” button?
You’re not the target of that.
Deranged AI-psychotic C-suites with a two-digit IQ and too much confidence are the target.
1. I am not sure what your profession is, but I spend most of my reading time at work, with code, emails and so on. Unfortunately I don't have much time to read for recreation these days.
Phrased this way, my aspiring product leader hat activates: that 5 years of work cannot be of much value to a company worth billions because a 5 year estimate is the lowest possible qualification you can make. Its a “will never happen” shelving.
all work is stack ranked against other opportunities. no matter how many teams help to parallelize, leadership basically comes down to ranking the stack correctly. Valuable work will be at the top by any means necessary. “mvp”, probes, task forces, code yellows, and so on.
edit: it’s even a backhanded compliment to agentic coding. Asana a public company considers a $12k outlay for something they’d never spend real resources on worth a case study =|
either it was a hail mary 5 year change-the-market offering, or it was something that staff can "get around to" whenever they have the cycles, and would, at current rates, take like 3-5 years to do.
was the tool worth 12k? if it's not a line in capex or opex budgets it's value is $0
Apparently the task was to remove dead weight. Haven't read what they replaced "Enzyme" with.
I did subagent based removal tasks a few times. These were the ones that required the least amount of input or thinking from me, because the requirements are abundantly clear. "Remove this part of the code without breaking any other part or by porting the tests done with it onto another system."
In these situations the code acts as the bookkeeping ledger itself, and coordination complexity is almost a no-brainer.
If you ask Sol or Claude how much time it will take to implement a plan they just came up with, they usually advise a timeframe in the weeks or months - assuming, I suppose, that human programmers will be building it. And then you ask the model to just "do it" and it takes an hour or two. I always find this entertaining.
It does, assuming "world" is made up of humans (which it is, in its training data). It does however not have any model whatsoever of itself, apart from some superficial "I am a helpful assistant built by $X".
I find that the models have increasingly started throwing out made up amounts of time around, as if they wouldn't be aware that the user is already using Claude Code or Codex.
It's like every time they make a plan, there's something about things taking "a week or two", "month of focused work", or whatever.
This is something that would've been RL'd out a long time ago if it wasn't great for business.
A model isn't "aware" of anything. It's trained on things, and that training doesn't include time estimates for working with AI because that didn't exist yet.
What I was going after with "aware", is that the actual people working at the companies, training the models, are aware that people aren't mostly going to be implementing the plan by hand, if they've already made the plan in Claude Code or Codex. As for a specific Claude / GPT instance, "aware" would definitely be the wrong word choice there, but the instance does have its stats and environment information in its context window, unless you specifically remove it.
Either way: Training does include estimates on working with the model, and adjustments of the model itself based on that. That's literally what RLHF is.
It's straightforward to have a portion in post training that aims specifically at the model being able to give better estimates on how long that model takes to complete a certain type of task.
In all honesty I don't think it's possible to put this into the model because it's not actually sufficiently understood yet by anyone or anything. AI capabilities are changing too fast, and so is the way AI is used (worst case, even including financial limitations that are now increasingly appearing.)
Every part of training a model have the timing data observed + saved on multiple different axis, it's one of the most inherent parts of the training process.
When the post training run is nearing its cutoff point, there's a massive amount of data on how long coding tasks take to complete by that model in the golden format of "task -> time task took to complete", separable to whatever amount of subtasks, in the same format. With the parts from the end of the dataset being useful for evaluating the finished model's capabilites, whether that data is then fed back into another step in post training or not.
Completely separate from even the actual training: If you have a model proactively giving estimates that are an order of magnitude wrong, you can already fix the worst of it as of this moment by just changing the system prompt. It's a dirty fix, but it's the type of fix that has been used by Anthropic and OpenAI since forever when a model is dishing out blatantly wrong outputs.
The misunderstanding might be that I'm thinking about how long it takes a human + an AI to get something done, not an AI by itself. My current understanding is that it's still not generally viable to have LLMs perform architecture/design tasks (it might work in the short run but degrades roughly exponentially in the long run, accruing serious tech debt). Most things I'd consider actual "projects" involve at least some level of design work as well, and that'd need to be included in the estimate.
(And also prompts will need to be refined, etc., unless you have amazing prompt skills it won't immediately deliver what you wanted. Even if it did, the work to put together the AI inputs also needs to be included.)
⇒ I don't think it's easily possible right now to give a time estimate for **the full picture of** a project to be implemented with AI assistance.
> For comparison: the previous plan was expected to take at least five years and estimated to cost roughly $6M.
If that's an estimate from an LLM, those have always been way off for me. I'm constantly amazed whenever something that an agent estimates would take weeks ends up being completed in an hour.
The LLM is reasoning about estimates from its training data... which is to say, from human engineering timescales.
I suspect the labs could improve the models such that they are estimating these sorts of things but they don't prioritize doing so (or perhaps RLHF selects it away) because, as you say, it feels amazing to do a week's worth of work in an hour.
Is this the react test framework enzyme? To me, $6 million and 5 years seems like a very high estimate for this even without any automation (AI or otherwise) for even relatively large applications. How could they possibly have a test suite this large?
The British government cleared a large chunk of 5 years of the NHS (hospital) waiting list by, in part, just removing people from it if they'd been waiting too long.
People may think it’s an overstatement. But I think it’s not.
This kind of projects are where AI is most helpful. Long tedious and highly testable projects like ports or legacy system replacements where humans have to grind through millions of lines of code without really thinking are the perfect target for AI.
I once did a C to C# port of several math libraries, and while I was able to automate most of it, it still required a ton of manual work. I bet if I had Claude, I could do what I did in 3 months in a day or two.
This is what I was thinking too. Ports are probably where AI has the one of the biggest time savings over humans.
Although also agree, that even if human's did it, they probably would try to automate as much as possible and speed up development, and maybe that original 5 year estimate was if it was done completely manually.
I would love to know the responsible people at Asana who approved this joint PR. Like, do you not get the blatant second degree negative implications of what’s stated here, albeit manipulated and hyperbolic, for a project management software?
> An engineer checked progress twice a day and reviewed every proposed change.
It doesn’t sound like that much changed. If one engineer can really review all the changes in under two weeks, the original estimate of 5 years of engineering work has to be waaaaaaay off.
When everything is “estimates”, anything sounds amazing.
The impressive thing here is that Asana has such a poorly run engineering org that replacing their testing framework was estimated to take 5 years and $6 million.
The fun part is how the article just says they removed "Enzyme" and there is nothing else mentioned that can make you gauge the technical complexity of what was done here
Comparisons such as 5 years and $6M are meaningless. Instead, they should compare with other AI, such as how much it would have cost using self-hosted open-weight models, or using a API that offers open-weight models etc.
The latter sounds about as attractive to me as the former, actually more because my customers are mostly hoping for their bugs to be fixed , not for the latest novel feature.
These conclusion can be obtained by any individual adult with a command of the English language and average written text comprehension abilities within 2 minutes. That’s how long it takes to read and understand the case study.
So, why did you surrender your cognition to an LLM? Do you have some form of mental impairment?
Unfortunately, this is the type of statements we can't verify. I'm not sure why these types of news are still coming out when we all have AI at work.
Whenever someone does such a huge drastic change like this, it's by ignoring a large chunk of code that most people were afraid to touch for good reasons. Now, that code is gone, AI is celebrated, things will break, people will work very hard in the background to fix it, with no fanfare.
> I'm not sure why these types of news are still coming out when we all have AI at work.
Because OpenAI is burning $15 billion/year, and outrageous stories like those get parroted in the media. It's free marketing for a company desperate to get middle management to believe that a $500/mo subscription is absolutely crucial for every single employee.
> parroted in the media
ugh just recently I saw a news article gushing about how AI had helped with some health/medicine study, and various patient organizations were all like "oh yeah this is a Good use of AI" and then I click the link to read the study and it's decision trees and clustering on a tiny dataset that you could analyze with a ten year old laptop.
I mean, sure at some point decision trees and clustering were called "AI", but the way the article was written you'd think OpenAI and Anthropic were responsible for the advancement of medicine.
yeah but you'd need skills to operate the 10 year old laptop. probably have to copy and paste and understand excel and maybe whatever the hell a confidence interval is
this allows said middle managers to do the analysis themselves and feel mostly confident in the results.
I can for sure say that this kind of statement is exaggeration if not a blatant lie. It may work for PR but it gives a wrong impression/ideas. Now engineers should be ready to be questioned by their management for all the rewrites that they have been estimating in months if not years.
But you _can_ verify if you had bothered: https://asana.com/inside-asana/migrating-off-enzyme-2-weeks
Key statement:
> But at the rate we were going, we were still roughly five years from finishing.
Everyone has seen how this sort of thing comes about: it's meaningful work for engineering but never business/product critical so it just drags along.
Seems like this time around someone just went "I wonder if we could do it this way" and it worked. Perfect example of ditching sunk-cost and starting from scratch. Great outcome for them.
This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting.
> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).
Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.
Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".
> where afraid to touch for good reasons
This appears entirely unreasonable.
Normally the reason is not good. The reason is that unit testing is missing or that downstream effects are not entirely mapped out.
Exactly activities that traditional software developers are loathing because they are boring and mentally straining.
> This appears entirely unreasonable.
Seems a bit strong.
Unit testing can get you some of the way, but its not a full-on all-case guarantee. Sometimes the code is encapsulating some particularly complex system/behaviour. Sometimes the reason is interop/compatilibity issues or some kind of politics.
P.S. you managed to introduce a typo in your quote (were –> where)
Oh, I did notice t introduce the typo. The commenter corrected after quoted.
What appears unreasonable is that the reasons must be good.
This comes directly after they wrote "statements we can't verify".
Why is it that we can not verify the statements, but we can believe the reasons that people will not touch the code base to be "good"?
Or the reason is that something is just unfeasible with the existing architecture, or the reason is that building this would break important technical assumptions... All kinds of things.
Exactly. You can simply close all open tickets with <WONTFIX> and claim that you've cleared 5 years of engineering work in 5 minutes. It doesn't mean anything.
> people will work very hard in the background to fix it, with no fanfare.
This. It looks like AI companies have managed to use this for their advantage. Can't really blame them.
It’s just sales claims. Do you see that “contact sales” button? You’re not the target of that. Deranged AI-psychotic C-suites with a two-digit IQ and too much confidence are the target.
"I took a speed-reading course and read War and Peace in twenty minutes. It involves Russia."
― Woody Allen
I think there might be a mixup here.
You see: normally one reads for recreation or enlightenment.
If that same one would approach their job as a recreational activity, or solely for enlightenment, you might not have that job for very long.
1. I am not sure what your profession is, but I spend most of my reading time at work, with code, emails and so on. Unfortunately I don't have much time to read for recreation these days.
2. It's a joke! Relax.
Yes, is you see my response to one of the parents, it should be apparent that it was clear to me.
ha:) Think you taking this person's statement too literally:)
No, but the parent is a half baked snark of the type that instills a type of false memes.
Latently they say that it is a quality of itself to do the work.
That is a reasonable statement, however, I am sure they also extend it to harvesting, baking etc. As to not use mechanical help.
Heck, if they actually believed their own statement then they'd not even be commenting here, as that require immense amounts of mechanised help.
on the other hand if you all you deliver are half-backed products you're bound to create a lot of grief, and move yourself up the corpo ladder.
Phrased this way, my aspiring product leader hat activates: that 5 years of work cannot be of much value to a company worth billions because a 5 year estimate is the lowest possible qualification you can make. Its a “will never happen” shelving.
all work is stack ranked against other opportunities. no matter how many teams help to parallelize, leadership basically comes down to ranking the stack correctly. Valuable work will be at the top by any means necessary. “mvp”, probes, task forces, code yellows, and so on.
edit: it’s even a backhanded compliment to agentic coding. Asana a public company considers a $12k outlay for something they’d never spend real resources on worth a case study =|
yeah this was my read.
either it was a hail mary 5 year change-the-market offering, or it was something that staff can "get around to" whenever they have the cycles, and would, at current rates, take like 3-5 years to do.
was the tool worth 12k? if it's not a line in capex or opex budgets it's value is $0
Apparently the task was to remove dead weight. Haven't read what they replaced "Enzyme" with.
I did subagent based removal tasks a few times. These were the ones that required the least amount of input or thinking from me, because the requirements are abundantly clear. "Remove this part of the code without breaking any other part or by porting the tests done with it onto another system."
In these situations the code acts as the bookkeeping ledger itself, and coordination complexity is almost a no-brainer.
If you ask Sol or Claude how much time it will take to implement a plan they just came up with, they usually advise a timeframe in the weeks or months - assuming, I suppose, that human programmers will be building it. And then you ask the model to just "do it" and it takes an hour or two. I always find this entertaining.
Funny but shows how it doesn't have a useful world model
It does, assuming "world" is made up of humans (which it is, in its training data). It does however not have any model whatsoever of itself, apart from some superficial "I am a helpful assistant built by $X".
I find that the models have increasingly started throwing out made up amounts of time around, as if they wouldn't be aware that the user is already using Claude Code or Codex.
It's like every time they make a plan, there's something about things taking "a week or two", "month of focused work", or whatever.
This is something that would've been RL'd out a long time ago if it wasn't great for business.
A model isn't "aware" of anything. It's trained on things, and that training doesn't include time estimates for working with AI because that didn't exist yet.
There's pre and post training.
What I was going after with "aware", is that the actual people working at the companies, training the models, are aware that people aren't mostly going to be implementing the plan by hand, if they've already made the plan in Claude Code or Codex. As for a specific Claude / GPT instance, "aware" would definitely be the wrong word choice there, but the instance does have its stats and environment information in its context window, unless you specifically remove it.
Either way: Training does include estimates on working with the model, and adjustments of the model itself based on that. That's literally what RLHF is.
It's straightforward to have a portion in post training that aims specifically at the model being able to give better estimates on how long that model takes to complete a certain type of task.
In all honesty I don't think it's possible to put this into the model because it's not actually sufficiently understood yet by anyone or anything. AI capabilities are changing too fast, and so is the way AI is used (worst case, even including financial limitations that are now increasingly appearing.)
Every part of training a model have the timing data observed + saved on multiple different axis, it's one of the most inherent parts of the training process.
When the post training run is nearing its cutoff point, there's a massive amount of data on how long coding tasks take to complete by that model in the golden format of "task -> time task took to complete", separable to whatever amount of subtasks, in the same format. With the parts from the end of the dataset being useful for evaluating the finished model's capabilites, whether that data is then fed back into another step in post training or not.
Completely separate from even the actual training: If you have a model proactively giving estimates that are an order of magnitude wrong, you can already fix the worst of it as of this moment by just changing the system prompt. It's a dirty fix, but it's the type of fix that has been used by Anthropic and OpenAI since forever when a model is dishing out blatantly wrong outputs.
Not sure if I'm misunderstanding your point?
The misunderstanding might be that I'm thinking about how long it takes a human + an AI to get something done, not an AI by itself. My current understanding is that it's still not generally viable to have LLMs perform architecture/design tasks (it might work in the short run but degrades roughly exponentially in the long run, accruing serious tech debt). Most things I'd consider actual "projects" involve at least some level of design work as well, and that'd need to be included in the estimate.
(And also prompts will need to be refined, etc., unless you have amazing prompt skills it won't immediately deliver what you wanted. Even if it did, the work to put together the AI inputs also needs to be included.)
⇒ I don't think it's easily possible right now to give a time estimate for **the full picture of** a project to be implemented with AI assistance.
From the article:
> For comparison: the previous plan was expected to take at least five years and estimated to cost roughly $6M.
If that's an estimate from an LLM, those have always been way off for me. I'm constantly amazed whenever something that an agent estimates would take weeks ends up being completed in an hour.
The LLM is reasoning about estimates from its training data... which is to say, from human engineering timescales.
I suspect the labs could improve the models such that they are estimating these sorts of things but they don't prioritize doing so (or perhaps RLHF selects it away) because, as you say, it feels amazing to do a week's worth of work in an hour.
Is this the react test framework enzyme? To me, $6 million and 5 years seems like a very high estimate for this even without any automation (AI or otherwise) for even relatively large applications. How could they possibly have a test suite this large?
The more tangible example for me around LLM enabling what's previously long if not impossible tasks: pgrust. https://github.com/malisper/pgrust
They probably meant 1 employee part time for 5 years on a project that was never a priority.
The British government cleared a large chunk of 5 years of the NHS (hospital) waiting list by, in part, just removing people from it if they'd been waiting too long.
The headlines feel similar
"Their old testing tool, Enzyme, had fallen out of active maintenance and was becoming a blocker to modernizing Asana’s frontend stack"
Non prod code, good use case.
Also a very trivial piece of work for an LLM to do - with a measurable end goal.
Hopefully a human did give a quick look to make sure it didn’t just delete the tests.
So clearly it doesn't need the workforce required to delivery 5 years man work, as per project estimations, thus brace for impact.
I don't trust anything OpenAI says anymore.
People may think it’s an overstatement. But I think it’s not.
This kind of projects are where AI is most helpful. Long tedious and highly testable projects like ports or legacy system replacements where humans have to grind through millions of lines of code without really thinking are the perfect target for AI.
I once did a C to C# port of several math libraries, and while I was able to automate most of it, it still required a ton of manual work. I bet if I had Claude, I could do what I did in 3 months in a day or two.
This is what I was thinking too. Ports are probably where AI has the one of the biggest time savings over humans.
Although also agree, that even if human's did it, they probably would try to automate as much as possible and speed up development, and maybe that original 5 year estimate was if it was done completely manually.
Cant wait some open model guys doing the check on this work from the security point of view.
I would love to know the responsible people at Asana who approved this joint PR. Like, do you not get the blatant second degree negative implications of what’s stated here, albeit manipulated and hyperbolic, for a project management software?
You see, I call bs on that. Not on the part that agents are an accelerator, they might be, but at the part that it was a 5 year project.
We engineers, had the power to dictate how our days will look, and we were able to give ballooned timelines to give us room to braeth.
This has changed. The power is no longer in our hands, for good or bad.
I suspect it as a two month project without agents. Max.
> An engineer checked progress twice a day and reviewed every proposed change.
It doesn’t sound like that much changed. If one engineer can really review all the changes in under two weeks, the original estimate of 5 years of engineering work has to be waaaaaaay off.
A nitpick. While this definitely does not apply to the problem described, some changes are indeed mostly alignment, not implementation.
When everything is “estimates”, anything sounds amazing.
The impressive thing here is that Asana has such a poorly run engineering org that replacing their testing framework was estimated to take 5 years and $6 million.
Also I think the marketing post compares removing a test framework (how is that not trivial?) with replacing it with another.
The fun part is how the article just says they removed "Enzyme" and there is nothing else mentioned that can make you gauge the technical complexity of what was done here
Pretty big self-own
"An engineer checked progress twice a day and reviewed every proposed change. Simpler instructions worked better than a more elaborate setup."
Oh dear, poor Asana customers...
I believe this about as much as the claim that the Bun rewrite cost $165k
When "5 years of work" take two weeks, you know two things. The estimates were wrong, and nobody will need a project management tool soon.
Idunno, maybe the point of the project management system was to be able to make grandiose statements about your progress
5 years? Bruh. There’s no way from scratch it would take 5 years.
[flagged]
[dead]
[flagged]
Comparisons such as 5 years and $6M are meaningless. Instead, they should compare with other AI, such as how much it would have cost using self-hosted open-weight models, or using a API that offers open-weight models etc.
It is a good PR for Asana. I checked this with Gemini & here is what Gemini said - https://share.gemini.google/B8nocObwgiih
Summary: The Takeaway Did an AI generate 5 years' worth of highly creative, novel software features from scratch in two weeks? No.
Did an AI complete 5 years' worth of tedious, widespread code-migration technical debt in two weeks? Yes.
The latter sounds about as attractive to me as the former, actually more because my customers are mostly hoping for their bugs to be fixed , not for the latest novel feature.
Who cares what an LLM hallucinates about it?
These conclusion can be obtained by any individual adult with a command of the English language and average written text comprehension abilities within 2 minutes. That’s how long it takes to read and understand the case study. So, why did you surrender your cognition to an LLM? Do you have some form of mental impairment?
I was curious to see what one LLM thinks about claims from another LLM.