While the speed of prototyping and even shipping to production has increased, I have been asking myself at what cost? I see a lot of garbage being shipped. Not because the code quality is bad, because execution has become cheap now. Ideas even though crap, are getting prototyped. Things which look effective on the surface, but has real UX problems in the underneath, are getting prioritised because someone in the room can talk better and enrol a leader to align with the idea. Good old user research or talking to users to validate ideas, iron out issues in the user flows has become too slow for the new process!!
What are people doing with prototypes afterward? Do you end up shipping it as is to production? What about at work? Are the prototypes useful in that context?
I've started making most of my prototypes single HTML documents with inline CSS and JavaScript, because a single file is a lot easier to store somewhere and share and will probably keep on working forever (browsers are really good at backwards compatibility).
I chuck some of them on my public tools.simonwillison.net collection, others in their own GitHub repos with GitHub Pages enabled, and some I just share in a Gist (served via gisthost.github.io) or stick in a public S3 bucket so I have a URL for them.
If a prototype is against an existing project sometimes I'll leave it to silently rot in a branch on GitHub.
You know how when you finish prompting some code generator to build something, and you look over what it has built and feel a sense of emptiness even if it does what you want? I think about what I wish the prototype looked like, and basically start describing details that I expect to exist (think longer versions of e.g. “this should be using our internal graph library, and I figure we can model this task as a traversal, how far have you strayed from this and why?”) and let the agent analyze what it built against my expectations. I’ve spent hours in conversation just “refining the context” this way, and then I channel that into an update process. I figure the prototype is just about proving out behavior, and this next phase is about refining it into the pieces I’ll use elsewhere. It’s kinda fun, I’d absolutely burn out a coworker if I grilled their PRs the way I roast AI contributions :P
I use AI mostly to prototype features in my existing projects. If I have an idea, I use the AI to implement it and try out different ways in which it could be implemented. Then I throw away the code and mostly write the code manually, with AI used primarily for review or docs.
> What are people doing with prototypes afterward?
I think what people ARENT doing is interesting.
Usability isn't even in your list, it is not something most people even think of.
The best thing you can do with a prototype is give it to (potential) users and observe what they do with it. Just because you think that you're clearly communicating the intent of your system does not mean you are.
I'm truly hopeful that AI will open a new of prototyping. Back in the day, prototyping was how you figured out what to build, you'd very deliberately toss the entire first (or second!) version, and you'd plan to do that.
Most places I've worked, devs were basically afraid to prototype
Either you would get chastised for wasting time with prototypes, or worse, your prototype would end up in production
I think the software industry really needs a cultural reset to embrace slower and deliberate development to build quality, but unfortunately AI has us racing recklessly in the wrong direction
I am so tired of it. Are there any companies out there that actually give devs time to build quality software anymore? I'm so burned out of the "move fast and break everything" grind
Quality must come from engineering. If you’re depending on a product manager to ask you that you can improve the quality of the code, you already lost.
So it requires soft skills, proper framing and ability to iterate quickly on quality-related tasks without leaving junk and multiple-versions behind.
But I completely understand push back for “doing improvements developers want to do”: A lot of developers confuse quality with familiarity or even complexity/verbosity. So business people have a reason to be reluctant.
And as an engineering manager I also had to push back several times. The thing that makes money is not the place to learn new skills, for example.
But is it really any faster than using an already existing code generator/scaffolding tool? How do you know your project isn’t just a regurgitation of another repository? Would it be just as fast to clone some existing project and hack on it?
These are the questions everyone seems to be ignoring and saying “only LLMs can make projects quickly” but ignoring everything those LLMs are built on (your llmis probably calling a code gen tool).
For the at work side, I personally haven’t experienced any disadvantages or missed any project deadlines because I didn’t use an LLM, so what does velocity get me? Thumb twiddling time?
I was thinking the other day how much better Drupal is. Want a online store? A few commands and bam, online store. Want a newspaper? A few commands and bam, newspaper with publishing workflows, user management, and caching.
Using coding agents isn't much different. There are several things the models are trained to do very well and a few commands will get something. If the developer wants to move the project beyond that, it requires domain knowledge and a lot of hacking.
I wonder if the coding agents will move towards the Drupal model where they create interchangeable components with common interfaces. Like Drupal the coding agents never provide anything truly inovative that hasn't been done before.
Drupal and WP etc all have plugins to switch stuff on in minutes, however, customising and making it as your client wants would take a lot of time. WP shops we work with for clients (we need to integrate some times) take weeks to get some plugin to do what they want by adding tags and config options.
It might centralize around a specific framework but I think part of the problem is that people want to generate their own framework or at least not care about what the framework is/does/can do. They treat the LLM as the framework which can be non-deterministic and structureless.
> But is it really any faster than using an already existing code generator/scaffolding tool?
Yes, very much so. Our team was fast with those tools and created many of our own before this LLM AI (we used other AIs though to go faster), however it still took weeks to months from idea to launch; the same complexity now takes days, including everything. We already had rigorous processes and those really help now moving at speed. No way anyone can beat this except better AI.
But “are you really moving at speed after you generate the majority of your application?” is my other point. If you were to start working somewhere with an existing product the changes you would apply are more than likely incremental. What is the advantage of using LLMs to change 1-10 lines of code on average? How do you measure the ROI for that?
What did the time savings gain you? A quicker release date? How can you prove that? “This would have taken weeks” is the old problem of project time estimation. How can I take any engineer seriously that they think they know it saved weeks?
Usually I know exactly what I want before hand. What structs. What protocols. How I want the event bus layered and what threads need to exist. And what make targets I want. So generally the generated code is strictly bound to my design pattern. Then it's all a matter of running it. To put it bluntly I'm running benchmarks and testing it while you're still deciding what to name your files.
And? What advantage does that have for you over me when it comes to a personal project? What does that velocity get me? More time to foolishly rewrite/regenerate the already built software from scratch? I don’t spend a lot of time naming my files personally.
So you do no validation of the code that’s generated? Just asking because you didn’t state that as a step in your process. You’re prototyping to running then you’re missing a big step that will most likely cost you later.
Why does it matter how much time I spent writing code for a project I’m most likely either not sharing or if I am sharing it can be obtained for free? Which market am I rushing to? Bluntness doesnt seem to be an advantage other than bragging.
Your tone makes me think you already decided that agents aren't worth your time, but I'll give it a try anyways.
I work as a DevOps engineer and have been using agents exclusively to code since the beginning of the year. Agents are really nice to quickly craft utilities to speed up planning. For instance I had it create a small cli for me that'll pull my cards from azure DevOps, load them as json, markdown and csv, and push updates once I'm done. Then I'll load into context transcripts of meetings and other written requirements, cross with current state of repos, to have meaningfully conrextualized work items without me having to implement these myself. I'll just have a long chat with the agent exploring these cards and defining the necessary refinements for description and acceptance criteria than I jusr push them all at once. Anything you can think of you just ask for the agent, so for me I don't trust code, so I'll have all my clis be no-op by default, so they will first print all they'll do and if I think the changes make sense I approve them and let the script commit to the canonical board.
Working with cloud consoles like Aws in general is a huge hassle, so crafting quick inventory utilities and tools for correlating data is a breeze.
Now the work itself is mainly ci pipelines, terraform files and automation. For these I'll base the agents on the specified work items and enrich them with my own understanding of the problem. I then launch the agents and read the agent output attentively. This is very important. You can't just prompt and leave, you need to be present all the time so you can steer the agent into solving the right problems. At the very least you need to review all the changes after an implementation session is done when you came back from making coffee. Many times it tries to create meaningless abstractions or very complicated solutions that I know can be done better. Or I have a different idea of how to organize the project so I do many follow-up sessions to refactor code.
In my personal projects I do a lot of small utilities. I spent some weeks designing and polishing a replacement for zurg and debridmediamanager the way I like it to be, simple and to the point, also tightly integrating them with jellyfin https://gitlab.com/gabriel.chamon/buzz
I have my own micro desktop environment on top of hyprland called Archie which recently I've been redesigning and improving a lot with agents https://gitlab.com/gabriel.chamon/archie
I have been improving my fork if gamma-launcher so that installing and managing the game on bazzite is simpler and more automated than relying on workarounds for workflows intended for windows https://gitlab.com/gabriel.chamon/gamma-launcher
Now for how I approach developing with agents. I think it's really important to get your constraints sorted out as soon as possible, so have your agent create a CI pipeline for code quality testing, like with ruff, pyright and pytest, to control style, code consistency and cyclomatic complexity. Put in the AGENTS.md explicit instructions that the agent must run these tools at the end of every coding session. If adopting a new project, use the agent to explore the code and see which refactoring points are worth tackling. Agents really thrive on good codebases, so this first code quality improvement pass is a must.
To sum it up, with agents you give up writing code manually for reading lots of code, exploring the domain with the help of the agent and architecting the solution at a strategic level. You trust the agent but you also verify. And lots and lots of manual testing. My personal take is that I'm infinitely productive now, only constrained by how much code and agent terminal output I can read, and also by the rate limits of the model providers and mental fatigue.
While the speed of prototyping and even shipping to production has increased, I have been asking myself at what cost? I see a lot of garbage being shipped. Not because the code quality is bad, because execution has become cheap now. Ideas even though crap, are getting prototyped. Things which look effective on the surface, but has real UX problems in the underneath, are getting prioritised because someone in the room can talk better and enrol a leader to align with the idea. Good old user research or talking to users to validate ideas, iron out issues in the user flows has become too slow for the new process!!
What are people doing with prototypes afterward? Do you end up shipping it as is to production? What about at work? Are the prototypes useful in that context?
I've started making most of my prototypes single HTML documents with inline CSS and JavaScript, because a single file is a lot easier to store somewhere and share and will probably keep on working forever (browsers are really good at backwards compatibility).
I chuck some of them on my public tools.simonwillison.net collection, others in their own GitHub repos with GitHub Pages enabled, and some I just share in a Gist (served via gisthost.github.io) or stick in a public S3 bucket so I have a URL for them.
If a prototype is against an existing project sometimes I'll leave it to silently rot in a branch on GitHub.
You know how when you finish prompting some code generator to build something, and you look over what it has built and feel a sense of emptiness even if it does what you want? I think about what I wish the prototype looked like, and basically start describing details that I expect to exist (think longer versions of e.g. “this should be using our internal graph library, and I figure we can model this task as a traversal, how far have you strayed from this and why?”) and let the agent analyze what it built against my expectations. I’ve spent hours in conversation just “refining the context” this way, and then I channel that into an update process. I figure the prototype is just about proving out behavior, and this next phase is about refining it into the pieces I’ll use elsewhere. It’s kinda fun, I’d absolutely burn out a coworker if I grilled their PRs the way I roast AI contributions :P
I use AI mostly to prototype features in my existing projects. If I have an idea, I use the AI to implement it and try out different ways in which it could be implemented. Then I throw away the code and mostly write the code manually, with AI used primarily for review or docs.
> What are people doing with prototypes afterward?
I think what people ARENT doing is interesting.
Usability isn't even in your list, it is not something most people even think of.
The best thing you can do with a prototype is give it to (potential) users and observe what they do with it. Just because you think that you're clearly communicating the intent of your system does not mean you are.
I'm truly hopeful that AI will open a new of prototyping. Back in the day, prototyping was how you figured out what to build, you'd very deliberately toss the entire first (or second!) version, and you'd plan to do that.
High quality ensued. Usually ;)
Most places I've worked, devs were basically afraid to prototype
Either you would get chastised for wasting time with prototypes, or worse, your prototype would end up in production
I think the software industry really needs a cultural reset to embrace slower and deliberate development to build quality, but unfortunately AI has us racing recklessly in the wrong direction
I am so tired of it. Are there any companies out there that actually give devs time to build quality software anymore? I'm so burned out of the "move fast and break everything" grind
I understand both sides.
Quality must come from engineering. If you’re depending on a product manager to ask you that you can improve the quality of the code, you already lost.
So it requires soft skills, proper framing and ability to iterate quickly on quality-related tasks without leaving junk and multiple-versions behind.
But I completely understand push back for “doing improvements developers want to do”: A lot of developers confuse quality with familiarity or even complexity/verbosity. So business people have a reason to be reluctant.
And as an engineering manager I also had to push back several times. The thing that makes money is not the place to learn new skills, for example.
Ask on sprint planning if time can be set aside to spike out a proof of concept, and then you go do that prototyping in the sprint.
Has this (for me, normal) process really been that arduous in your past jobs? It's a slam dunk to leadership, as we do this to corral time wasted.
All these companies want devs with top-engineering talent and coding skill, but then fire them because they aren't using LLMs enough.
Prototype? Why stop there..
But is it really any faster than using an already existing code generator/scaffolding tool? How do you know your project isn’t just a regurgitation of another repository? Would it be just as fast to clone some existing project and hack on it?
These are the questions everyone seems to be ignoring and saying “only LLMs can make projects quickly” but ignoring everything those LLMs are built on (your llmis probably calling a code gen tool).
For the at work side, I personally haven’t experienced any disadvantages or missed any project deadlines because I didn’t use an LLM, so what does velocity get me? Thumb twiddling time?
It reminds me of Drupal circa 2009.
I was thinking the other day how much better Drupal is. Want a online store? A few commands and bam, online store. Want a newspaper? A few commands and bam, newspaper with publishing workflows, user management, and caching.
Using coding agents isn't much different. There are several things the models are trained to do very well and a few commands will get something. If the developer wants to move the project beyond that, it requires domain knowledge and a lot of hacking.
I wonder if the coding agents will move towards the Drupal model where they create interchangeable components with common interfaces. Like Drupal the coding agents never provide anything truly inovative that hasn't been done before.
> If the developer wants to move the project beyond that, it requires domain knowledge and a lot of hacking.
Reminds me a bit of this blog post[0].
I remember doing a Drupal project around that time and being astonished at how powerful it was.
I also remember feeling more like a technician connecting various components than like a software engineer, writing code.
I totally saw the value for the client but I really disliked my experience, so I avoided it afterwards.
0: https://www.rickmanelius.com/p/the-website-rfp-and-the-impos...
Drupal and WP etc all have plugins to switch stuff on in minutes, however, customising and making it as your client wants would take a lot of time. WP shops we work with for clients (we need to integrate some times) take weeks to get some plugin to do what they want by adding tags and config options.
It might centralize around a specific framework but I think part of the problem is that people want to generate their own framework or at least not care about what the framework is/does/can do. They treat the LLM as the framework which can be non-deterministic and structureless.
> But is it really any faster than using an already existing code generator/scaffolding tool?
Yes, very much so. Our team was fast with those tools and created many of our own before this LLM AI (we used other AIs though to go faster), however it still took weeks to months from idea to launch; the same complexity now takes days, including everything. We already had rigorous processes and those really help now moving at speed. No way anyone can beat this except better AI.
But “are you really moving at speed after you generate the majority of your application?” is my other point. If you were to start working somewhere with an existing product the changes you would apply are more than likely incremental. What is the advantage of using LLMs to change 1-10 lines of code on average? How do you measure the ROI for that?
What did the time savings gain you? A quicker release date? How can you prove that? “This would have taken weeks” is the old problem of project time estimation. How can I take any engineer seriously that they think they know it saved weeks?
Considering that engineer never reliably estimated anything beyond a few days remotely accurately before… but now they can…
Yea cause it's done while you're still reading the docs for your code generator. lol
Code generators are usually one short command. It’s less typing than a prompt would take.
That statement tells me you have less experience with code generators than me and I'll just leave it at that.
Then how much time do you spend debugging and fixing the generated code? lol
Usually I know exactly what I want before hand. What structs. What protocols. How I want the event bus layered and what threads need to exist. And what make targets I want. So generally the generated code is strictly bound to my design pattern. Then it's all a matter of running it. To put it bluntly I'm running benchmarks and testing it while you're still deciding what to name your files.
And? What advantage does that have for you over me when it comes to a personal project? What does that velocity get me? More time to foolishly rewrite/regenerate the already built software from scratch? I don’t spend a lot of time naming my files personally.
So you do no validation of the code that’s generated? Just asking because you didn’t state that as a step in your process. You’re prototyping to running then you’re missing a big step that will most likely cost you later.
Why does it matter how much time I spent writing code for a project I’m most likely either not sharing or if I am sharing it can be obtained for free? Which market am I rushing to? Bluntness doesnt seem to be an advantage other than bragging.
Your tone makes me think you already decided that agents aren't worth your time, but I'll give it a try anyways.
I work as a DevOps engineer and have been using agents exclusively to code since the beginning of the year. Agents are really nice to quickly craft utilities to speed up planning. For instance I had it create a small cli for me that'll pull my cards from azure DevOps, load them as json, markdown and csv, and push updates once I'm done. Then I'll load into context transcripts of meetings and other written requirements, cross with current state of repos, to have meaningfully conrextualized work items without me having to implement these myself. I'll just have a long chat with the agent exploring these cards and defining the necessary refinements for description and acceptance criteria than I jusr push them all at once. Anything you can think of you just ask for the agent, so for me I don't trust code, so I'll have all my clis be no-op by default, so they will first print all they'll do and if I think the changes make sense I approve them and let the script commit to the canonical board.
Working with cloud consoles like Aws in general is a huge hassle, so crafting quick inventory utilities and tools for correlating data is a breeze.
Now the work itself is mainly ci pipelines, terraform files and automation. For these I'll base the agents on the specified work items and enrich them with my own understanding of the problem. I then launch the agents and read the agent output attentively. This is very important. You can't just prompt and leave, you need to be present all the time so you can steer the agent into solving the right problems. At the very least you need to review all the changes after an implementation session is done when you came back from making coffee. Many times it tries to create meaningless abstractions or very complicated solutions that I know can be done better. Or I have a different idea of how to organize the project so I do many follow-up sessions to refactor code.
In my personal projects I do a lot of small utilities. I spent some weeks designing and polishing a replacement for zurg and debridmediamanager the way I like it to be, simple and to the point, also tightly integrating them with jellyfin https://gitlab.com/gabriel.chamon/buzz
I have my own micro desktop environment on top of hyprland called Archie which recently I've been redesigning and improving a lot with agents https://gitlab.com/gabriel.chamon/archie
I have my own agile based methodology for creating and managing work items with tight integration with gitlab https://gitlab.com/gabriel.chamon/orisun
I have been improving my fork if gamma-launcher so that installing and managing the game on bazzite is simpler and more automated than relying on workarounds for workflows intended for windows https://gitlab.com/gabriel.chamon/gamma-launcher
Now for how I approach developing with agents. I think it's really important to get your constraints sorted out as soon as possible, so have your agent create a CI pipeline for code quality testing, like with ruff, pyright and pytest, to control style, code consistency and cyclomatic complexity. Put in the AGENTS.md explicit instructions that the agent must run these tools at the end of every coding session. If adopting a new project, use the agent to explore the code and see which refactoring points are worth tackling. Agents really thrive on good codebases, so this first code quality improvement pass is a must.
To sum it up, with agents you give up writing code manually for reading lots of code, exploring the domain with the help of the agent and architecting the solution at a strategic level. You trust the agent but you also verify. And lots and lots of manual testing. My personal take is that I'm infinitely productive now, only constrained by how much code and agent terminal output I can read, and also by the rate limits of the model providers and mental fatigue.
I think you're right
Imagine if instead of f AI generated code, we all just started copying and pasting code from open source repos.
Imagine my velocity! I cloned the Linux kernel in seconds!
Instead we're basically doing exactly that, except through an AI remixer.
It leaves a very sour taste in my mouth