> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common.
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time.
Some packages may also drop support for whatever distro you're still using, and the mirrors may disappear (e.g. older Python versions, ROS, etc.).
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
Worth mentioning CI is not part of production outages!
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.
Of course, anything deployed for use by the target customer is a production system. And anything that is under development is a dev system. Pipeline itself is not under development and it is in use by the target customer (developer). What's the confusion here?
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
There's more to making an argument than stating your conclusion forcefully and repeatedly.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem?
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
There is a weird duality when analyzing this... On the one hand, I think factory assembly lines are just a high bar to clear as as metaphor for internal processes. Almost no business actually runs a super organized backend workflow; there is always "I thought you followed up with the customer" "I thought you did"
On the other hand if the email server is down that's just as big a problem for any biz as if customer facing product was not working so some workflow things are already treated as production tier priority
this muddying of terms is not useful. No, my IDE breaking down is not a production system failure. It's a development system failure and sure, that can very easily have consequences on the production system but usually not immediately. It should be obvious to everyone how having a functional hammer is going to be important if you want to fix a house. There is no need to start calling the hammer the house.
At my (now former, thanks layoffs) company, we used to gripe about getting nontechnical management to understand how stupid it was to skimp on pre-prod funding . . . because pre-prod was our prod.
You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on.
And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd.
Tldr? Go ahead and page yourself for non-prod alerts. No one will stop you.
Being a production system doesn't make it a priority.
> A team with a broken development pipeline can’t produce software, and must treat this as a production outage.
While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing.
To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes.
Increasingly though founders and executive team are non-technical and simply don't care until it shows up on a graph that piques their interest (the ones with $ somewhere).
> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common.
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're still using, and the mirrors may disappear (e.g. older Python versions, ROS, etc.).
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
Worth mentioning CI is not part of production outages!
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.
Of course, anything deployed for use by the target customer is a production system. And anything that is under development is a dev system. Pipeline itself is not under development and it is in use by the target customer (developer). What's the confusion here?
> What's the confusion here?
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
There's more to making an argument than stating your conclusion forcefully and repeatedly.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem?
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
https://dora.dev/guides/dora-metrics/
From the article title, I was hoping that the author would argue that software development could be modeled as one of these. https://en.wikipedia.org/wiki/Production_system_(computer_sc...
There is a weird duality when analyzing this... On the one hand, I think factory assembly lines are just a high bar to clear as as metaphor for internal processes. Almost no business actually runs a super organized backend workflow; there is always "I thought you followed up with the customer" "I thought you did"
On the other hand if the email server is down that's just as big a problem for any biz as if customer facing product was not working so some workflow things are already treated as production tier priority
this muddying of terms is not useful. No, my IDE breaking down is not a production system failure. It's a development system failure and sure, that can very easily have consequences on the production system but usually not immediately. It should be obvious to everyone how having a functional hammer is going to be important if you want to fix a house. There is no need to start calling the hammer the house.
At my (now former, thanks layoffs) company, we used to gripe about getting nontechnical management to understand how stupid it was to skimp on pre-prod funding . . . because pre-prod was our prod.
Sounds like someone wants to go on-call.
You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on.
And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd.
Tldr? Go ahead and page yourself for non-prod alerts. No one will stop you.
2:34 AM
zZzzZ...
2:35 AM
ALERT #42 - CRITICAL
!!!
ALERT #42 - CRITICAL - Joan forgot feir password to npm
Help feir!
Being a production system doesn't make it a priority.
> A team with a broken development pipeline can’t produce software, and must treat this as a production outage.
While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing.
To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes.
Increasingly though founders and executive team are non-technical and simply don't care until it shows up on a graph that piques their interest (the ones with $ somewhere).