> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
Those of us who care. Those optimizing for economic returns or out of ideology are not. More carnage will be required for incentives to swing back I think.
"Security is for losers"
> Andy Yen, the CEO of Proton, had a message for an employee in the company’s security Slack channel who urged the company to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” In a follow-up message, Yen reiterated his point: “only losers think like that.” His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”
Moving faster is good. But only if it means sending more agents (and humans, that part is important) to do the same amount of testing in less time. Dropping critical work is not that.
Vote with your feet or prepare for your eventual breach notification and data loss event. You control who you choose to do business with, and who custodians and processes your data, it is a choice. Make good choices.
The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
Either he's been like this the entire time, trying to crack the whip and failing, and thus a bad leader who has repeatedly failed to get results, or he's (as his employees have said) very deep in AI Psychosis and completely abandoned his principles.
It doesn't matter which it is, they're both extremely damning for the company and his leadership.
But why stop there? Aim higher!! End of all world’s wars, no more hunger anywhere anymore, free abundant clean electric power for everyone everywhere!
Jokes aside, you can’t get anywhere near zero fault unless to impose strict operating regulations , requirements and procedures and regular checks… just like in aviation.
And even there, boeing manages to fuck it up somehow.
People always find new ways to make mistakes. The approach in aviation is to investigate, identify root causes, and do what is possible to prevent the same mistake happening again.
There are no guarantees we'll ever reach perfection.
Maybe it's different in your neck of the woods, but all the government portals I use tend to have access to rather sensitive PII, and I would really prefer their ops teams be up to snuff as a result.
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
This is kind of an interesting thought (underneath the grumpiness, which I guess is where the downvotes came from). Programmers seem to hold aviation processes in very high regard, but their industry is also under cost pressures. The “UX” of riding in a plane does kind of suck, I wonder if some of that comes from the costs involved.
Not too surprised to see pressure to reduce the number of pilots.
Aviation is different because their rent-seeking is enforced by the required legal hedging. I'm not sure the same incentive structure exists in the software side of things.
In other words, we might need to find some hobbies.
Rent seeking has a specific definition and it fits here. When Autopilot does almost all the critical work needed (besides announcing frivolous stuff in carbon-based voice), and you have mandatory seats for humans to babysit the Autopilot mainly to satisfy some law, the term fits.
I am not convinced humans can do much better in "disasters" than Autopilot, despite the emotional stories to the contrary (survivorship bias is one hell of a drug).
This take is so ignorant and provably false. I can't tell if you're trying to ragebait or if you actually believe this.
The autopilot does not do "almost all critical work", not even close. It only does some of the routine work (definitely not all of it). But the real critical work is when stuff goes wrong, dealing with emergencies.
The inherent complexity of aviation means that there are so many different failure modes and combinations thereof that no traditional system could ever handle all possibilities and edge cases.
I'm not going to claim that an AI could never do it. But as of right now, with current technology, well-trained pilots are absolutely critical to safety in even the most advanced airliners.
Also, the survivorship bias argument runs against you. Most events where pilots save the day are completely invisible. Like catching an incorrect clearance, a misconfigured approach, dealing with a technical malfunction, etc.
And even when disasters happen, it is very rare that simply enabling the autopilot would have saved the day. In fact, there are plenty of examples where the opposite is the case, and dropping to a lower level of automation would, in hindsight, have been better.
I work at a startup and often deal with every part of the stack--programming, deployment, db, cloud environment, networking blah blah. AI agents have just gotten superhuman at computer use. I will never be able to use AWS CLI as well as claude does. I can spend time poking around in AWS, manually reading logs, looking at routing tables, cross checking with data in db, whatever, or I can give claude readonly access to all of that stuff and it will solve the issue literally 10x faster than I could, write a detailed report I can pass on to whoever, etc. Is it sometimes BSing? yes, but it just keeps getting better.
All that said, skill atrophy is real. Thinking about getting back on the interview circuit is p scary because of it. I dunno wtf is gonna happen.
SRE was always about understanding; understanding data flow, understanding bottlenecks, understanding logs, understanding application architecture, understanding end user functionality, understanding break points, SLIs, SLOs. Understanding how to mitigate, gracefully deteriorate, restart, restore, recreate, scale, etc.
There is no understanding with AI-generated applications, full stop.
For the last 10 years, I don't think I met a lot of senior architects or dev who could understand all that.
Companies loves to have "specialized experts" who skilled up in one afternoon in their field.
It might be different in startup, but in consulting world in relatively large enterprises, it's just a fucking joke.
Agree up to certain levels but your examples are too specifc.
Understanding should come in different levels abstraction of what you are building,and managing.
There are different levels SREs, from beginner to say Fellow level.
Do you expect the Fellow level to understand nitty gritty details of each component his apps or services are interacting with and how each of them start? No. Especially when they have to deal with hundres or thousands of services each of them hosted by different teams, orgs, or different companies.
junior SREs, yeah probably got their hands dirty with those nitty gritty details you listed because they tend to deal with smaller or lower levels of components
So my point is, I agree withyour "Understanding" but not specific details as they differ per one's level of experience
There's nothing unique about junior SREs. They're replaceable with almost any "vanilla Dev". You don't really build a unique experience or expertise until Senior+.
idk, I develop a lot of connected systems: Pis, Macs, lights, Google speakers, custom ESP32 boards. Understanding the capabilities and communication methods to pool them together is still very much part of what I do. If you're thinking one prompt -> production via AI, sure. But I don't think we are anywhere near that because there are so many moving pieces which still require human sign off: legal, design, QA, etc. To me, it just shift ups the level of understanding rather than replaces it.
SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
DevOps was a reasonable idea, but it rarely actually manifested. If a company had an actual "DevOps group" separate from the developers, by definition they weren't doing DevOps.
Part of the problem, and what ultimately drove me out of coding altogether, is that Ops is a specialized skillset completely orthoganal to the business of building applications. And, I was really bad at it due to a complete lack of interest. As cloud environments get more and more complicated one ends up spending more and more time dealing with that at the expense of building things.
Conversely, there are lots of folks who love ops type problems and are not well suited to building apps. So, it makes sense to put them on separate teams where they can each do the thing that they're best at, rather than fumble through the parts of the jobs they hate. But yes, at this point it's not really DevOps anymore, it's just ops.
IMO it was a worthwhile experiment that probably went too far before being rejected.
Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful. Especially for certain kinds of products, if executed with discipline you can get really good engineering reaction time with no degradation of quality, and often with much cheaper infra costs.
The problem is what you said though. People with both skillsets are rare, and combining them is legitimately a third skill on its own. And even "both" skillsets is a simplification once you start accounting for data, security, compliance, large scaling DB work etc.
It was mostly a dead end for these reasons but it was worth exploring for its potential. And some genuinely good engineering techniques went mainstream out of it, not a total waste.
> Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful.
You’re so right. This is the key, whether a team is doing devops or not.
One of my major contributions when I took on the CTO role was getting dev, ops, and support teams to talk about the pain points. There were so many easy wins to make things run better because the teams didn’t understand the difficulties the other teams were going through.
Isn't that just anyone with an "architect" job title though? I don't think that kind of dev is particularly rare at all.
I've found that "ops" has become far too overabstracted and jargony. Their moat is endless "certifications". Their "experts" are the dogmatic personality types who try everything in their power to prevent their feet from ever touching the ground like it's lava. They lack the kind of dev or sysadmin experience where you have to stand under your own constructions like a Roman engineer.
Any formerly-known-as "full stack" dev who kept at it for the last decade and a half would run circles around anyone in ops. That's every senior dev working right now who doesn't like to manage people.
It used to be that the ones who went SRE just wanted a pay bump for a bullshit job of relatively light coding puzzles and parroting stuff in meetings, but I guess that well has dried up with "AI".
Coming from the Ops side, “full stack” usually translates to “our entire infrastructure stack is managed AWS services, we don’t know how they work”.
It turns out about as well as if you set the Ops team to shipping features.
Eg I’ve seen more than one full stack dev team build a synchronous RPC-over-message-bus framework because they can’t figure out how to make HTTP or actual RPC calls reliable. It is hands down the most poorly and overengineered thing I have ever seen on a service that takes single digit qps.
The ops teams fight that shit tooth and nail because if it blows up, they’re the ones who get called. And it is 100% too late to fix it by then. Ripping it out will take months, and the dev team will say they’ll do it, but they never will because the VP is leaning on them about some feature.
You either say no early, or you are forever on the hook for whatever abomination the intern has cooked up.
Right but the dream and promise of devops was that early-to-mid career line engineers could do these things, that these weren't rare skills and combining them wasn't particularly challenging. This turned out not to be the case, these skills are the requirement not the outcome.
But if you have the budget and connections to hire a department of people with these skills and experience, and a problem interesting enough to keep them engaged in the day to day building, and a leader capable and charismatic enough to align them all into the business's goals... you should do that.
> Ops is a specialized skillset completely orthog[o]nal to the business of building applications.
Exactly this was the idea: persons in a DevOps role have skills in both areas, i.e. are very wide skill set, so that one can assign them to very diverse tasks for more flexibility. Additionally, these persons can implement synergies to make things more efficient in areas that intersect both development and operation.
No, what you descrive is SRE. DevOps was the idea to give people that work 100% as developers the ability to do some amount of ops work (let say 5%), as a way to break the silo between Dev and Ops that existed in older organizations.
Where it worked well is the "devops" or sre/sysadmin would sit in the same team and take away the rough edges of the infra.
conversly they would also keep an eye on the web people and see what they are doing and why, learning the tooling they use and try and rationalise it so they don't install 50 messaging queues.
At one point I was very hopeful that the progression from Infrastructure as a service to Platform as a service was going to simplify things. Instead we seem to have ended up with mind boggling complex services.
Some places do have really well built out platforms where logging, instrumentation, databases, deployment methodologies, IaC, etc have well-built happy paths that exclude a bunch of accidental complexity.
I think a lot of other places are either too small or grow too quickly with the wrong set of values and admit too much into their "platform" which had very little design to begin with.
Real DevOps is actually just Dev - in that the Devs are doing the Ops. The people called DevOps are just there to backstop the devs oncall when shit hits fan and you need someone more experienced in ops / incident management to escalate to.
Yes, big companies have a tendency to over-structure things. Devops? Sounds great, let's have a dedicated team for it. Agile? Absolutely, and to make sure it's done right, we need a rigid process around it.
I need the company to have skin in the game, so I prefer that I am entirely unreachable outside M-F 9-5. Otherwise they just exploit people who submit far too quickly to being on-call 24/7. It also gives me motivation to make my stuff work as best as possible so that I'm not being annoyed by management about my non-existence outside core hours.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
Google's SRE was always about building infrastructure to make things reliable, and started in a world where LAMP on bare-metal + DNS load balancing was common. It makes sense to have a separate org/team/role responsible building real load balancing with health checks and failover, figuring out how to do zero-downtime updates, etc.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
> Real ones are oncall for the systems they build.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
> Real ones are oncall for the systems they build.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
there's eventually a point in organization size where it makes sense to have some kind of centralization to make it easier for teams to actually be on call for the systems they build without grinding them to dust.
SRE at google works closely with devs to architect and build systems together. Most of them are on the SWE ladder, are able to bug fix and scale directly. They're not just ops.
The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
If SRE cannot throttle the release cadence or fix bugs or change architecture in binaries, it's not really SRE.
When I worked at Google SREs only handled legacy products that were maintained but not in active development. SRE was its own vertical separate from SWE and they would only take on maintenance of a new service or product if SWE could prove to them it was stable.
Oddly the industry mostly rejected googles application of SRE even as they adopted the term. I've only seen Google use it to refer to engineers maintaining / operating someone elses code.
It’s cause they don’t know what they are talking about.
Easier to shit on people and things with hot take then actually have a grounded and coherent argument or realise that different companies/teams have different constraints, problems and environments they operate as well as different solutions.
It turns out it's easier to hire people who specialize in scalability, reliability, debugging of distributed complex systems, cost ops, efficiency gains, etc. than it is to instantly ensure every single engineer that works there is an expert in all those things, without anybody on staff who is an expert in them. Much like database skills, IT security skills, etc.
It's still something everyone can theoretically learn, of course, and someone who works in that role is most effective by instilling those disciplines in others as they go, and making everyone successful.
"Zero" Interest Rate Period, during the COVID pandemic, where most governments offered very cheap bonds or printed money to reduce interest rates. This caused a lot of investment as money was "cheap" for investors and needed to find a worthwhile investment in order to grow faster than inflation.
This caused a lot of money to be poured into all kinds of projects, not all of which had a positive return on investment.
We should do it again but the right way this time.
Give literal zero interest loans from a state bank based on social value and economic priorities, just like they do in China. And set up an extreme progressive tax (90% top bracket) on returns that are paid out to owners to make sure everything gets invested back into production.
More investment is the driver of economic growth. All the bad parts of ZIRP came from letting companies use the money for unproductive activity. If it's used efficiently the downward pressure from higher product supply will be stronger than inflation from more money in the system. And you get more demand in the labor market leading to higher wages.
> SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
Yea cause you wouldn't escalate to 50 people on the weekend when one person can do it. Op is basically saying they never worked on a big enough system.
If you look at the history of SRE, it was distinctly NOT sysadmins, Google specifically wanted software engineers hired to automate scaling as well as sysadmin type ops work.
Kubernetes is probably more responsible for deskilling, or at least constraining, the expertise, of SRE/devops/platform/infrastructure engineers than anything.
I don't think that's a bad thing, it makes infrastructure more consistent and less weird (assuming you understand k8s) but it is a very high level of abstraction and for a bunch of people is just a black box.
Continuing the never ending trend of Sys Admins/DevOps/SRE job title holders with skills in specific tools rather than foundational skills always wondering why they are out of a job every 5 years.
That’s what companies hire for. I’ve learned those skills and no one ever really wants them.
If you build tools with that context, they’ll have to replace you with someone that has that context. There’s few of them, and they’re expensive.
It’s easier to hire folks that should be interns or juniors that know that one tool, and then replace the whole department every decade.
It’s why everything is YAML now, despite it being horribly verbose and clunky. Can’t have our people needing to know how load balancing works, just add another 95 lines of YAML boilerplate so they don’t need to learn 5 lines of nginx config.
I never understand this sentiment. I do SRE/DevOps/Platform Engineering/Infrastructure Engineering, and I definitely do need to know how underlying technologies work. Even when everything is kubernetes and YAML.
If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that. Your company can also choose to outsource complexity to the cloud, and you’ll just be writing YAML. Then there’s also companies where you host everything yourself AND you have large/high workloads, then you still need people with understanding.
With your type of comments, it feels like you’re pretending a developer who writes WordPress websites does the same work as someone doing Erlang for instance. You’ll have different skill sets for every type of engineer, and they’ll all be necessary.
That's reinforced by how hiring is done though. I rarely if ever run into companies that hire for fundamentals. Only the Google tier of companies do that.
> Tier 3 incidents: Novel, complex, cascading failures stay human-led, with AI providing supporting context.
> But here’s the catch.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
How did they do this before AI? I have done a bunch of complex production level debugging and these Tier 3 incidents are always "starting from scratch" never seen before issues. The main "intuition" I built is persistence and verifying/refuting random hypothesis. In other words, I think we can build an intuition for Tier 3 issues only by engaging with Tier 3 issues.
Although, psychologically, some success in T1 and T2 issues is what gives us the confidence to even approach the T3 issues.
I don't even know if this is an AI thing. I've worked at places that have varying levels of operational maturity, and some of the ones that had very old SRE orgs also had a lot of their SREs operating mostly in the tier 3 given by the article.
One of these orgs had basically all of the noisy pages under control through automated remediation and a culture that pushed towards aggressive automation of pages that had the same root causes. Almost everything I was paged for was some novel or uncommon fault condition. Developing an intuition about how the system worked (well, more specifically, how the system failed) took a lot longer because most things I had to look at were some edge case failure.
I think people are getting funneled into a new career path of being a technologist or technology director. People who can ask the right questions at the right time and direct attention to the appropriate areas given the goals and feedback.
People don't appreciate how much of engineering is just as you've described.
When they imagine someone using AI (except for themself), they imagine the least curious, least inquisitive vibe-coder. And they invent the idea that the only way to be someone who asks questions or remains curious is to have paid the dues that they see themself as having paid. Therefore the world must always need them.
Well, all you have to do now is imagine the vibe-coder who has the same curiosity and inquisitiveness that you had especially when you were starting out and weren't yet a professional programmer. The only difference is that now you don't have to bootstrap a whole software career to understand how things work; you can skip that and merely ask.
Admittedly, a lot of my identity was wrapped up in those hard earned skills. I have empathy for those struggling to chart a new path for themselves. It does not, however, change the fact that tools evolve. Ultimately, using tools effectively is how we are measured and while you may have needed a shovel yesterday, today you have an excavator.
This has always been the most valuable part of the job if you ask me. Its just that the other parts of the job were quite hard to get right letting people build a career out of that specialization.
We use AI at work, Claude specifically, and it’s faster than avg SRE for trivial tasks like cross signals correlation. But it requires a lot of effort to build a harness from your company contexts for it to work like an SRE. And it’s just a tool so if you have to build your knowledge to make it useful, otherwise it’s just a blunt. Or in other word, incompetent people are incompetent no matter of tools
What contexts would the AI model need for analyzing, let's say, cluster-wide incident other than the observability data that such systems anyway collect already?
Is it really that complicated to set up? A basic version is make everything work from a CLI in terminal coding agent of choice (probably already mostly there), hook it up to alerts, and let it run in a corner?
Not OP, for me the issue is nuance of all of your infrastructure, application and outside layers. There's a lot of documentation and such, but often times things don't work as expected to a degree and it requires filling in the blanks correctly. The amount of times I've had an AI Agent misinterpret an alert and assume it was XYZ when it was some random layer it had no idea about is often.
It can certainly get closer but unless your tech stack is a webserver and an application it can get complicated.
I like this analogy that I've come up with: Non-writers can produce impressive articles that are unbearable to read the same way non-programmers can produce impressive software that is unbearable to use.
I dug into general AI deskilling a couple days back[1] - no good signs anywhere, but there's a few things we can do like make concerted efforts to do programming (like a coding hour every week), do a Leet code question, and getting the agent to quiz before merging something (stolen from Notion).
I've been having this same gnawing feeling. I just started reading the book "The Collision" by Eric C So which promises to cover this subject. I haven't finished it but seems decent so far.
It's so depressing a thought that if I continue on this path, in a few years, let alone a decade, I'll be utterly unable to produce or even understand code. I realize the skill may not be very economically useful, but losing it is still a depressing outcome that's preventable in theory. At the same time I'm feeling a deep apathy towards building anything. Playing a Zachtronics game is the farthest my internal motivation can push me towards...
I’m personally using it to augment my coding more than replace it. I find joy in it.
I use my agent to prototype a rough version that mostly works, and then I iterate through (with or without my LLM) to clean it up. Then back to the top with a new feature until I’m satisfied with the project and code.
I do tend to like it. My real selling point is that I’m not “locked in” to any choices until I’m happy with how it looks. I used to have to build the thing to see how it looks and acts, and going back on those choices was painful.
I do think there will be value in it, but I’ve been wrong before. My agent occasionally makes stupid choices that make every prompt after that 4x as hard to fix. Like I screwed up a pydantic model in an unreleased library and it supported a “legacy” path no one would ever use for like the next 5k LoC. By the time I noticed, legacy handlers were like 25% of the codebase.
That’s gonna burn a looot of tokens if you scale it up to a million line of code app.
I don't know if it's true. I didn't program at all between 2000 and 2015 and I've been able to restart without too much effort. I forgot many things, but I found it relatively easy to recover...
I'm more worried about people that did never program and started directly with LLM support...
Nice video, but I have said it before, the difference between just “vibe coding” and reliably using AI is the specs you need to write first and the tests after. If you start prompting the end goal/s the AI will basically guide you on what needs to be done and you just accept it as long as it runs, and this is following what most software developers were doing before LLM, the agile way, the quick ship way, the daily standup and ship first discus later way, the code pairing way, which worked mostly (even though I hated personally) in teams, so these developers are replacing what they were used to do with an agent and that’s the mistake. However, if you go with traditional engineering ways, waterfall, specs based, blueprints, and then tests, you control the agents and guide them not the other way around, in your specs you define the tiniest possible thing to what library to be used or excluded, the methods and approaches, among others, the agent here will produce the syntax and help you in the tests, just like in other engineering aspects how machines didn’t take over, they follow your specs and frameworks. The syntax is the past, AI can produce better syntax than any human alive or all combined, if you think otherwise you are delusional and still didn’t move on, it’s time to stop the “developer” mindset and do the engineering one.
I personally have a project that have been spending 2 months now making the specs and blueprint, more than 1000 files defining every bit including the future features too, and not a single code has been written yet, later, an agent will use this to produce the syntax and all the needed tests to have exactly what’s been needed.
Let's not rush to such judgment - our desires change as we age, the field has changed (including what nose-to-the-grindstone coding looks like) and burnout affects many.
Plus, some people enjoy the rigors of solving in-the-weeds code problems and others enjoy building systems-level designs. This was always true.
You guys act like the React component nonsense from the past 10 years is the same as the code I was learning at age 16. It's by far and away completely different. And none of it has been audited by Daybreak.
I'm very happy to let the robot handle any incidental React or other extraneous tools that end up adjacent to the core work I'm doing. It avoids getting blocked without interrupting teammates or spending time and cognitive capacity on stuff that has no lasting value in my work.
It's fairly hard to understand to me, but also entirely unsurprising that swapping one occupation for another would be interpreted as a net loss at individual skill level.
You guys are still doing things, correct? Is my experience so outside the norm, or is stuff still challenging?
* AI-resistant organizations led by graybeards who refuse to send telemetry off-prem (to either observability vendors or to inference providers) and will be dragged, kicking and screaming, into the future, over their cold, dead bodies. There's a lot of large, conservative organizations in this category like banks and governments, where budgets are rubber-stamped and Don't Break It If It Ain't Broken attitudes are tacitly approved of.
* AI-accepting organizations where Ops will go through the same great narrowing that Dev is going through now - build the dark factory and let it run. You do still need people who know what good architecture looks like and to shepherd changes to improve the overall system to make it more observable, scalable, and maintainable. You do still need Non-Functional Software Engineers who don't answer to Product, for the sheer political reason that you need people who have a long-term perspective, compared to Product that has a short-term perspective. The business will make decisions like "Vendor X is too expensive so we're migrating to Vendor Y" and you will need somebody to drive the agents to make that happen in a way that respects availability and compliance requirements. You probably don't need, in the long-term, a whole team of people to handle it, let alone a whole group or a whole division.
At the end of the day... focus on being a strong engineer. Somebody will always be needed to take care of the trees.
Previous discussion on an extremely similar article: https://news.ycombinator.com/item?id=49574167. This one even name-checks The Ironies of Automation in pretty much the same way.
I feel like I did re-read the same article, or at least a version of it. Is this how content marketing operates these days? Find a popular blog post, rewrite it and attach your company's name to the end?
If it deskills that guy who reaches for complex bash when they should reach for golang but they don't want to bother with writing the build layer in the image, then I'm all for it.
> Some immediate targets include multi-signal correlation across metrics, logs, and traces, autonomous root-cause analysis for partially understood failures
AI is already pretty good at this in "SRE" workflows.
Is HN upvoting just because they agree with the title?
Clearly the "author" didn't even put in the effort necessary to make a real title. Not "SRE persists but a skill crisis looms" not "SRE careers remain but competency lowers in spite of AI" not "AI lowers SRE skill but demand is steady". Moreover, hasn't this type of insufferability been dramatically reduced in recent models? It screams that in addition to being slop, it's outdated.
I stopped reading halfway because it doesn’t describe competent SREs in an organization where the culture values reliability.
> Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.
Let’s stop here and ask, why do we not fix the underlying issue? Alert noise? Design better alerts. Runbook execution? Make it a cronjob. Known issues? Why aren’t people fixing these known issues? Root cause suggestion from historical data? Sorry if root cause A is in the historical data you expect problem A is fully solved so it wouldn’t appear again in the next incident. Restarting things? Why doesn’t your health monitor restart things automatically?
SREs also need to climb up the abstraction ladder. Instead of using AI to assist with grunt work, use AI or human thinking to build more reliable systems with fewer incidents. Every incident should be novel.
Deskilling is not necessarily a bad thing, if what you refer to as skills are mundane things that should be automated. I do agree with the author in that people are becoming lazy when it comes to critical thinking and evaluating the decisions or work that AI presents to them. These are people who have little passion to do what they do, and for whom their role is primarily a chore.
I don't think this is as black and white. You skill up by spending time with the domain. Of course, you learn better when you are involved in situations or jobs that actually matter. But getting rid of all the "mundane" distances you from the nitty-gritty and you lose that feel for the problem.
Right, there's a reason students that never dealt with a real problem, even when they get As, aren't as good as professionals. You need to do real work to get really good.
No it's forcing everyone to realize all the jargon was made up to sell books and land speaking engagements. Money from patents (2016 Alice ruling upending that whole scam).
Yall this was always just a money making scheme.
Never was it meant to carry humanity into the future.
Here's a thought experiment: why is it that I can't do long division?
The Calculator!
Calculators deskilled long division!
Thing is though: who gives a damn about long division? The real task is your taxes or making a massive BI report for your presentation tomorrow.
AI "deskilling" due to automating things makes old key skills become something you can just ask for and have done for you. Why bother staying skilled at those things when you can broaden? It clears the way for what used to take forever and be a specialist skill, to become something to steamroll through on your way to finishing a much greater outcome/task. It's the micro way that, in the macro, humanity becomes far more capable as a species. The gates are being flung wide open and the gatekeepers are being lapped and they have blogs!
It's not a tragedy, it's universal access to leverage.
Well, I'm sorry to hear that. I have a calculator and some pretty neat computing hardware. If I go into the attic and search long enough I can probably find my old sliderule. I can also do long division, as can all of my kids. I think it is an essential skill to understand all basic mathematical operations, and to be able to use them on small data sets. If I didn't know how to do division without a calculator I would consider that a failure of my basic education.
I think a closer analogy would be forgetting basic arithmetic as a concept. Imagine how hard it would be to work with more complex math without understanding what division is. How do you prompt the genie, or understand its outputs, without the foundational knowledge?
Is basic arithmetic something that can be forgotten? Studies have suggested that even infants have an understanding of basic arithmetic. It seems to be something innate.
But as we're only imagining anyway, let's forget about the genie for a moment. If we did lose our sense of basic arithmetic all of a sudden what would life at its core even be like? If there were two apples on a table you wouldn't be able to comprehend what happens if you took one. Would we even be able to function like that?
> in the macro, humanity becomes far more capable as a species
Not humanity, a small number of people in this planet, will benefit about this, because now they no longer have to depend on under payed workers to do the job, and will use AI to make the lives of all other human being poorer.
I think that we need to oppose this tech dictatorship, first thing stopping using AI and thus stop depending on an handful of companies that are able to provide it, that while they gain more and more power we all gets poorer.
What was wrong in using our brain, in the end? I still use my brain on my job and I always will, maybe one day they will say that there is no more need of computer experts because AI can do everything (still people bring me their laptop to reinstall Windows, something I've done for 20 years and that AI didn't change), I will just find another job but I will refuse to use this AI shit.
Of all the comments on this post, this landed the most.
I feel there is a potential to "distill" the frontier models into useful tools that can run on smaller hardware. Or distill that knowledge to produce hardware, compute or otherwise. Iff one day the frontier model become inaccessible, the tools that were carved with them may remain valuable.
They will remain most valuable if they are designed in a way that is maintainable even without LLMs. If not, they can be studied as artifacts, or reference designs and then recreated using human intelligence.
The analog I'm playing with:
You can rent time at a high end machine shop and use the best CNC machines to produce extremely valuable tools. Once you leave, you don't have access to those tools, so it is advantageous to produce durable artifacts that are valuable on their own — on a different frontier.
Imagine you're going beyond the frontier physically, exploring new parts of the earth or space. You can't bring the giant machine shop, let alone the machinery of society, the entire economy etc. What do you bring? What do you build now while you have access to the best machines to take with you?
I recently won the red ribbon, second place, at the Lambtown fiber festival, for my lovely skein of undyed handspun yarn. Weaving it is next. However, I am probably an outlier, and should not be counted.
(It was a full 250g of this fantastical kempy Herdwick, absolutely unlike anything you can get commercially, and I'm honestly more proud of it than of anything I built in six years working at NVIDIA.)
This is a poor analogy because I don't think your inability to do long division will cause as much damage as an inability to bring the site back up in an emergency situation.
Except as every damn AI app says: AI is prone to making mistakes. If AI deskills you with a non-zero chance of getting the problem wrong, that seems deeply, profoundly bad.
You can type the same long division problem into a calculator 65,000 times and you will get the right answer 65,000 times.
Computing tapped out on sources of leverage for solving deterministic problems a while ago. Computers got fast and our algorithms got really good.
We're on to the next frontier, and it's a bit weird. Jagged frontier and all. But it's opening up new worlds.
Luckily loads of problems are amenable to a non-deterministic approach - even better, many have lots of solutions. SRE is in that bucket. There are as many ways to SRE as there are SREs
(source: one of the first SREs at Uber, co-invented Chaos Engineering as a sub-branch of SRE, hosted first chaos eng conf at Uber, played a key role in Uber's global SRE and incident response "Ring0" initiative, managed many SRE teams over a period of roughly 7 years before a career pivot. You could say I know a thing or two about SRE.)
To SRE well you look at the situation and pull from a huge range of tools. With such a broad range of solves, you're better to focus on the outcome: did we achieve our SLOs? Are they aligned with customer expectations? Focus on defining that and providing the right support for it to be achieved and you'll be fine - you might even get a surprise.
I'm (still) a software engineer, and I've found it crucial to solve some LeetCode problems from time to time. It's honestly refreshing to write code yourself, make your own mistakes, and not end up low-key mad at a robot for failing to understand your lousy description because, apparently, it still can't read your mind.
I wanted to use this post as a source to write on my web server project's blog on why debuggable web infrastructure matters even if AI SRE agents exists, but the idea got delayed and I ended up posting it on HN... :')
This is happening to SWEs too. If you have a certain amount of experience, you are capable of critically analyzing AI output. In Big Tech parlance, these are the Senior SWEs and higher. You require little to no supervision and can work independently.
But how do you make Senior+ SWEs? By having a pipeline of juniors who get trained and mentored. That pipeline is cooked now.
There is going to be a brief period where some like Staff+ SWEs are going to make a lot of money by being AI specialists. But the flood waters are rising for everyone. Those opportunities are going to shrink.
It's eventually going to be a problem that the ecosystem has died because there's no new blood and the people there lack the experience and critical thinking to understand what's being done.
I can use AI on topics I know because I can spot the errors. LLMs lie a lot and get things wrong a lot. What happens when nobody can check the output?
The point I got from the article was that AI is a cause for skill atrophy for any level of seniority, and tends to worsen over time. I can see this happening to myself in some areas, while in others I’m able to learn new skills I’m interested in.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
Ime this doesn't happen or reflect the reality of work. I had one mythical job where I wasn't directly on call and only received escalations and didn't have a problem with debugging whole system issues. I still worked on the whole system day to day.
Also I feel like this post is also my frustration with sre. In people's minds it's just the job that deals with issues pumped out by swes and they have to work magic because no one else can be bothered to try at their job. This just reinforces that idea.
Seeing this posted a week after I had to emergency push cve fixes to 1200 servers in production is pretty funny.
Every person on my team is capable of networking, product engineering, database, job processing and we have command of the entire stack. We will push updates for CVEs with stuff we manually compiled. We own the build system and have the ability to change product as needed. We all have experience in product and we're all capable of pushing fixes as needed. Frankly the SRE team is by far the smartest team I have ever worked with at any company I've ever worked at.
Is your org at the point where it would make sense just to upstream patches for FOSS projects and pull rather than patch and push to just your infra?
Just a matter of curiosity for me as my career progresses, where I think that is the ideal, but I don't work in a heavily privatized nor regulated space at the moment of posting.
I did read your latest blog post looking for an answer, but it wasn't clear to me from that reading.
The ubiquity of this rhetoric is not an indication of its correctness. Repetitive tropes like "deskilling", "losing the ability to think" reflect a deep seated distrust of AI alone. No matter how many people parrot these fears the fact remains that you choose your attention. My manager has started coding, heavily, aided by frontier LLMs. He instinctively knows his boundaries and defers much of the more complex work to his engineers. We use the clankers too. Have we been deskilled? Hardly appropriate. Trans-skilled, or learning a new emphasis built upon our long experience? Much closer to the truth.
I work a "dev"ops job, and like yeah, I'm pretty sure this field is even more fucked than software engineering is.
IaC codebases, configs, utility scripts are relatively tiny, and fit into context windows whole. Models are eating them for breakfast.
They're really good at incident response. Sadly, much better than our dedicated colleagues from <location> ever will be. They'll chew through cloud service logs, application logs, application code, iac code and config, CI output, etc. to get whatever is needed, cause wide knowledge is no issue. Even disassembling stuff is no barrier, tracing things down from app level, to runtime level (e.g. JVM), to native calls.
They're really good at handling deployments, end to end infra setup and migrations, following checklists. Hell, they don't even really need checklists. Something that cannot be said about other dedicated colleagues from <location>, who cannot breathe without a checklist, will skip over steps, or otherwise very creatively misinterpret them or improvise something wicked.
They're really good at spinning up mad nice dashboards, and keeping business types happy with impressive looking write-ups they'll never read. It's really refreshing to not have to write those write-ups anymore (or the aforementioned checklists).
It's straight up both not looking good and looking better than ever. Having it be fully hands off remains an issue though.
> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
Not losing production data and not getting breached is also something we try to make zero mistakes at.
Those of us who care. Those optimizing for economic returns or out of ideology are not. More carnage will be required for incentives to swing back I think.
"Security is for losers"
> Andy Yen, the CEO of Proton, had a message for an employee in the company’s security Slack channel who urged the company to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” In a follow-up message, Yen reiterated his point: “only losers think like that.” His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”
Proton CEO Andy Yen: "This will be the last year that we hire engineers." - https://news.ycombinator.com/item?id=50295112 - October 2026
Manifestos, Fear, and 'Loser Thinking': Inside Proton's AI Reckoning - https://news.ycombinator.com/item?id=50289692 - October 2026
Good luck.
Edit: There appears to be some consequences for acting like this and those actions being made public.
Proton's CEO Cancels AI 'Re-Interviews' - https://news.ycombinator.com/item?id=50307183 - October 2026
That's the sort of thing you really don't want to hear from your encrypted mail company.
Moving faster is good. But only if it means sending more agents (and humans, that part is important) to do the same amount of testing in less time. Dropping critical work is not that.
I can imagine people are quickly looking for an alternative now. I wouldn't want to hear this from any company I'm a customer of.
Vote with your feet or prepare for your eventual breach notification and data loss event. You control who you choose to do business with, and who custodians and processes your data, it is a choice. Make good choices.
Wow this makes me hesitate to renew with them
The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
Either he's been like this the entire time, trying to crack the whip and failing, and thus a bad leader who has repeatedly failed to get results, or he's (as his employees have said) very deep in AI Psychosis and completely abandoned his principles.
It doesn't matter which it is, they're both extremely damning for the company and his leadership.
But why stop there? Aim higher!! End of all world’s wars, no more hunger anywhere anymore, free abundant clean electric power for everyone everywhere!
Jokes aside, you can’t get anywhere near zero fault unless to impose strict operating regulations , requirements and procedures and regular checks… just like in aviation.
And even there, boeing manages to fuck it up somehow.
Aviation is an example of how there's no way to have zero mistakes, or even zero fatal mistakes.
People always find new ways to make mistakes. The approach in aviation is to investigate, identify root causes, and do what is possible to prevent the same mistake happening again.
There are no guarantees we'll ever reach perfection.
Maybe it's different in your neck of the woods, but all the government portals I use tend to have access to rather sensitive PII, and I would really prefer their ops teams be up to snuff as a result.
> I'm fine if some of the stuff is having scheduled or unscheduled maintenance.
You might be, and your customers may not! And your competitor could be more than happy to offer a more reliable service.
I’m using Elixir for every project now specifically because the cost is negligible now.
Just like pilots, SREs will have to spend a minimum number of annual hours in the simulator to keep their skills sharp.
SREs will be de-materialized and converted to a Docker image that will be run on Kubernetes, in the prod env.
Exactly as foretold: https://www.youtube.com/watch?v=0otdTJa9IUc.
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
This is kind of an interesting thought (underneath the grumpiness, which I guess is where the downvotes came from). Programmers seem to hold aviation processes in very high regard, but their industry is also under cost pressures. The “UX” of riding in a plane does kind of suck, I wonder if some of that comes from the costs involved.
Not too surprised to see pressure to reduce the number of pilots.
Aviation is different because their rent-seeking is enforced by the required legal hedging. I'm not sure the same incentive structure exists in the software side of things.
In other words, we might need to find some hobbies.
In what way is requiring someone holding the title pilot to know how to manually fly a plane in case of disaster “rent-seeking”?
Rent seeking has a specific definition and it fits here. When Autopilot does almost all the critical work needed (besides announcing frivolous stuff in carbon-based voice), and you have mandatory seats for humans to babysit the Autopilot mainly to satisfy some law, the term fits.
I am not convinced humans can do much better in "disasters" than Autopilot, despite the emotional stories to the contrary (survivorship bias is one hell of a drug).
This take is so ignorant and provably false. I can't tell if you're trying to ragebait or if you actually believe this.
The autopilot does not do "almost all critical work", not even close. It only does some of the routine work (definitely not all of it). But the real critical work is when stuff goes wrong, dealing with emergencies.
The inherent complexity of aviation means that there are so many different failure modes and combinations thereof that no traditional system could ever handle all possibilities and edge cases.
I'm not going to claim that an AI could never do it. But as of right now, with current technology, well-trained pilots are absolutely critical to safety in even the most advanced airliners.
Also, the survivorship bias argument runs against you. Most events where pilots save the day are completely invisible. Like catching an incorrect clearance, a misconfigured approach, dealing with a technical malfunction, etc.
And even when disasters happen, it is very rare that simply enabling the autopilot would have saved the day. In fact, there are plenty of examples where the opposite is the case, and dropping to a lower level of automation would, in hindsight, have been better.
Pilots are nothing near powerful enough to seek any rent on that situation.
Instead, they are the weakest part on that market.
It is a good industry to learn from. I'm not suggesting SW should become exactly like aviation.
I work at a startup and often deal with every part of the stack--programming, deployment, db, cloud environment, networking blah blah. AI agents have just gotten superhuman at computer use. I will never be able to use AWS CLI as well as claude does. I can spend time poking around in AWS, manually reading logs, looking at routing tables, cross checking with data in db, whatever, or I can give claude readonly access to all of that stuff and it will solve the issue literally 10x faster than I could, write a detailed report I can pass on to whoever, etc. Is it sometimes BSing? yes, but it just keeps getting better.
All that said, skill atrophy is real. Thinking about getting back on the interview circuit is p scary because of it. I dunno wtf is gonna happen.
SRE was always about understanding; understanding data flow, understanding bottlenecks, understanding logs, understanding application architecture, understanding end user functionality, understanding break points, SLIs, SLOs. Understanding how to mitigate, gracefully deteriorate, restart, restore, recreate, scale, etc.
There is no understanding with AI-generated applications, full stop.
For the last 10 years, I don't think I met a lot of senior architects or dev who could understand all that.
Companies loves to have "specialized experts" who skilled up in one afternoon in their field. It might be different in startup, but in consulting world in relatively large enterprises, it's just a fucking joke.
Really? I think that in most orgs up to medium size, you expect senior and especially staff engineers to have that scope
They wouldn't be able to draw it all onto a whiteboard from memory. But give them a half day and they will have mapped out the data flow
Agree up to certain levels but your examples are too specifc.
Understanding should come in different levels abstraction of what you are building,and managing.
There are different levels SREs, from beginner to say Fellow level.
Do you expect the Fellow level to understand nitty gritty details of each component his apps or services are interacting with and how each of them start? No. Especially when they have to deal with hundres or thousands of services each of them hosted by different teams, orgs, or different companies.
junior SREs, yeah probably got their hands dirty with those nitty gritty details you listed because they tend to deal with smaller or lower levels of components
So my point is, I agree withyour "Understanding" but not specific details as they differ per one's level of experience
There's nothing unique about junior SREs. They're replaceable with almost any "vanilla Dev". You don't really build a unique experience or expertise until Senior+.
idk, I develop a lot of connected systems: Pis, Macs, lights, Google speakers, custom ESP32 boards. Understanding the capabilities and communication methods to pool them together is still very much part of what I do. If you're thinking one prompt -> production via AI, sure. But I don't think we are anywhere near that because there are so many moving pieces which still require human sign off: legal, design, QA, etc. To me, it just shift ups the level of understanding rather than replaces it.
> To me, it just shift ups the level of understanding rather than replaces it.
This just sounds like convoluted way of saying you understanding much less than you would otherwise.
Agreed 100% with this.
> There is no understanding with AI-generated applications, full stop.
Full slop, you mean.
SRE: https://wikipedia.org/wiki/Site_Reliability_Engineering
Thank you! I wish people would define acronyms more.
Agreed. Neither the subject article nor the one it linked to defined this acronym, leaving the reader to guess.
SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
DevOps was a reasonable idea, but it rarely actually manifested. If a company had an actual "DevOps group" separate from the developers, by definition they weren't doing DevOps.
Part of the problem, and what ultimately drove me out of coding altogether, is that Ops is a specialized skillset completely orthoganal to the business of building applications. And, I was really bad at it due to a complete lack of interest. As cloud environments get more and more complicated one ends up spending more and more time dealing with that at the expense of building things.
Conversely, there are lots of folks who love ops type problems and are not well suited to building apps. So, it makes sense to put them on separate teams where they can each do the thing that they're best at, rather than fumble through the parts of the jobs they hate. But yes, at this point it's not really DevOps anymore, it's just ops.
IMO it was a worthwhile experiment that probably went too far before being rejected.
Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful. Especially for certain kinds of products, if executed with discipline you can get really good engineering reaction time with no degradation of quality, and often with much cheaper infra costs.
The problem is what you said though. People with both skillsets are rare, and combining them is legitimately a third skill on its own. And even "both" skillsets is a simplification once you start accounting for data, security, compliance, large scaling DB work etc.
It was mostly a dead end for these reasons but it was worth exploring for its potential. And some genuinely good engineering techniques went mainstream out of it, not a total waste.
> Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful.
You’re so right. This is the key, whether a team is doing devops or not.
One of my major contributions when I took on the CTO role was getting dev, ops, and support teams to talk about the pain points. There were so many easy wins to make things run better because the teams didn’t understand the difficulties the other teams were going through.
Isn't that just anyone with an "architect" job title though? I don't think that kind of dev is particularly rare at all.
I've found that "ops" has become far too overabstracted and jargony. Their moat is endless "certifications". Their "experts" are the dogmatic personality types who try everything in their power to prevent their feet from ever touching the ground like it's lava. They lack the kind of dev or sysadmin experience where you have to stand under your own constructions like a Roman engineer.
Any formerly-known-as "full stack" dev who kept at it for the last decade and a half would run circles around anyone in ops. That's every senior dev working right now who doesn't like to manage people.
It used to be that the ones who went SRE just wanted a pay bump for a bullshit job of relatively light coding puzzles and parroting stuff in meetings, but I guess that well has dried up with "AI".
Coming from the Ops side, “full stack” usually translates to “our entire infrastructure stack is managed AWS services, we don’t know how they work”.
It turns out about as well as if you set the Ops team to shipping features.
Eg I’ve seen more than one full stack dev team build a synchronous RPC-over-message-bus framework because they can’t figure out how to make HTTP or actual RPC calls reliable. It is hands down the most poorly and overengineered thing I have ever seen on a service that takes single digit qps.
The ops teams fight that shit tooth and nail because if it blows up, they’re the ones who get called. And it is 100% too late to fix it by then. Ripping it out will take months, and the dev team will say they’ll do it, but they never will because the VP is leaning on them about some feature.
You either say no early, or you are forever on the hook for whatever abomination the intern has cooked up.
Right but the dream and promise of devops was that early-to-mid career line engineers could do these things, that these weren't rare skills and combining them wasn't particularly challenging. This turned out not to be the case, these skills are the requirement not the outcome.
But if you have the budget and connections to hire a department of people with these skills and experience, and a problem interesting enough to keep them engaged in the day to day building, and a leader capable and charismatic enough to align them all into the business's goals... you should do that.
> Ops is a specialized skillset completely orthog[o]nal to the business of building applications.
Exactly this was the idea: persons in a DevOps role have skills in both areas, i.e. are very wide skill set, so that one can assign them to very diverse tasks for more flexibility. Additionally, these persons can implement synergies to make things more efficient in areas that intersect both development and operation.
No, what you descrive is SRE. DevOps was the idea to give people that work 100% as developers the ability to do some amount of ops work (let say 5%), as a way to break the silo between Dev and Ops that existed in older organizations.
it only really worked as part of a team.
Where it worked well is the "devops" or sre/sysadmin would sit in the same team and take away the rough edges of the infra.
conversly they would also keep an eye on the web people and see what they are doing and why, learning the tooling they use and try and rationalise it so they don't install 50 messaging queues.
At one point I was very hopeful that the progression from Infrastructure as a service to Platform as a service was going to simplify things. Instead we seem to have ended up with mind boggling complex services.
I think this one is really localized.
Some places do have really well built out platforms where logging, instrumentation, databases, deployment methodologies, IaC, etc have well-built happy paths that exclude a bunch of accidental complexity.
I think a lot of other places are either too small or grow too quickly with the wrong set of values and admit too much into their "platform" which had very little design to begin with.
Real DevOps is actually just Dev - in that the Devs are doing the Ops. The people called DevOps are just there to backstop the devs oncall when shit hits fan and you need someone more experienced in ops / incident management to escalate to.
Yes, big companies have a tendency to over-structure things. Devops? Sounds great, let's have a dedicated team for it. Agile? Absolutely, and to make sure it's done right, we need a rigid process around it.
I need the company to have skin in the game, so I prefer that I am entirely unreachable outside M-F 9-5. Otherwise they just exploit people who submit far too quickly to being on-call 24/7. It also gives me motivation to make my stuff work as best as possible so that I'm not being annoyed by management about my non-existence outside core hours.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
To me the best compromise is put a price on it.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
Google's SRE was always about building infrastructure to make things reliable, and started in a world where LAMP on bare-metal + DNS load balancing was common. It makes sense to have a separate org/team/role responsible building real load balancing with health checks and failover, figuring out how to do zero-downtime updates, etc.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
> Real ones are oncall for the systems they build.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
> Real ones are oncall for the systems they build.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
there's eventually a point in organization size where it makes sense to have some kind of centralization to make it easier for teams to actually be on call for the systems they build without grinding them to dust.
SRE at google works closely with devs to architect and build systems together. Most of them are on the SWE ladder, are able to bug fix and scale directly. They're not just ops.
The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
If SRE cannot throttle the release cadence or fix bugs or change architecture in binaries, it's not really SRE.
When I worked at Google SREs only handled legacy products that were maintained but not in active development. SRE was its own vertical separate from SWE and they would only take on maintenance of a new service or product if SWE could prove to them it was stable.
I know Google SREs who do feature development on the infrastructure support they support alongside their devs.
Nice I started a timer for how quickly a "well actually, SRE at google" would show up and you did not fail to deliver.
> The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
Sounds like a cultural issue. Google should look into that.
Why is your response so pithy? Google invented the term SRE, and I think how they use SRE is pretty relevant to the discussion.
Oddly the industry mostly rejected googles application of SRE even as they adopted the term. I've only seen Google use it to refer to engineers maintaining / operating someone elses code.
It’s cause they don’t know what they are talking about.
Easier to shit on people and things with hot take then actually have a grounded and coherent argument or realise that different companies/teams have different constraints, problems and environments they operate as well as different solutions.
It turns out it's easier to hire people who specialize in scalability, reliability, debugging of distributed complex systems, cost ops, efficiency gains, etc. than it is to instantly ensure every single engineer that works there is an expert in all those things, without anybody on staff who is an expert in them. Much like database skills, IT security skills, etc.
It's still something everyone can theoretically learn, of course, and someone who works in that role is most effective by instilling those disciplines in others as they go, and making everyone successful.
He's right you know.
A cultural issue that many organizations face. Same reason most large companies have an infosec department that runs separately from IT.
ZIRP? That's not an acronym I'm familiar with.
https://en.wikipedia.org/wiki/Zero_interest-rate_policy
"Zero" Interest Rate Period, during the COVID pandemic, where most governments offered very cheap bonds or printed money to reduce interest rates. This caused a lot of investment as money was "cheap" for investors and needed to find a worthwhile investment in order to grow faster than inflation.
This caused a lot of money to be poured into all kinds of projects, not all of which had a positive return on investment.
We should do it again but the right way this time.
Give literal zero interest loans from a state bank based on social value and economic priorities, just like they do in China. And set up an extreme progressive tax (90% top bracket) on returns that are paid out to owners to make sure everything gets invested back into production.
More investment is the driver of economic growth. All the bad parts of ZIRP came from letting companies use the money for unproductive activity. If it's used efficiently the downward pressure from higher product supply will be stronger than inflation from more money in the system. And you get more demand in the labor market leading to higher wages.
ZIRP goes back to 2010 actually, when interest rates dropped so much in an attempt to stop the great recession.
It wasn't COVID that kicked off ZIRP, it was the global financial crisis in 2008.
Zero interest rate policy
> SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
Yea cause you wouldn't escalate to 50 people on the weekend when one person can do it. Op is basically saying they never worked on a big enough system.
I call that job SysOps - caring about systems and operation
SRE just came from the big guys calling their sysadmins SRE
If you look at the history of SRE, it was distinctly NOT sysadmins, Google specifically wanted software engineers hired to automate scaling as well as sysadmin type ops work.
there is always infra/platform underneath the system, they are the SREs
Sre seems to be disappearing anyway in favor of infrastructure and platform engineering. Which seems to just be kubernetes admin jobs using terraform.
Kubernetes is probably more responsible for deskilling, or at least constraining, the expertise, of SRE/devops/platform/infrastructure engineers than anything.
I don't think that's a bad thing, it makes infrastructure more consistent and less weird (assuming you understand k8s) but it is a very high level of abstraction and for a bunch of people is just a black box.
Continuing the never ending trend of Sys Admins/DevOps/SRE job title holders with skills in specific tools rather than foundational skills always wondering why they are out of a job every 5 years.
That’s what companies hire for. I’ve learned those skills and no one ever really wants them.
If you build tools with that context, they’ll have to replace you with someone that has that context. There’s few of them, and they’re expensive.
It’s easier to hire folks that should be interns or juniors that know that one tool, and then replace the whole department every decade.
It’s why everything is YAML now, despite it being horribly verbose and clunky. Can’t have our people needing to know how load balancing works, just add another 95 lines of YAML boilerplate so they don’t need to learn 5 lines of nginx config.
I never understand this sentiment. I do SRE/DevOps/Platform Engineering/Infrastructure Engineering, and I definitely do need to know how underlying technologies work. Even when everything is kubernetes and YAML.
If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that. Your company can also choose to outsource complexity to the cloud, and you’ll just be writing YAML. Then there’s also companies where you host everything yourself AND you have large/high workloads, then you still need people with understanding.
With your type of comments, it feels like you’re pretending a developer who writes WordPress websites does the same work as someone doing Erlang for instance. You’ll have different skill sets for every type of engineer, and they’ll all be necessary.
That's reinforced by how hiring is done though. I rarely if ever run into companies that hire for fundamentals. Only the Google tier of companies do that.
You have no idea what you're talking about if that's what you think SREs are.
That doesn't apply when your org is big.
> Tier 3 incidents: Novel, complex, cascading failures stay human-led, with AI providing supporting context.
> But here’s the catch.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
How did they do this before AI? I have done a bunch of complex production level debugging and these Tier 3 incidents are always "starting from scratch" never seen before issues. The main "intuition" I built is persistence and verifying/refuting random hypothesis. In other words, I think we can build an intuition for Tier 3 issues only by engaging with Tier 3 issues.
Although, psychologically, some success in T1 and T2 issues is what gives us the confidence to even approach the T3 issues.
I don't even know if this is an AI thing. I've worked at places that have varying levels of operational maturity, and some of the ones that had very old SRE orgs also had a lot of their SREs operating mostly in the tier 3 given by the article.
One of these orgs had basically all of the noisy pages under control through automated remediation and a culture that pushed towards aggressive automation of pages that had the same root causes. Almost everything I was paged for was some novel or uncommon fault condition. Developing an intuition about how the system worked (well, more specifically, how the system failed) took a lot longer because most things I had to look at were some edge case failure.
I think people are getting funneled into a new career path of being a technologist or technology director. People who can ask the right questions at the right time and direct attention to the appropriate areas given the goals and feedback.
People don't appreciate how much of engineering is just as you've described.
When they imagine someone using AI (except for themself), they imagine the least curious, least inquisitive vibe-coder. And they invent the idea that the only way to be someone who asks questions or remains curious is to have paid the dues that they see themself as having paid. Therefore the world must always need them.
Well, all you have to do now is imagine the vibe-coder who has the same curiosity and inquisitiveness that you had especially when you were starting out and weren't yet a professional programmer. The only difference is that now you don't have to bootstrap a whole software career to understand how things work; you can skip that and merely ask.
Admittedly, a lot of my identity was wrapped up in those hard earned skills. I have empathy for those struggling to chart a new path for themselves. It does not, however, change the fact that tools evolve. Ultimately, using tools effectively is how we are measured and while you may have needed a shovel yesterday, today you have an excavator.
This has always been the most valuable part of the job if you ask me. Its just that the other parts of the job were quite hard to get right letting people build a career out of that specialization.
We use AI at work, Claude specifically, and it’s faster than avg SRE for trivial tasks like cross signals correlation. But it requires a lot of effort to build a harness from your company contexts for it to work like an SRE. And it’s just a tool so if you have to build your knowledge to make it useful, otherwise it’s just a blunt. Or in other word, incompetent people are incompetent no matter of tools
What contexts would the AI model need for analyzing, let's say, cluster-wide incident other than the observability data that such systems anyway collect already?
Is it really that complicated to set up? A basic version is make everything work from a CLI in terminal coding agent of choice (probably already mostly there), hook it up to alerts, and let it run in a corner?
Not OP, for me the issue is nuance of all of your infrastructure, application and outside layers. There's a lot of documentation and such, but often times things don't work as expected to a degree and it requires filling in the blanks correctly. The amount of times I've had an AI Agent misinterpret an alert and assume it was XYZ when it was some random layer it had no idea about is often.
It can certainly get closer but unless your tech stack is a webserver and an application it can get complicated.
I like this analogy that I've come up with: Non-writers can produce impressive articles that are unbearable to read the same way non-programmers can produce impressive software that is unbearable to use.
I dug into general AI deskilling a couple days back[1] - no good signs anywhere, but there's a few things we can do like make concerted efforts to do programming (like a coding hour every week), do a Leet code question, and getting the agent to quiz before merging something (stolen from Notion).
[1] https://www.youtube.com/watch?v=EWuf2EekRSM
I've been having this same gnawing feeling. I just started reading the book "The Collision" by Eric C So which promises to cover this subject. I haven't finished it but seems decent so far.
It's so depressing a thought that if I continue on this path, in a few years, let alone a decade, I'll be utterly unable to produce or even understand code. I realize the skill may not be very economically useful, but losing it is still a depressing outcome that's preventable in theory. At the same time I'm feeling a deep apathy towards building anything. Playing a Zachtronics game is the farthest my internal motivation can push me towards...
I'd love to hear what others are thinking/doing.
I’m personally using it to augment my coding more than replace it. I find joy in it.
I use my agent to prototype a rough version that mostly works, and then I iterate through (with or without my LLM) to clean it up. Then back to the top with a new feature until I’m satisfied with the project and code.
I do tend to like it. My real selling point is that I’m not “locked in” to any choices until I’m happy with how it looks. I used to have to build the thing to see how it looks and acts, and going back on those choices was painful.
I do think there will be value in it, but I’ve been wrong before. My agent occasionally makes stupid choices that make every prompt after that 4x as hard to fix. Like I screwed up a pydantic model in an unreleased library and it supported a “legacy” path no one would ever use for like the next 5k LoC. By the time I noticed, legacy handlers were like 25% of the codebase.
That’s gonna burn a looot of tokens if you scale it up to a million line of code app.
I don't know if it's true. I didn't program at all between 2000 and 2015 and I've been able to restart without too much effort. I forgot many things, but I found it relatively easy to recover...
I'm more worried about people that did never program and started directly with LLM support...
I've actually been thinking that designing a Zachtronics-like would be both great practice and great fun!
I've been doing more networking recently. But any skill building is probably worthwhile.
Nice video, but I have said it before, the difference between just “vibe coding” and reliably using AI is the specs you need to write first and the tests after. If you start prompting the end goal/s the AI will basically guide you on what needs to be done and you just accept it as long as it runs, and this is following what most software developers were doing before LLM, the agile way, the quick ship way, the daily standup and ship first discus later way, the code pairing way, which worked mostly (even though I hated personally) in teams, so these developers are replacing what they were used to do with an agent and that’s the mistake. However, if you go with traditional engineering ways, waterfall, specs based, blueprints, and then tests, you control the agents and guide them not the other way around, in your specs you define the tiniest possible thing to what library to be used or excluded, the methods and approaches, among others, the agent here will produce the syntax and help you in the tests, just like in other engineering aspects how machines didn’t take over, they follow your specs and frameworks. The syntax is the past, AI can produce better syntax than any human alive or all combined, if you think otherwise you are delusional and still didn’t move on, it’s time to stop the “developer” mindset and do the engineering one.
I personally have a project that have been spending 2 months now making the specs and blueprint, more than 1000 files defining every bit including the future features too, and not a single code has been written yet, later, an agent will use this to produce the syntax and all the needed tests to have exactly what’s been needed.
> make concerted efforts to do programming (like a coding hour every week), do a Leet code question
I'll probably get downvoted for this but: At 15 YoE, hell no.
Day-to-day coding lost it's allure to me about 3-4 years ago; it's just a means to an end. I think AI came at the right time for me.
At 40 YoE, it seems like your heart was probably never in it. I really miss coding at my day job. I still code plenty for my personal projects.
Let's not rush to such judgment - our desires change as we age, the field has changed (including what nose-to-the-grindstone coding looks like) and burnout affects many.
Plus, some people enjoy the rigors of solving in-the-weeds code problems and others enjoy building systems-level designs. This was always true.
Exactly, thank you. I'm much more interested in how systems interact gracefully, addressing failure modes and building for resiliency/scale.
Do you need good quality code to do that? Sure, but that's just another line-item on the laundry list of things to make it happen.
You guys act like the React component nonsense from the past 10 years is the same as the code I was learning at age 16. It's by far and away completely different. And none of it has been audited by Daybreak.
I'm very happy to let the robot handle any incidental React or other extraneous tools that end up adjacent to the core work I'm doing. It avoids getting blocked without interrupting teammates or spending time and cognitive capacity on stuff that has no lasting value in my work.
That's precisely my point when I say the field has changed even for "true code writing."
I haven't touched AI, so this SRE is still skilled!
Now if only companies wanted to hire people that still have skills and aren't just prompt engineers.
It's fairly hard to understand to me, but also entirely unsurprising that swapping one occupation for another would be interpreted as a net loss at individual skill level.
You guys are still doing things, correct? Is my experience so outside the norm, or is stuff still challenging?
I don’t understand what this title is saying, feels a bit circular. You could say this about so many jobs that are or are not being replaced by AI.
Do LLMs “deskill” us in general? I suspect we are just replacing old skills with new skills.
Ops is bifurcating in two:
* AI-resistant organizations led by graybeards who refuse to send telemetry off-prem (to either observability vendors or to inference providers) and will be dragged, kicking and screaming, into the future, over their cold, dead bodies. There's a lot of large, conservative organizations in this category like banks and governments, where budgets are rubber-stamped and Don't Break It If It Ain't Broken attitudes are tacitly approved of.
* AI-accepting organizations where Ops will go through the same great narrowing that Dev is going through now - build the dark factory and let it run. You do still need people who know what good architecture looks like and to shepherd changes to improve the overall system to make it more observable, scalable, and maintainable. You do still need Non-Functional Software Engineers who don't answer to Product, for the sheer political reason that you need people who have a long-term perspective, compared to Product that has a short-term perspective. The business will make decisions like "Vendor X is too expensive so we're migrating to Vendor Y" and you will need somebody to drive the agents to make that happen in a way that respects availability and compliance requirements. You probably don't need, in the long-term, a whole team of people to handle it, let alone a whole group or a whole division.
At the end of the day... focus on being a strong engineer. Somebody will always be needed to take care of the trees.
Previous discussion on an extremely similar article: https://news.ycombinator.com/item?id=49574167. This one even name-checks The Ironies of Automation in pretty much the same way.
Thought about this one too. I was expecting to re-read this same article to see how I feel about it now.
I feel like I did re-read the same article, or at least a version of it. Is this how content marketing operates these days? Find a popular blog post, rewrite it and attach your company's name to the end?
If it deskills that guy who reaches for complex bash when they should reach for golang but they don't want to bother with writing the build layer in the image, then I'm all for it.
I assigned myself to re-read "Flowers for Algernon" based on my current frustrations with the team / cultural impacts of AI usage.
It's mildly orthogonal, but I think the metaphor is really quite apropos.
Learning to Be Me by Greg Egan is also apropos.
So they get deskilled which means they're no longer considered professionals but merely technicians which means they're now elligible for overtime.
[delayed]
> Some immediate targets include multi-signal correlation across metrics, logs, and traces, autonomous root-cause analysis for partially understood failures
AI is already pretty good at this in "SRE" workflows.
> AI-driven change risk assessment
And half decent at this.
Is HN upvoting just because they agree with the title?
Clearly the "author" didn't even put in the effort necessary to make a real title. Not "SRE persists but a skill crisis looms" not "SRE careers remain but competency lowers in spite of AI" not "AI lowers SRE skill but demand is steady". Moreover, hasn't this type of insufferability been dramatically reduced in recent models? It screams that in addition to being slop, it's outdated.
Why would anyone take this at face value?
I stopped reading halfway because it doesn’t describe competent SREs in an organization where the culture values reliability.
> Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.
Let’s stop here and ask, why do we not fix the underlying issue? Alert noise? Design better alerts. Runbook execution? Make it a cronjob. Known issues? Why aren’t people fixing these known issues? Root cause suggestion from historical data? Sorry if root cause A is in the historical data you expect problem A is fully solved so it wouldn’t appear again in the next incident. Restarting things? Why doesn’t your health monitor restart things automatically?
SREs also need to climb up the abstraction ladder. Instead of using AI to assist with grunt work, use AI or human thinking to build more reliable systems with fewer incidents. Every incident should be novel.
Deskilling is not necessarily a bad thing, if what you refer to as skills are mundane things that should be automated. I do agree with the author in that people are becoming lazy when it comes to critical thinking and evaluating the decisions or work that AI presents to them. These are people who have little passion to do what they do, and for whom their role is primarily a chore.
I don't think this is as black and white. You skill up by spending time with the domain. Of course, you learn better when you are involved in situations or jobs that actually matter. But getting rid of all the "mundane" distances you from the nitty-gritty and you lose that feel for the problem.
Right, there's a reason students that never dealt with a real problem, even when they get As, aren't as good as professionals. You need to do real work to get really good.
I think that's one reason - another reason is that people are time-constrained, backlogged and expected to do more than ever.
Agreed. I think there's also an argument to be made that it's upskilling non SREs to now be able to do SRE things.
No it's forcing everyone to realize all the jargon was made up to sell books and land speaking engagements. Money from patents (2016 Alice ruling upending that whole scam).
Yall this was always just a money making scheme.
Never was it meant to carry humanity into the future.
If you are a single-task person and never stop to think why you do things probably AI can replace you.
SREs are about super-thinking, context switching, recognize patters and workflows, data movements, etc..
Here's a thought experiment: why is it that I can't do long division?
The Calculator!
Calculators deskilled long division!
Thing is though: who gives a damn about long division? The real task is your taxes or making a massive BI report for your presentation tomorrow.
AI "deskilling" due to automating things makes old key skills become something you can just ask for and have done for you. Why bother staying skilled at those things when you can broaden? It clears the way for what used to take forever and be a specialist skill, to become something to steamroll through on your way to finishing a much greater outcome/task. It's the micro way that, in the macro, humanity becomes far more capable as a species. The gates are being flung wide open and the gatekeepers are being lapped and they have blogs!
It's not a tragedy, it's universal access to leverage.
> why is it that I can't do long division?
> The Calculator!
Well, I'm sorry to hear that. I have a calculator and some pretty neat computing hardware. If I go into the attic and search long enough I can probably find my old sliderule. I can also do long division, as can all of my kids. I think it is an essential skill to understand all basic mathematical operations, and to be able to use them on small data sets. If I didn't know how to do division without a calculator I would consider that a failure of my basic education.
I think a closer analogy would be forgetting basic arithmetic as a concept. Imagine how hard it would be to work with more complex math without understanding what division is. How do you prompt the genie, or understand its outputs, without the foundational knowledge?
Is basic arithmetic something that can be forgotten? Studies have suggested that even infants have an understanding of basic arithmetic. It seems to be something innate.
But as we're only imagining anyway, let's forget about the genie for a moment. If we did lose our sense of basic arithmetic all of a sudden what would life at its core even be like? If there were two apples on a table you wouldn't be able to comprehend what happens if you took one. Would we even be able to function like that?
> in the macro, humanity becomes far more capable as a species
Not humanity, a small number of people in this planet, will benefit about this, because now they no longer have to depend on under payed workers to do the job, and will use AI to make the lives of all other human being poorer.
I think that we need to oppose this tech dictatorship, first thing stopping using AI and thus stop depending on an handful of companies that are able to provide it, that while they gain more and more power we all gets poorer.
What was wrong in using our brain, in the end? I still use my brain on my job and I always will, maybe one day they will say that there is no more need of computer experts because AI can do everything (still people bring me their laptop to reinstall Windows, something I've done for 20 years and that AI didn't change), I will just find another job but I will refuse to use this AI shit.
the analogy doesn't work unless your role means having to occasionally do long division, on the spot, without a calculator
Just to be clear are you suggesting one might find oneself without a clanker?
Where're you doing your debugging? The tundra??
You still need to be able to do long division so you can tell when you screwed up on the calculator.
You can own a calculator for $2, good luck owning your own frontier model
Of all the comments on this post, this landed the most.
I feel there is a potential to "distill" the frontier models into useful tools that can run on smaller hardware. Or distill that knowledge to produce hardware, compute or otherwise. Iff one day the frontier model become inaccessible, the tools that were carved with them may remain valuable.
They will remain most valuable if they are designed in a way that is maintainable even without LLMs. If not, they can be studied as artifacts, or reference designs and then recreated using human intelligence.
The analog I'm playing with:
You can rent time at a high end machine shop and use the best CNC machines to produce extremely valuable tools. Once you leave, you don't have access to those tools, so it is advantageous to produce durable artifacts that are valuable on their own — on a different frontier.
Imagine you're going beyond the frontier physically, exploring new parts of the earth or space. You can't bring the giant machine shop, let alone the machinery of society, the entire economy etc. What do you bring? What do you build now while you have access to the best machines to take with you?
Nobody can read a map anymore, either. Or operate a loom, etc, etc.
I recently won the red ribbon, second place, at the Lambtown fiber festival, for my lovely skein of undyed handspun yarn. Weaving it is next. However, I am probably an outlier, and should not be counted.
(It was a full 250g of this fantastical kempy Herdwick, absolutely unlike anything you can get commercially, and I'm honestly more proud of it than of anything I built in six years working at NVIDIA.)
This is a poor analogy because I don't think your inability to do long division will cause as much damage as an inability to bring the site back up in an emergency situation.
If a team can't bring the site back up in an emergency situation that's poor planning, not poor debugging.
Planning for mitigation before resolution lets a team address production issues at leisure (because the customer impact is mitigated)
All this knowledge is in your agent - ask it for ideas!
Except as every damn AI app says: AI is prone to making mistakes. If AI deskills you with a non-zero chance of getting the problem wrong, that seems deeply, profoundly bad.
You can type the same long division problem into a calculator 65,000 times and you will get the right answer 65,000 times.
Can AI get even CLOSE to that?
Computing tapped out on sources of leverage for solving deterministic problems a while ago. Computers got fast and our algorithms got really good.
We're on to the next frontier, and it's a bit weird. Jagged frontier and all. But it's opening up new worlds.
Luckily loads of problems are amenable to a non-deterministic approach - even better, many have lots of solutions. SRE is in that bucket. There are as many ways to SRE as there are SREs
(source: one of the first SREs at Uber, co-invented Chaos Engineering as a sub-branch of SRE, hosted first chaos eng conf at Uber, played a key role in Uber's global SRE and incident response "Ring0" initiative, managed many SRE teams over a period of roughly 7 years before a career pivot. You could say I know a thing or two about SRE.)
To SRE well you look at the situation and pull from a huge range of tools. With such a broad range of solves, you're better to focus on the outcome: did we achieve our SLOs? Are they aligned with customer expectations? Focus on defining that and providing the right support for it to be achieved and you'll be fine - you might even get a surprise.
I'm (still) a software engineer, and I've found it crucial to solve some LeetCode problems from time to time. It's honestly refreshing to write code yourself, make your own mistakes, and not end up low-key mad at a robot for failing to understand your lousy description because, apparently, it still can't read your mind.
I wanted to use this post as a source to write on my web server project's blog on why debuggable web infrastructure matters even if AI SRE agents exists, but the idea got delayed and I ended up posting it on HN... :')
if you want a vision of the future, winston, imagine 'X isn't Y, it's Z'... forever
10000% agree. I am seeing this first hand. SRE and NOCs folks are turning into meat proxies.
These posts are always pointing outwards.
Like it's the fault of someone else that you cannot say no.
Fuck you, you are just part of a privileged wanking industry and the whining is like hearing children fight.
I am going back to making my Napoletanian pizzas, so shut the fuck up and leave me at, at least these pizzas are not changing.
Wanks
Serious question: why are you reading this site?
This is happening to SWEs too. If you have a certain amount of experience, you are capable of critically analyzing AI output. In Big Tech parlance, these are the Senior SWEs and higher. You require little to no supervision and can work independently.
But how do you make Senior+ SWEs? By having a pipeline of juniors who get trained and mentored. That pipeline is cooked now.
There is going to be a brief period where some like Staff+ SWEs are going to make a lot of money by being AI specialists. But the flood waters are rising for everyone. Those opportunities are going to shrink.
It's eventually going to be a problem that the ecosystem has died because there's no new blood and the people there lack the experience and critical thinking to understand what's being done.
I can use AI on topics I know because I can spot the errors. LLMs lie a lot and get things wrong a lot. What happens when nobody can check the output?
The point I got from the article was that AI is a cause for skill atrophy for any level of seniority, and tends to worsen over time. I can see this happening to myself in some areas, while in others I’m able to learn new skills I’m interested in.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
Ime this doesn't happen or reflect the reality of work. I had one mythical job where I wasn't directly on call and only received escalations and didn't have a problem with debugging whole system issues. I still worked on the whole system day to day.
Also I feel like this post is also my frustration with sre. In people's minds it's just the job that deals with issues pumped out by swes and they have to work magic because no one else can be bothered to try at their job. This just reinforces that idea.
Seeing this posted a week after I had to emergency push cve fixes to 1200 servers in production is pretty funny.
Every person on my team is capable of networking, product engineering, database, job processing and we have command of the entire stack. We will push updates for CVEs with stuff we manually compiled. We own the build system and have the ability to change product as needed. We all have experience in product and we're all capable of pushing fixes as needed. Frankly the SRE team is by far the smartest team I have ever worked with at any company I've ever worked at.
Is your org at the point where it would make sense just to upstream patches for FOSS projects and pull rather than patch and push to just your infra?
Just a matter of curiosity for me as my career progresses, where I think that is the ideal, but I don't work in a heavily privatized nor regulated space at the moment of posting.
I did read your latest blog post looking for an answer, but it wasn't clear to me from that reading.
Yeah that. Watched our lot spending 2 hours using Claude to debug shit that was blindingly obvious if they actually didn't go to Claude first.
If Claude goes down or stops existing when we go down, we're fucked.
The best is when it's just two devs blindly sharing output from Claude back and forth during a major outage instead of USING THEIR BRAINS
Something tells me they were not very big on doing that even prior to 1 BC (Before Claude).
Our lot were. It's like they're now a 5 year old that has been exposed to Italian brain rot for 6 months. https://www.youtube.com/watch?v=DaJ0JAsJEAc
The ubiquity of this rhetoric is not an indication of its correctness. Repetitive tropes like "deskilling", "losing the ability to think" reflect a deep seated distrust of AI alone. No matter how many people parrot these fears the fact remains that you choose your attention. My manager has started coding, heavily, aided by frontier LLMs. He instinctively knows his boundaries and defers much of the more complex work to his engineers. We use the clankers too. Have we been deskilled? Hardly appropriate. Trans-skilled, or learning a new emphasis built upon our long experience? Much closer to the truth.
I work a "dev"ops job, and like yeah, I'm pretty sure this field is even more fucked than software engineering is.
IaC codebases, configs, utility scripts are relatively tiny, and fit into context windows whole. Models are eating them for breakfast.
They're really good at incident response. Sadly, much better than our dedicated colleagues from <location> ever will be. They'll chew through cloud service logs, application logs, application code, iac code and config, CI output, etc. to get whatever is needed, cause wide knowledge is no issue. Even disassembling stuff is no barrier, tracing things down from app level, to runtime level (e.g. JVM), to native calls.
They're really good at handling deployments, end to end infra setup and migrations, following checklists. Hell, they don't even really need checklists. Something that cannot be said about other dedicated colleagues from <location>, who cannot breathe without a checklist, will skip over steps, or otherwise very creatively misinterpret them or improvise something wicked.
They're really good at spinning up mad nice dashboards, and keeping business types happy with impressive looking write-ups they'll never read. It's really refreshing to not have to write those write-ups anymore (or the aforementioned checklists).
It's straight up both not looking good and looking better than ever. Having it be fully hands off remains an issue though.
At this point a company can effectively fire you by taking away access the Enterprise LLMs.
I am more worried that an LLM company could destroy your entire company by removing access or losing enough cash to go under.
AWS, Azure, GCP, etc could already do this more effectively. Your bank even more effectively.
At least for LLM absolute worst case scenario you can run models yourself.
Yeah don't start me on using the cloud. I am well aware of that concern.
As long as I receive my paycheck I don't care, be passive aggressive to me as much as you want!