Hah, love that now they say "Our priorities are clear: availability first, then capacity, then new features" when 6 months ago, it was seemingly exactly the same except Azure supposedly was gonna save them:
> GitHub Will Prioritize Migrating to Azure Over Feature Development - GitHub is working on migrating all of its infrastructure to Azure, even though this means it'll have to delay some feature development.
> In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes.
So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. And it was "existential" 6 months ago yet they keep stumbling on the exact same issue today?
Even if they're focused exclusively on reliability and uptime, we get the experience that we have today, kind of incredible how a company with the resources of Microsoft seemingly are unable to stop continuously shot themselves in the foot. It's kind of impressive actually. As icing on the cake, they've decided to buy up all popular developer services then migrate them all to the same platform, great idea too.
This seems uncharitable. Priorities aren't exclusive, especially at scale across large engineering orgs like GitHub. It could be that these are the top level priorities, but teams or individuals who aren't able to contribute to these priorities will work on other things like new features.
Ditto. I agree though, just because the priority is reliability, doesn't mean others can't work on features, especially features that might help with reliability, which I read was the motivation behind the new single-issue view, so that's my bad, might have been a bit much.
I still think the rest of my point stands, especially the last one which is the move that has the biggest impact to the most of us developers.
> So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example.
It's entirely possible the move to Azure has made the availability problems worse. Dedicated hardware is much more predictable than cloud. "Let's not move to Azure and instead buy a few more racks" was likely a decision beyond the pay grade of github's management.
This entire exercise if anything is a huge indictment of Azure.
But that doesn't matter because the kind of person that buys Azure, just like the kind of person that buys MS Teams, is entirely driven by price and does not care about anything else.
It's kind of hard to read this with a straight face.
The unlabelled graph with big numbers on top, the priorities that don't match with what we're experiencing, and a list of things that they're doing without a real acknowledgement of the _dire_ uptime over the last 12 months....
These are not the worst graphs in the world... Sure the bottom left axis is not labeled, but it still conveys the point correctly. The growth between 2023->2024->2025->2026 is growing quickly. And that in the end/beginning of 2026 they say more growth than the three years before, combined!
You don't need to know the bottom left axis number. We do have to assume the graph is linear, and not some kind of negative exponent log graph. But given the rest of the content, I think that is safe to assume.
Any company that experiences significantly more growth than they were planning for will have capacity issues.
The priorities are most inline with that. The are way beyond the point that they can just add more hardware. They need to make the backend more efficient, and all the stated goals are about helping there.
> You don't need to know the bottom left axis number.
We very much do. The graph suggests an insane growth in PRs from almost zero to 90M. Now compare this misleading graph with this much clearer one, which shows that the growth over the last three years has been less than 80%: https://github.blog/wp-content/uploads/2025/10/octoverse-202...
> These are not the worst graphs in the world... Sure the bottom left axis is not labeled, but it still conveys the point correctly.
No, they're completely useless. Using the "New repos per month" as an example, if the bottom left is 1m, then that's a 20x increase in 2 years which is a lot. If the bottom left is 19m, it's a 5% increase in 2 years which is nothing.
The massive surge on their labelled X axis starts in 2026, and these issues have been going on for a lot longer than that. GHA has been borderline unusable for a year at this point, if not longer.
> But given the rest of the content, I think that is safe to assume.
The rest of the content is "we're working on it", and "here's two outages in the last 14 days, one of which caused actual data loss"
What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale, when 10x YoY is not enough?
As a business user, our costs have gone up while service has gone down dramatically. Meanwhile our marginal cost to GitHub has hardly changed. Where our costs to them have increased, they mostly charge us per cpu minute, so obviously aren’t making any kind of loss on our account.
I’m sure they’re experiencing scaling issues across the platform, but it’s unacceptable for that to have a negative impact on us when we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.
I'm curious how Azure DevOps reliability has been for comparison. My current job is managing stories in DevOps with SCC in GitHub ent. While I like Github slightly more, have been curious about the decision.
I understand that, and maybe GitHub became a bad deal because of that.
But if anything, their post and your reply are precisely an endorsement of usage based billing.
The bit that's growing 13x YoY (and which they expect will easily blow past that) is unmetered - commits. The bit that is metered (for some, not all folks) - action minutes, grew only 2x YoY.
GitHub was not built to limit the number of commits, checkouts, forks, issues, PRs, etc - nor do we want them to - but that's what's growing ridiculously as people unleash hordes of busy beaver agents on GitHub, because their either free or unlimited.
Where there are limits - or usage based billing - people add guardrails and find optimizations.
Because for all the talk, agents don't bring a 10x value increase; otherwise, they'd justify a 10x cost increase.
Besides, other forges are having issues too. Even running your own. We have Anubis everywhere protecting them for a reason.
These numbers should have been in the blog post, not the graphs that are present.
> What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale
I think you're putting words in my mouth here; I didn't say either of those things. I'm saying that this blog post is a meaningless platitude when the github stability issues predate this, and that all this post says is "we hear you're having issues".
I just think their charts, taken at face value, show substantially the same thing (for PRs, commits, new repos).
Either those charts are a bald-faced lie (the tweet could be as well) or there is no way for that chart to be something else.
The only way to fake exponential growth like that would be to use an inverse log scale (which would be a bald-faced lie).
It doesn't even really matter what's the y-axis baseline, unless we really think growth was huge in 2020, then cratered to zero by 2023, now back to the previous normal.
As for the rest of the post, I do think it's panic mode platitudes. But I honestly don't know what I'd write instead that's better.
You can already see people complaining loudly where they instead of "we'll do better" decided to limit usage.
> I just think their charts, taken at face value, show substantially the same thing (for PRs, commits, new repos).
The problem is that these charts show the massive exponential growth in 2026. But this didn't start in 2026, this has been going on since early last year. My team had more build failures in 2025 due to actions outages or "degraded performance" than _any other reason_ and that includes PR's that failed linting or tests that developer were working on.
> As for the rest of the post, I do think it's panic mode platitudes. But I honestly don't know what I'd write instead that's better.
IMO, this needed to be written a 6 months ago (around the time that the memo of them prioritising the migration to Azure was released), and then this post should have been "We're still struggling, this isn't good enough. Here's the amount of growth, here's what we've done to try and fix it, and here's what we're planning over the next 3-6 months", instead of "Our priorities are clear: availability first, then capacity, then new features" and "We are committed to improving availability, increasing resilience, scaling for the future of software development, and communicating more transparently along the way." This isn't transparency (yet).
Is this microsoft stating that they aren't able to get acceptable reliability from Azure? (I mean, I think a lot of us have heard that, but it's interesting to hear it from microsoft themselves).
Pretty damming that two Microsoft subsidiaries - GitHub and LinkedIn - either shelved their forced migration to Azure or are looking at non-Azure options.
You’d think they could have had the existing GitHub on whatever continue as is (maybe for paying customers) while all the AI new inrush goes to the Azure setup.
Man, you should have been there 6 months ago when they decided to start tearing down GitHub's own data centers and move everything exclusively to Azure. Seems they themselves realized this after they started moving, but imagine if you could have helped them realize this before they even started :)
> Seems they themselves realized this after they started moving
I guess most people at Github knew exactly it makes no sense but they didn't really have a choice. Maybe some voiced their statement, got "we hear you" in response and were told to proceed anyway.
Yeah, I don't know how it went down, but I also know exactly how it went down:
Microsoft Execs: Everyone needs to move to Azure!
GitHub developers: But Azure is not gonna be able to handle our load, we literally have our own data centers!
Microsoft Execs: Sure, but you're Microsoft now, please publish blog post about how in half a year you'll be 100% on Azure.
Few months later...
GitHub Developer: We've tried our best, users are leaving in droves and Azure can't keep up!
Microsoft Execs: Ok fine, you can use something else too, but only if you mainly use Azure and continue publishing blog posts about how great Azure is.
Really? I thought retail was. It's been almost a decade since I worked at prime video but I think everything was running on AWS. (Some things didn't use brazil etc, but I think all the servers etc. were on AWS)
It's a distinction without a difference. All new development is nAWS (native AWS) legacy is mAWS (not sure about the acronym) which is still AWS under the hood and is mostly just a pool of EC2 instances with preconfigured networks. Nothing made in the last five or six years is on maws, and amazon is a micro service shop so things are always being built new. If you joined today there's a good chance you'd join a team without any maws infra
MAWS is “Move to AWS”, the name of the internal campaign to get legacy services into a somewhat-retrofitted AWS environment. It was a single VPC at one point.
The entire concept of multi cloud is amusing if you think what cloud originally was supposed to be. They could call them meta clouds (might infringe trademarks), and with the current growth trajectory of AI generated code eventually multi-meta-clouds, renamed to beyond-clouds, and then multi-beyond-clounds. I see no limits.
GitHub stability has been bad for me. And recently even the data they show me in the web has been unreliably.
Since yesterday, me and several colleagues noticed that the pull request lists on the website are incomplete, across many repositories. For example, on https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`)
In many of my projects don't show any closed pull requests for the last 6 days. The CLI can list them, but anything going through search shows nothing.
Their support acknowledged the issue, but has been silent since then, and the status page still shows nothing other than the potentially-related issue on the 27th. It looks like it has been resolved on some repositories in the meantime, but I still have the issue across multiple orgs and repositories.
> For example, on https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`)
Surely a scaling hack where they use "estimation" queries that return "kind of right" results instead of 100% correct data, as it's less load on the infrastructure. Not necessarily a bug as much a shit choice from product perspective.
If the numbers were all that is wrong, that'd be OK. But it fails to list all data -- so the only way to navigate to the missing PRs is to know their number, and manually inserting the right URL (or to go to another PR, and then edit the URL in the navigation).
Sorry, but I don't think there is any way this can be classified as "not actually a bug"
Glad that they released some data about new repo/issues/commits over the last years. It confirms what everyone else already believed from the outside: agents are putting a lot of extra, sudden pressure on GitHub. It's like a startup that is growing exponentially, with the difference that they already have a large user base to serve - and that keeps them in the bullseye - and probably a not-so-fast-moving organization when it comes down to changes. On the other side of the coin, they also have a lot of talent, infra and money a startup might not have yet.
Stop subsidizing tokens now that we extracted enough training data from you and we have enough agentic junkies business to keep the flywheel going up and cut on the loss leaders. [0]
I can not figure out what on Earth they've done with these graphs, it almost seems like these are an artists impression of a graph.
Looking at the commit graph: Why do commits have big steps followed by slow rolloffs? Why do the steps not happen at uniform points Why do larger steps sometimes have less of a slope than smaller steps but not all the time?
Then looking at the other graphs there's completely different effects going on.
I'm biased (founder of tangled.org), but the future really should be federated forges. Host repositories on sovereign infra with global identity + federated "metadata" (issues, pulls, etc.).
Global indices for this should be trivial to spin up so availability is never a concern (we're working towards this!).
I would love if it coding agents didn't default to GitHub for their deep VCS integration.
If I could get the same bells and whistles by wiring up another forge, so long as it offered a decent API and/or sent events over a webhook, I'd have everything self-hosted.
The agents would need to expose an interface on their own end but as long as you implemented it with a plugin, it'd take the dependency of GitHub and you could use MCP or skills for the rest of it.
The neat thing about Tangled is it's built on an open protocol (https://atproto.com)—this allows us to effectively build an API-free system since all data on Tangled can effectively be ingested via the AT Protocol firehose.
Which is to say, this is perfect for agents given they don't need any bespoke SDK from us: simply write Tangled records for issues, pulls, whatever to your PDS and it'll show up on Tangled. We plan to start working on some exemplar agents first-party that would 1. enhance Tangled itself, 2. showcase cool things you can do with an open data firehose.
But, there are? I can host a repo on GitHub, Codeberg and self host it too. Then I need to watch over main to keep it consistent between those. After that's established, I can do updates from wherever. Link'em in the README.
There are distributed forges? Yes, git is distributed, but often everything around it isn't. The case parent is trying to make, is that the rest ("federated forges") should also be distributed, not just git.
> what is the missing killer feature that needs federation?
Issues, pull requests, collaboration/permissions/access, "staring"/"favoriting", etc.
I think ultimately the goal is that people can run their own forges, yet still collaborate on repositories hosted in other forges, leveraging your existing authentication so you no longer need to sign up individually for each forge.
Love the idea, would replace the LLM generated content ony our site, though.
I recently migrated to codeberg because I'm okay with self-hosting big runners, while using codeberg's available runners for smaller cron-based things (they even have lazy runners for this).
I know it's just marketing speak, but the term made me think of the scenes in the Matrix where what's left of humanity (ignoring all the cyclical lore that was added on top of it) has to make sure the machines can't remote in to any of their tech.
So if a company self hosts their physical infrastructure which will burn down once a fire sets in, they are more "sovereign" than a company running on a redundant cloud? I definitely would not want to be "sovereign" then.
Point is: This discussion is much more multi-dimensional than some suggest.
Yeah sorry it's marketing BS speak for self-hosted or just infra that you control. It could be a VPS, it could be a Raspberry Pi at home. Your repos live on your servers. (And we support this on Tangled today!)
But a VPS isn't actually infrastructure you control, you essentially have as much control over it as "cloud", so I don't think that'd be counted as "sovereign", would it?
The internet should not be centralised, but you can't make a billion dollar company without capturing the world and selling your company to a trillion dollar company
Not enough attention is being put in the production/delivery mismatch.
GitHub is claiming they require 30x scale due to the giant increase in repository creation, PRs, commits, etc.
I have not seen a single product increase in features or quality as an end user, nor new significant products have come out in this period (other than the LLMs themselves).
I for one believe Microsoft when they say this code is going to Github... to die.
Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once.
In companies, I see people vibe-coding "miracle apps" that fall under the smallest amount of scrutiny.
Basically people are doing the same developers do when they say "I can do this in a weekend", which is getting a prototype sort of running and then immediately losing energy (or in this case lacking ability) to push it forward.
> Vladimir Fedorov is GitHub's Chief Technology Officer .... He currently serves on the board of Codepath.org, an organization dedicated to reprogramming higher education to create the first AI-native generation of engineers, CTOs, and founders.
openai, anthropic, google and a plethora of chinese models all end up pushing code into github. you can discuss whether gpt 5.5 is better than opus 4.7, but for github it doesn't matter: they'll be receiving the code no matter which llm spits it out.
amazing on one hand, quite scary on the other for github and all other forges if this continues and there is no reason why it wouldn't.
The AI agent growth explanation is interesting but also a bit of a deflection. If a meaningful portion of your traffic is now automated agents, your capacity planning model is fundamentally different, you're no longer scaling for human paced workflows but for burst patterns that look nothing like historical load.
The unlabeled graphs don't help the credibility case. When you are already in the hole on trust, shipping a post that requires readers to assume favorable baselines is exactly the wrong move.
This latest incident was the nail in the coffin for me. I've been on GitHub since 2012 but I'm feeling the pull to migrate out to Gitea/Forgejo. Has anybody done this recently? How'd it go?
When one of the incident they write about here happened, I wrote about my experience moving from GitHub to Forgejo which I happened to complete just the night before that happened: https://news.ycombinator.com/item?id=47878192 (lots of other people sharing their experience as replies too)
I was thinking of maybe doing a proper write up about how to host your own Forgejo + Action runners on Linux, Windows and macOS, not sure if there is enough interest. What would people for sure want to know in a guide/explanation of this?
I moved over back when GitHub was planning to charge per minute to use my own runner. It was easy with Claude, the gh API, and forgejo web API. I even set up daily backups to my S3 clone of choice.
The only repos I left on GitHub are forks and one with a bit of public engagement.
* we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL)
* * redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load.
* we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go.
I'd like to know what database backend they migrated to. I was also surprised to read that the migration from Ruby to a more performant language had not already been completed. I assume this is because it a large code base with many moving parts, etc.
> The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply.
GitHub instability has started way before that. I understand it’s too much to ask of a trillion-dollar corporation to consider the impact of their own actions, but perhaps they should’ve thought of that before forcing LLM development down everyone’s throats.
While they contributed, they were still following the market trend anyway. If they weren't letting folks use it directly, other companies would have (and are).
> they were still following the market trend anyway.
They started the trend with Copilot.
> If they weren't letting folks use it directly
There is a chasm of difference between “letting you use it” and “forcing it down your throat”. Microsoft is doing the latter, not the former. Copilot is annoyingly present by default at every step on GitHub.
I had to postpone a call with developers (in 2 different countries) because I didn't had access to the issues board, which is a single source of truth for us.
I understand the rapid growth (because of AI agents), but if such critical software service becomes unstable then it's time to migrate? Thinking about self-hosting GitLab.
> but if such critical software service becomes unstable then it's time to migrate?
Right way to think about this:
> If things we need/see as critical for our work are hosted on a platform with really bad reliability, it's time for us to migrate
My internet connection at home is really shit, and almost every week there is a multi-hour downtime for some reason, not to mention when La Liga games are on TV anything using Cloudflare is unavailable, so I've had to spend extra energy and time to setup things in a way so I can still work whenever this happens.
The bit nobody covers in these write-ups: small teams without dual-cloud failover budget. Last big GitHub outage cost me a deploy day. Not catastrophic but the kind of thing you don't budget for when GitHub is your single source of truth.
Status page is also still doing that thing where every component is green but in practice clone is hanging, push is timing out, actions are stuck. Per-service uptime is a managed number. The user-experience number is the one that matters and it's not in the post-mortem.
I wonder if they’ll end the free lunch we’ve been having since the MS takeover. There’s been a deluge of spam and crapware projects due to the LLM wave which is visible in that graph. Can’t see them sustaining being a public dustbin for low value projects forever.
I could see them expiring/archiving/deleting inactive projects after some time.
I feel like this would have negative impacts (lots of interesting historical archives on Github) but maybe if a project hasn't been touched, or cloned, in some time, it just gets deleted with some notice.
I’d guess it has much more to do with the extra load agentic ai generates. If we take the charts in the OP at face value, do you think gh suddenly exploded in popularity? At this point I think almost everyone who has any use for gh already has an account and use it as much as they ever would. But all the charts go to the moon. Gh obviously didn’t take into account that ai agents can generate a lot of activity they don’t have capacity for.
> While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud. This longer-term measure is necessary to achieve the level of resilience, low latency, and flexibility that will be needed in the future.
I have a hard time believing anything what's said in a blog post where a graph lacks axes labels/scale. It tells me that nobody who cares about correctness had any say on the content of the post. Maybe I'm being 8am cranky and pedantic, but I'm sticking with it.
> availability first, then capacity, then new features.
I'd love to experience first-hand a leadership team who says, "stop accepting new paying customers until we've got availability sorted out!"
I know that I'm simplifying (probably too much), but it seems like things were fine when GitHub was still a Ruby on Rails monolith and all the rigmarole with microservices etc. only made things worse.
Unless everything else stays the same (underlying traffic etc.) then you can't really compare. Could be that you hit some fundamental scaling limit with the old design and it completely falls over after a certain scale.
Oh as said I'm pretty sure things are more complex. It's just funny in a way that all these technologies that are usually being sold as "enablers for scale" don't seem to do their job very well.
It is, but everyone is entitled to beliefs. Anecdotally, I feel the same way. Everywhere I’ve been, there has always been a legacy monolith that was stable as a rock, with dozens of new microservices scattered around it in an attempt to exit the monolith. The microservices have never once been stable. People fail to take the most basic things into consideration, like “you can’t have Consistency and Availability when everything is a network call.”
I’m sure survivor bias is at play here, but when I look through the older code bases - especially the data model - it’s an entirely different world than the newer stuff, and it’s clear which of the two was written by people who understand systems.
GitHub been oscillating between long phases of "Never any new features but rock-solid and no downtime" and "New features every week but also unicorns (used to be the "service unavailable page") every week" for as long as I can remember. Seems they're on some interval switching between the two.
Personally I'd never use codeberg. Their FAQ on licensing [0] is basically everything that anyone who supports free software should abhor - it's "we might allow you to do what you want to".
I don't think it's 100% compatible, but Gitea's/Forgejo's (which Codeberg runs on) own Action implementation is pretty much the same as GitHub Actions, with minor differences.
on another note - is the exponential growth from 'agentic' workflows actually resulting in productive software in the wild. Or it is just noise. On my end I haven't seen the software I use getting better.
Personally, I’m sympathetic. We know that GitHub did a huge amount of work over the last decade to make Git scale, which has benefited us all. These new scaling challenges are real challenges, 30x growth would be a nightmare for any system that was already pushing the limits of what was possible, I think we are being far too hard on GitHub, they deserve a little grace.
For all the negatives about github I agree. They offer a lot of free stuff, and LLMs seem likely to put massively increase their costs with no guarantee they'll be making money off it. I can't think of many (any?) large businesses which could scale up to meet so much new demand without some significant growing pains along the way.
GitHub's scaling issues are caused by their own vendor-lock approach and monopoly. Yes, of course _their_ goal is to be even bigger and even more all-consuming, so _they_ have to deal with the scale. Why a user would be sympathetic to that?
The user (and not a big tech monopoly) answer to scaling issues is almost always to stop scaling and start federating and interoperating.
So they haven't even finished migrating from their datacenters to Azure and have now started a project to add another cloud provider ("multi cloud")? Madness.
anyone who's actually worked there, could you explain why they're finding scalability and reliability so hard? naively it seems like 'repo groups', ie clusters of repositories linked by being mutual forks, would be fairly isolated for the whole git storage layer, and everything else feels pretty easily parallelisable (issues, actions, etc, modulo taking locks now and then to submit results or whatever). and given that, surely you can incrementally deploy changes across those many shards to avoid most big outages?
are there big conceptual serialisations that I've missed? is it just not well factored? was the move to Azure just a catastrophically bad idea? some other thing?
Almost every high volume service on the internet is write a little, read a lot, and when there are writes, they're relatively small, a few bytes into a database that can fan out. GitHub is very different: constant writes, large files, it is under far more pressure than the systems the rest of us build. And then, as the article says, vibecoding happens, and suddenly they're receiving 30x the volume of expensive operations. GitHub are responsible for many of the performance improvements made to Git over the years, Git scales today because of work GitHub did, but that work was never intended to scale to volume of today.
Even as recently as 18 months ago, Lovable appeared, seemingly overnight, and caused huge problems for GitHub because they were creating repositories on GitHub for every single Lovable project, offloading the very high cost onto GitHub, hundreds of thousands of repositories. A couple of years before that, Homebrew used GitHub as a de facto CDN and that was a huge problem, too.
Nowadays it is easy to imagine how we can scale out a service like Twitter or YouTube or Facebook because everything has been done before, but that's not true of Git, Git hasn't ever scaled like this before, there are very few examples of service with GitHub's characteristics.
recently there was a twit how GitHub PR diffs had 10 React components PER LINE. And how they optimized that to only 2 React components per line or something.
> To summarize, for every v1 diff line there would be:
I'm asking about the infrastructure, obviously they chose for some reason to make my computer fans turn on to show some red and green lines on a text file.
Good chuckle out of this post, it’s crazy that neither Atlassian (Bitbucket) or Gitlab are capturing value out of this same agentic coding boom. I wish github was separately publicly traded outside of Microsoft.
Hah, love that now they say "Our priorities are clear: availability first, then capacity, then new features" when 6 months ago, it was seemingly exactly the same except Azure supposedly was gonna save them:
> GitHub Will Prioritize Migrating to Azure Over Feature Development - GitHub is working on migrating all of its infrastructure to Azure, even though this means it'll have to delay some feature development.
> In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes.
https://thenewstack.io/github-will-prioritize-migrating-to-a...
So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. And it was "existential" 6 months ago yet they keep stumbling on the exact same issue today?
Even if they're focused exclusively on reliability and uptime, we get the experience that we have today, kind of incredible how a company with the resources of Microsoft seemingly are unable to stop continuously shot themselves in the foot. It's kind of impressive actually. As icing on the cake, they've decided to buy up all popular developer services then migrate them all to the same platform, great idea too.
This seems uncharitable. Priorities aren't exclusive, especially at scale across large engineering orgs like GitHub. It could be that these are the top level priorities, but teams or individuals who aren't able to contribute to these priorities will work on other things like new features.
Ditto. I agree though, just because the priority is reliability, doesn't mean others can't work on features, especially features that might help with reliability, which I read was the motivation behind the new single-issue view, so that's my bad, might have been a bit much.
I still think the rest of my point stands, especially the last one which is the move that has the biggest impact to the most of us developers.
> So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example.
They did that as a panic mode hack to mitigate performance: https://news.ycombinator.com/item?id=47912521
It's entirely possible the move to Azure has made the availability problems worse. Dedicated hardware is much more predictable than cloud. "Let's not move to Azure and instead buy a few more racks" was likely a decision beyond the pay grade of github's management.
Azure is easily the least reliable and least secure of the 3 hyperscalers, which is crazy because GCP was an also-ran underdog not that long ago.
This entire exercise if anything is a huge indictment of Azure.
But that doesn't matter because the kind of person that buys Azure, just like the kind of person that buys MS Teams, is entirely driven by price and does not care about anything else.
It's kind of hard to read this with a straight face.
The unlabelled graph with big numbers on top, the priorities that don't match with what we're experiencing, and a list of things that they're doing without a real acknowledgement of the _dire_ uptime over the last 12 months....
These are not the worst graphs in the world... Sure the bottom left axis is not labeled, but it still conveys the point correctly. The growth between 2023->2024->2025->2026 is growing quickly. And that in the end/beginning of 2026 they say more growth than the three years before, combined!
You don't need to know the bottom left axis number. We do have to assume the graph is linear, and not some kind of negative exponent log graph. But given the rest of the content, I think that is safe to assume.
Any company that experiences significantly more growth than they were planning for will have capacity issues.
The priorities are most inline with that. The are way beyond the point that they can just add more hardware. They need to make the backend more efficient, and all the stated goals are about helping there.
> You don't need to know the bottom left axis number.
We very much do. The graph suggests an insane growth in PRs from almost zero to 90M. Now compare this misleading graph with this much clearer one, which shows that the growth over the last three years has been less than 80%: https://github.blog/wp-content/uploads/2025/10/octoverse-202...
The graph is labelled Open Source, could it exclude private internal repos?
That link shows the number of PRs created to be less than 10M though.
Yes, to be honest, that graph could use some improvements as well. I should probably just link to the blog post with actual numbers: https://github.blog/news-insights/octoverse/octoverse-a-new-...
> These are not the worst graphs in the world... Sure the bottom left axis is not labeled, but it still conveys the point correctly.
No, they're completely useless. Using the "New repos per month" as an example, if the bottom left is 1m, then that's a 20x increase in 2 years which is a lot. If the bottom left is 19m, it's a 5% increase in 2 years which is nothing.
The massive surge on their labelled X axis starts in 2026, and these issues have been going on for a lot longer than that. GHA has been borderline unusable for a year at this point, if not longer.
> But given the rest of the content, I think that is safe to assume.
The rest of the content is "we're working on it", and "here's two outages in the last 14 days, one of which caused actual data loss"
You mean since GH acquisition 6 years ago https://damrnelson.github.io/github-historical-uptime/
More numbers: https://x.com/kdaigle/status/2040164759836778878
What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale, when 10x YoY is not enough?
As a business user, our costs have gone up while service has gone down dramatically. Meanwhile our marginal cost to GitHub has hardly changed. Where our costs to them have increased, they mostly charge us per cpu minute, so obviously aren’t making any kind of loss on our account.
I’m sure they’re experiencing scaling issues across the platform, but it’s unacceptable for that to have a negative impact on us when we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.
I'm curious how Azure DevOps reliability has been for comparison. My current job is managing stories in DevOps with SCC in GitHub ent. While I like Github slightly more, have been curious about the decision.
I understand that, and maybe GitHub became a bad deal because of that.
But if anything, their post and your reply are precisely an endorsement of usage based billing.
The bit that's growing 13x YoY (and which they expect will easily blow past that) is unmetered - commits. The bit that is metered (for some, not all folks) - action minutes, grew only 2x YoY.
GitHub was not built to limit the number of commits, checkouts, forks, issues, PRs, etc - nor do we want them to - but that's what's growing ridiculously as people unleash hordes of busy beaver agents on GitHub, because their either free or unlimited.
Where there are limits - or usage based billing - people add guardrails and find optimizations.
Because for all the talk, agents don't bring a 10x value increase; otherwise, they'd justify a 10x cost increase.
Besides, other forges are having issues too. Even running your own. We have Anubis everywhere protecting them for a reason.
> we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.
You know, you can just host your own code forge. Or you can just drop gitolite on a server. Or pull directly from each others' dev machines on a LAN.
GitHub is not git.
In that case, why are you using them at all?
> we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.
so start a GitHub competitor which bills $50/dev/yr for solving this easy problem and make a lot of money?
These numbers should have been in the blog post, not the graphs that are present.
> What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale
I think you're putting words in my mouth here; I didn't say either of those things. I'm saying that this blog post is a meaningless platitude when the github stability issues predate this, and that all this post says is "we hear you're having issues".
Sorry if I misread your intent.
I just think their charts, taken at face value, show substantially the same thing (for PRs, commits, new repos).
Either those charts are a bald-faced lie (the tweet could be as well) or there is no way for that chart to be something else.
The only way to fake exponential growth like that would be to use an inverse log scale (which would be a bald-faced lie).
It doesn't even really matter what's the y-axis baseline, unless we really think growth was huge in 2020, then cratered to zero by 2023, now back to the previous normal.
As for the rest of the post, I do think it's panic mode platitudes. But I honestly don't know what I'd write instead that's better.
You can already see people complaining loudly where they instead of "we'll do better" decided to limit usage.
No problem - it's tough online sometimes.
> I just think their charts, taken at face value, show substantially the same thing (for PRs, commits, new repos).
The problem is that these charts show the massive exponential growth in 2026. But this didn't start in 2026, this has been going on since early last year. My team had more build failures in 2025 due to actions outages or "degraded performance" than _any other reason_ and that includes PR's that failed linting or tests that developer were working on.
> As for the rest of the post, I do think it's panic mode platitudes. But I honestly don't know what I'd write instead that's better.
IMO, this needed to be written a 6 months ago (around the time that the memo of them prioritising the migration to Azure was released), and then this post should have been "We're still struggling, this isn't good enough. Here's the amount of growth, here's what we've done to try and fix it, and here's what we're planning over the next 3-6 months", instead of "Our priorities are clear: availability first, then capacity, then new features" and "We are committed to improving availability, increasing resilience, scaling for the future of software development, and communicating more transparently along the way." This isn't transparency (yet).
"We hear you" in ~300 words, basically.
You can do the same with so many clients.
> we started working on path to multi cloud.
Is this microsoft stating that they aren't able to get acceptable reliability from Azure? (I mean, I think a lot of us have heard that, but it's interesting to hear it from microsoft themselves).
It’s pretty damning. But as someone who has used Azure, I buy it.
Pretty damming that two Microsoft subsidiaries - GitHub and LinkedIn - either shelved their forced migration to Azure or are looking at non-Azure options.
I think this is more tailored towards enterprise clients that lose money when Github is down, that would probably help with retention.
You’d think they could have had the existing GitHub on whatever continue as is (maybe for paying customers) while all the AI new inrush goes to the Azure setup.
Yeah that's a top tier enterprise plan feature if I have ever seen ut
Seems pretty sensible to not rely on a single provider for their large complex system?
Man, you should have been there 6 months ago when they decided to start tearing down GitHub's own data centers and move everything exclusively to Azure. Seems they themselves realized this after they started moving, but imagine if you could have helped them realize this before they even started :)
> Seems they themselves realized this after they started moving
I guess most people at Github knew exactly it makes no sense but they didn't really have a choice. Maybe some voiced their statement, got "we hear you" in response and were told to proceed anyway.
Yeah, I don't know how it went down, but I also know exactly how it went down:
Microsoft Execs: Everyone needs to move to Azure!
GitHub developers: But Azure is not gonna be able to handle our load, we literally have our own data centers!
Microsoft Execs: Sure, but you're Microsoft now, please publish blog post about how in half a year you'll be 100% on Azure.
Few months later...
GitHub Developer: We've tried our best, users are leaving in droves and Azure can't keep up!
Microsoft Execs: Ok fine, you can use something else too, but only if you mainly use Azure and continue publishing blog posts about how great Azure is.
Azure is the MS Teams of clouds.
Made me think. Why not convert Github datacenters into Azure datacenters that have Github as their sole customer?
Then it's up to Azure how they will manage this
I mean, amazon (shopping, along with prime video e.t.c.) runs on AWS.
It was more "we built AWS to run our stuff and figured out we can sell it too".
While Azure feels like Temu clone of Cloud
When I was at AWS, retail was not yet running on AWS. Has that changed?
Prime video does use some AWS services, but live and on-demand are two entirely different beasts.
Really? I thought retail was. It's been almost a decade since I worked at prime video but I think everything was running on AWS. (Some things didn't use brazil etc, but I think all the servers etc. were on AWS)
It's a distinction without a difference. All new development is nAWS (native AWS) legacy is mAWS (not sure about the acronym) which is still AWS under the hood and is mostly just a pool of EC2 instances with preconfigured networks. Nothing made in the last five or six years is on maws, and amazon is a micro service shop so things are always being built new. If you joined today there's a good chance you'd join a team without any maws infra
MAWS is “Move to AWS”, the name of the internal campaign to get legacy services into a somewhat-retrofitted AWS environment. It was a single VPC at one point.
Prime video uses a non-AWS CDN when I watch football on it here in the UK
The BBC were unable to find a single CDN that could serve the UK during its peak football matches. https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk...
This isn't a mom and pop shop. They have locations all over the world: https://datacenters.microsoft.com/
There's no intrinsic reason they should be vulnerable to themselves.
+1. Multi-cloud is typically done for vendor independence.
But Github don't have that rationale.
That website (for me) uses Cloudflare via WPEngine, which also isn't Azure
The entire concept of multi cloud is amusing if you think what cloud originally was supposed to be. They could call them meta clouds (might infringe trademarks), and with the current growth trajectory of AI generated code eventually multi-meta-clouds, renamed to beyond-clouds, and then multi-beyond-clounds. I see no limits.
GitHub stability has been bad for me. And recently even the data they show me in the web has been unreliably.
Since yesterday, me and several colleagues noticed that the pull request lists on the website are incomplete, across many repositories. For example, on https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`)
And that despite <https://www.githubstatus.com> reporting "all systems operational".
In many of my projects don't show any closed pull requests for the last 6 days. The CLI can list them, but anything going through search shows nothing.
Their support acknowledged the issue, but has been silent since then, and the status page still shows nothing other than the potentially-related issue on the 27th. It looks like it has been resolved on some repositories in the meantime, but I still have the issue across multiple orgs and repositories.
https://github.com/orgs/community/discussions/193388
> For example, on https://github.com/gap-system/gap/pulls it says "Pull requests 78" in the "tab list", but the PR list view reports "35 open" (the number 78 is correct, and confirmed by e.g. `gh pr list`)
Surely a scaling hack where they use "estimation" queries that return "kind of right" results instead of 100% correct data, as it's less load on the infrastructure. Not necessarily a bug as much a shit choice from product perspective.
If the numbers were all that is wrong, that'd be OK. But it fails to list all data -- so the only way to navigate to the missing PRs is to know their number, and manually inserting the right URL (or to go to another PR, and then edit the URL in the navigation).
Sorry, but I don't think there is any way this can be classified as "not actually a bug"
Glad that they released some data about new repo/issues/commits over the last years. It confirms what everyone else already believed from the outside: agents are putting a lot of extra, sudden pressure on GitHub. It's like a startup that is growing exponentially, with the difference that they already have a large user base to serve - and that keeps them in the bullseye - and probably a not-so-fast-moving organization when it comes down to changes. On the other side of the coin, they also have a lot of talent, infra and money a startup might not have yet.
What data is that? There's an unlabelled graph and a number at the current peak.
Some previous numbers: https://x.com/kdaigle/status/2040164759836778878
This is the data that should be in the blog post. Thanks for sharing.
What are we doing?
Stop subsidizing tokens now that we extracted enough training data from you and we have enough agentic junkies business to keep the flywheel going up and cut on the loss leaders. [0]
[0] https://news.ycombinator.com/item?id=47923357
With this prioritization Github IPv6 support is gonna happen the next decade.
I can not figure out what on Earth they've done with these graphs, it almost seems like these are an artists impression of a graph.
Looking at the commit graph: Why do commits have big steps followed by slow rolloffs? Why do the steps not happen at uniform points Why do larger steps sometimes have less of a slope than smaller steps but not all the time?
Then looking at the other graphs there's completely different effects going on.
It's because they are your standard PowerPoint graph that just shows "thing goes up" rather than actual data, or the meaning of the data.
I'm biased (founder of tangled.org), but the future really should be federated forges. Host repositories on sovereign infra with global identity + federated "metadata" (issues, pulls, etc.).
Global indices for this should be trivial to spin up so availability is never a concern (we're working towards this!).
I would love if it coding agents didn't default to GitHub for their deep VCS integration.
If I could get the same bells and whistles by wiring up another forge, so long as it offered a decent API and/or sent events over a webhook, I'd have everything self-hosted.
The agents would need to expose an interface on their own end but as long as you implemented it with a plugin, it'd take the dependency of GitHub and you could use MCP or skills for the rest of it.
The neat thing about Tangled is it's built on an open protocol (https://atproto.com)—this allows us to effectively build an API-free system since all data on Tangled can effectively be ingested via the AT Protocol firehose.
Which is to say, this is perfect for agents given they don't need any bespoke SDK from us: simply write Tangled records for issues, pulls, whatever to your PDS and it'll show up on Tangled. We plan to start working on some exemplar agents first-party that would 1. enhance Tangled itself, 2. showcase cool things you can do with an open data firehose.
But, there are? I can host a repo on GitHub, Codeberg and self host it too. Then I need to watch over main to keep it consistent between those. After that's established, I can do updates from wherever. Link'em in the README.
There's also a tool to automatically push it to multiple repos: https://github.com/prashantsengar/GitEcho
Disclaimer: the author is a colleague of mine
Though to be fair, what the parent meant by federated forges is different than this approach.
git itself can push to multiple URLs btw:
https://stackoverflow.com/questions/849308/how-can-i-pull-pu...
There are distributed forges? Yes, git is distributed, but often everything around it isn't. The case parent is trying to make, is that the rest ("federated forges") should also be distributed, not just git.
Ok, gotcha. So there's a demand for the additional features that are not bundled within git to be federated somehow.
I'd say we have emails, mailing lists and bug trackers. Or maybe: what is the missing killer feature that needs federation?
> what is the missing killer feature that needs federation?
Issues, pull requests, collaboration/permissions/access, "staring"/"favoriting", etc.
I think ultimately the goal is that people can run their own forges, yet still collaborate on repositories hosted in other forges, leveraging your existing authentication so you no longer need to sign up individually for each forge.
Love the idea, would replace the LLM generated content ony our site, though.
I recently migrated to codeberg because I'm okay with self-hosting big runners, while using codeberg's available runners for smaller cron-based things (they even have lazy runners for this).
It’s… all hand written? We just sound “professional”.
I've never heard of this before, going to sign up and check it out!
Thanks! If you need anything, email me anirudh@!
What is "sovereign infra" exactly?
I know it's just marketing speak, but the term made me think of the scenes in the Matrix where what's left of humanity (ignoring all the cyclical lore that was added on top of it) has to make sure the machines can't remote in to any of their tech.
No less than self hosted, imo. If youre on some cloud it doesnt really matter that you pay them absurd amounts of money, you arent sovereign.
So if a company self hosts their physical infrastructure which will burn down once a fire sets in, they are more "sovereign" than a company running on a redundant cloud? I definitely would not want to be "sovereign" then.
Point is: This discussion is much more multi-dimensional than some suggest.
So literally a computer at home/in the office, as with anything else you don't really "own" the infrastructure? Or is this just about "cloud"?
Yeah sorry it's marketing BS speak for self-hosted or just infra that you control. It could be a VPS, it could be a Raspberry Pi at home. Your repos live on your servers. (And we support this on Tangled today!)
> just infra that you control
But a VPS isn't actually infrastructure you control, you essentially have as much control over it as "cloud", so I don't think that'd be counted as "sovereign", would it?
Perhaps, but it's still better than nothing!
> the future really should be federated
The internet should not be centralised, but you can't make a billion dollar company without capturing the world and selling your company to a trillion dollar company
Not enough attention is being put in the production/delivery mismatch.
GitHub is claiming they require 30x scale due to the giant increase in repository creation, PRs, commits, etc.
I have not seen a single product increase in features or quality as an end user, nor new significant products have come out in this period (other than the LLMs themselves).
Where is all this code going?
I understood it to mean, GitHub is being crushed by LLM/AI/Agentic code review and submission, not GitHub’s code itself
What I’m not seeing here but I am seeing with the Linux kernel is, most of the automatically submitted code is irrelevant or not useful
(Maybe that’s what you were getting at, apologies)
I for one believe Microsoft when they say this code is going to Github... to die.
Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once.
In companies, I see people vibe-coding "miracle apps" that fall under the smallest amount of scrutiny.
Basically people are doing the same developers do when they say "I can do this in a weekend", which is getting a prototype sort of running and then immediately losing energy (or in this case lacking ability) to push it forward.
> Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once.
Some people I know can't even explain what they are trying to create.
To die. I’m sure that’s nothing new for GitHub, but now it can happen at scale.
> Vladimir Fedorov is GitHub's Chief Technology Officer .... He currently serves on the board of Codepath.org, an organization dedicated to reprogramming higher education to create the first AI-native generation of engineers, CTOs, and founders.
I think I found the issue.
AS IF THEY POST THIS WHILE THEIR SEARCH IS BROKEN, what a circus
There are no words that Microsoft can use that would make me trust Microsoft.
Two incidents? Just two?
In seriousness, looking at their scale, this is an insane engineering challenge.
Especially if they’re moving databases, not easy ever, and certainly not at that scale
openai, anthropic, google and a plethora of chinese models all end up pushing code into github. you can discuss whether gpt 5.5 is better than opus 4.7, but for github it doesn't matter: they'll be receiving the code no matter which llm spits it out.
amazing on one hand, quite scary on the other for github and all other forges if this continues and there is no reason why it wouldn't.
Simple solution: charge all users. Charge more for higher usage.
so what they're saying is that Co-Authored-By claude@anthropic.com is overloading their systems?
and that azure cannot scale fast enough to handle the load so they're embracing multi-cloud as a company... owned by microsoft?
woah. what am I reading.
AI is the new DNS when it comes to service failure.
The AI agent growth explanation is interesting but also a bit of a deflection. If a meaningful portion of your traffic is now automated agents, your capacity planning model is fundamentally different, you're no longer scaling for human paced workflows but for burst patterns that look nothing like historical load.
The unlabeled graphs don't help the credibility case. When you are already in the hole on trust, shipping a post that requires readers to assume favorable baselines is exactly the wrong move.
> The main driver is a rapid change in how software is being built.
Leopard, meet face.
Too little too late, yesterday was the straw that broke the camel’s back for us and we’ve started a migration to a self-hosted GitLab.
This latest incident was the nail in the coffin for me. I've been on GitHub since 2012 but I'm feeling the pull to migrate out to Gitea/Forgejo. Has anybody done this recently? How'd it go?
When one of the incident they write about here happened, I wrote about my experience moving from GitHub to Forgejo which I happened to complete just the night before that happened: https://news.ycombinator.com/item?id=47878192 (lots of other people sharing their experience as replies too)
I was thinking of maybe doing a proper write up about how to host your own Forgejo + Action runners on Linux, Windows and macOS, not sure if there is enough interest. What would people for sure want to know in a guide/explanation of this?
I moved over back when GitHub was planning to charge per minute to use my own runner. It was easy with Claude, the gh API, and forgejo web API. I even set up daily backups to my S3 clone of choice.
The only repos I left on GitHub are forks and one with a bit of public engagement.
fwiw, I've had good luck scaling git, specially doing clones, in the HTTP layer, using Varnish. this was CI bringing Github Enterprise to it's knees.
Some interesting tid bits:
* we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL)
* * redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load.
* we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go.
I'd like to know what database backend they migrated to. I was also surprised to read that the migration from Ruby to a more performant language had not already been completed. I assume this is because it a large code base with many moving parts, etc.
Another interesting bit: they are hitting performance issues due to the rise of monorepos. GitHub and frankly Git were not designed for monorepos
Yet the Linux kernel is a monorepo
When there's a gold rush invest in checks notes jewellery makers?
> The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply.
GitHub instability has started way before that. I understand it’s too much to ask of a trillion-dollar corporation to consider the impact of their own actions, but perhaps they should’ve thought of that before forcing LLM development down everyone’s throats.
While they contributed, they were still following the market trend anyway. If they weren't letting folks use it directly, other companies would have (and are).
> they were still following the market trend anyway.
They started the trend with Copilot.
> If they weren't letting folks use it directly
There is a chasm of difference between “letting you use it” and “forcing it down your throat”. Microsoft is doing the latter, not the former. Copilot is annoyingly present by default at every step on GitHub.
I had to postpone a call with developers (in 2 different countries) because I didn't had access to the issues board, which is a single source of truth for us.
I understand the rapid growth (because of AI agents), but if such critical software service becomes unstable then it's time to migrate? Thinking about self-hosting GitLab.
> but if such critical software service becomes unstable then it's time to migrate?
Right way to think about this:
> If things we need/see as critical for our work are hosted on a platform with really bad reliability, it's time for us to migrate
My internet connection at home is really shit, and almost every week there is a multi-hour downtime for some reason, not to mention when La Liga games are on TV anything using Cloudflare is unavailable, so I've had to spend extra energy and time to setup things in a way so I can still work whenever this happens.
The bit nobody covers in these write-ups: small teams without dual-cloud failover budget. Last big GitHub outage cost me a deploy day. Not catastrophic but the kind of thing you don't budget for when GitHub is your single source of truth.
Status page is also still doing that thing where every component is green but in practice clone is hanging, push is timing out, actions are stuck. Per-service uptime is a managed number. The user-experience number is the one that matters and it's not in the post-mortem.
One thing is clear: an LLM wrote this.
I wonder if they’ll end the free lunch we’ve been having since the MS takeover. There’s been a deluge of spam and crapware projects due to the LLM wave which is visible in that graph. Can’t see them sustaining being a public dustbin for low value projects forever.
I could see them expiring/archiving/deleting inactive projects after some time.
I feel like this would have negative impacts (lots of interesting historical archives on Github) but maybe if a project hasn't been touched, or cloned, in some time, it just gets deleted with some notice.
I wonder if this mess has anything to do with talent loss resulting from layoffs after the pandemic
I’d guess it has much more to do with the extra load agentic ai generates. If we take the charts in the OP at face value, do you think gh suddenly exploded in popularity? At this point I think almost everyone who has any use for gh already has an account and use it as much as they ever would. But all the charts go to the moon. Gh obviously didn’t take into account that ai agents can generate a lot of activity they don’t have capacity for.
> While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud. This longer-term measure is necessary to achieve the level of resilience, low latency, and flexibility that will be needed in the future.
Wild
I have a hard time believing anything what's said in a blog post where a graph lacks axes labels/scale. It tells me that nobody who cares about correctness had any say on the content of the post. Maybe I'm being 8am cranky and pedantic, but I'm sticking with it.
> availability first, then capacity, then new features.
I'd love to experience first-hand a leadership team who says, "stop accepting new paying customers until we've got availability sorted out!"
Like they did with Copilot last week? https://github.blog/news-insights/company-news/changes-to-gi...
> New sign-ups for GitHub Copilot Pro, Pro+, and Student plans are paused. Pausing sign-ups allows us to serve existing customers more effectively.
I know that I'm simplifying (probably too much), but it seems like things were fine when GitHub was still a Ruby on Rails monolith and all the rigmarole with microservices etc. only made things worse.
Unless everything else stays the same (underlying traffic etc.) then you can't really compare. Could be that you hit some fundamental scaling limit with the old design and it completely falls over after a certain scale.
Oh as said I'm pretty sure things are more complex. It's just funny in a way that all these technologies that are usually being sold as "enablers for scale" don't seem to do their job very well.
This sounds more like a belief, based on little more than "correlation is causation", than analysis that controls for macro-trends backed by evidence.
It is, but everyone is entitled to beliefs. Anecdotally, I feel the same way. Everywhere I’ve been, there has always been a legacy monolith that was stable as a rock, with dozens of new microservices scattered around it in an attempt to exit the monolith. The microservices have never once been stable. People fail to take the most basic things into consideration, like “you can’t have Consistency and Availability when everything is a network call.”
I’m sure survivor bias is at play here, but when I look through the older code bases - especially the data model - it’s an entirely different world than the newer stuff, and it’s clear which of the two was written by people who understand systems.
GitHub been oscillating between long phases of "Never any new features but rock-solid and no downtime" and "New features every week but also unicorns (used to be the "service unavailable page") every week" for as long as I can remember. Seems they're on some interval switching between the two.
So I gather that nobody is working on a search that stays on the current branch?
As a solo dev GitHub going down is scary all my code, all my history, one platform. This makes me want to keep local backups more seriously.
Sorry to ask but... Do you have any idea how git works???
Yea or use another provider like codeberg
Personally I'd never use codeberg. Their FAQ on licensing [0] is basically everything that anyone who supports free software should abhor - it's "we might allow you to do what you want to".
[0] https://docs.codeberg.org/getting-started/faq/#how-about-pri...
True but switching is not that easy when all your ci pipelines and integration on in GitHub.
I don't think it's 100% compatible, but Gitea's/Forgejo's (which Codeberg runs on) own Action implementation is pretty much the same as GitHub Actions, with minor differences.
Good to know might actually try it for one project 1 st before switching
blame MySQL. Blame Ruby.
on another note - is the exponential growth from 'agentic' workflows actually resulting in productive software in the wild. Or it is just noise. On my end I haven't seen the software I use getting better.
Nice, they have availability numbers now on their status page, but they aren't aggregating.
If you multiply all current numbers together (as of Apr 28), you find out that GitHub has a 97.26% uptime.
One ... single ... 9.
They can do better.
Kind of unfair though, do the same for any platform with multiple services and you'd probably get <99% for most of them.
> you find out that GitHub has a 97.26% uptime
Calculating that to "Downtime per day" you get ~40 minutes of downtime per day, almost a week per year. Crazy stuff for something essential like this.
Availability is priority? Does not seem like it is https://mrshu.github.io/github-statuses/
> Our priorities are clear: availability first
That's a delayed April fool's right?
No, just a 6 month old memo that was first opened today, as they said literally the same 6 months ago.
Personally, I’m sympathetic. We know that GitHub did a huge amount of work over the last decade to make Git scale, which has benefited us all. These new scaling challenges are real challenges, 30x growth would be a nightmare for any system that was already pushing the limits of what was possible, I think we are being far too hard on GitHub, they deserve a little grace.
For all the negatives about github I agree. They offer a lot of free stuff, and LLMs seem likely to put massively increase their costs with no guarantee they'll be making money off it. I can't think of many (any?) large businesses which could scale up to meet so much new demand without some significant growing pains along the way.
GitHub's scaling issues are caused by their own vendor-lock approach and monopoly. Yes, of course _their_ goal is to be even bigger and even more all-consuming, so _they_ have to deal with the scale. Why a user would be sympathetic to that?
The user (and not a big tech monopoly) answer to scaling issues is almost always to stop scaling and start federating and interoperating.
Load from paying customers vs. load from nonpaying users would be interesting to know. No doubt omitted deliberately.
Regarding their image with stats (https://github.blog/wp-content/uploads/2026/04/record-accell...) - what exactly are the ranges on y-axes? I doubt they had close to 0 PRs merged in 2023 ;)
> we accelerated parts of migrating performance or scale sensitive code out of Ruby monolith into Go.
I am surprised that Microsoft is allowed to use Go. How long will it be before a bean counter forces a rewrite to a Microsoft favored language?
So they haven't even finished migrating from their datacenters to Azure and have now started a project to add another cloud provider ("multi cloud")? Madness.
anyone who's actually worked there, could you explain why they're finding scalability and reliability so hard? naively it seems like 'repo groups', ie clusters of repositories linked by being mutual forks, would be fairly isolated for the whole git storage layer, and everything else feels pretty easily parallelisable (issues, actions, etc, modulo taking locks now and then to submit results or whatever). and given that, surely you can incrementally deploy changes across those many shards to avoid most big outages?
are there big conceptual serialisations that I've missed? is it just not well factored? was the move to Azure just a catastrophically bad idea? some other thing?
Almost every high volume service on the internet is write a little, read a lot, and when there are writes, they're relatively small, a few bytes into a database that can fan out. GitHub is very different: constant writes, large files, it is under far more pressure than the systems the rest of us build. And then, as the article says, vibecoding happens, and suddenly they're receiving 30x the volume of expensive operations. GitHub are responsible for many of the performance improvements made to Git over the years, Git scales today because of work GitHub did, but that work was never intended to scale to volume of today.
Even as recently as 18 months ago, Lovable appeared, seemingly overnight, and caused huge problems for GitHub because they were creating repositories on GitHub for every single Lovable project, offloading the very high cost onto GitHub, hundreds of thousands of repositories. A couple of years before that, Homebrew used GitHub as a de facto CDN and that was a huge problem, too.
Nowadays it is easy to imagine how we can scale out a service like Twitter or YouTube or Facebook because everything has been done before, but that's not true of Git, Git hasn't ever scaled like this before, there are very few examples of service with GitHub's characteristics.
https://lovable.dev/blog/incident-github-outage
https://news.ycombinator.com/item?id=42659111
recently there was a twit how GitHub PR diffs had 10 React components PER LINE. And how they optimized that to only 2 React components per line or something.
> To summarize, for every v1 diff line there would be:
> - Minimum of 10-15 DOM tree elements
> - Minimum of 8-13 React Components
> - Minimum of 20 React Event Handlers
> - Lots of small re-usable React Components
https://github.blog/engineering/architecture-optimization/th...
I'm asking about the infrastructure, obviously they chose for some reason to make my computer fans turn on to show some red and green lines on a text file.
terrible frontend architecture suggests poor engineering culture which typically spreads to all teams, including the infrastructure team
Ruby catching strays
Good chuckle out of this post, it’s crazy that neither Atlassian (Bitbucket) or Gitlab are capturing value out of this same agentic coding boom. I wish github was separately publicly traded outside of Microsoft.
Nowhere to get exposure to this
I’m pretty sure my Forgejo instance on a Raspberry Pi is outperforming GitHub reliability. It’s faster that’s for sure.