Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.
I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"
Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?
> has something changed to cause this many outages?
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
Based on the comments in this thread, you'd think that Reddit was down.
Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.
I'm surprised people still rely so much on GitHub. Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
So, "git push" is down ... how are github engineers gonna push the fix now?
rsync and sftp, like god intended
Like the olden days
Nah in the olden days we'd load a tape with patches and the code would read a patch off tape, apply it, then read the next one. lol.
in vim over ssh of course.. agents can use vim right?
Yeah, but they keep trying to use Emacs commands...
C-x M-c M-unfuck-git
sudo git push
Make no mistakes.
I can attest to the outage starting around 11:05, as that’s when my GitHub auth for Tailscale SSH broke. Of course, the next 10 minutes of troubleshooting and checking GitHub’s status page showed “all systems operational,” so I assumed it was something on my end. Then, it started working again.
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
Well it is a Wednesday.
I noticed because SSO was down. They don't even report on that as far as I can tell.
gitea, forgejo, gitlab CE, and ssh still working fine
I never noticed downtime with public GitLab. It's also generally better than GitHub and Actions.
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
Of course. Azure is the reason GitHub is down so often, too.
Exactly. I'm running forgejo in a container, pushing backups to a private S3 storage, and sent the forgejo team a small liberapay payment.
Maybe 30 minutes to get it all setup and then I moved on with my life.
GitHub added new webhook IP ranges this morning, curious if that's related.
That might be the cause but let's see.
There 4 updates contains webhook mention in 2 of the updates.
"Webhooks is experiencing degraded performance"
What's frustrating is that:
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
Funny, yesterday git operations had an unreported outage too. I couldn't pull changes for ~10 minutes.
Actions being down is one thing - I can't even land a PR.
Same here. I wanted to push something on Github and its was giving 500. This is most severe issue in recent times.
PR? I can't even push haha
I can't even create issues (error 500)
I'm tired, boss.
One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"
99% of the time, this is a server returning a generic 500 HTML response to an HTTP call that was expecting a JSON response.
I thought it was odd that we'd not had one of these for a few weeks.
A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
There was one yesterday.
Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?
> has something changed to cause this many outages?
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
It also coincides with GitHub moving to exclusively Azure infrastructure. Azure is a terrible shit show for reasons that have nothing to do with AI.
Give them some slack. Their core product is AI, not version control
> Was github always this shoddy or has something changed to cause this many outages?
The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/
They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
I mean... they got acquired by Microsoft.
Almost 10 years ago
Three days in a row now right?
first time seeing "git push" on github throwing an "Internal Server Error"
The last time GitHub went down was...yesterday.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
This is becoming way more frequent than a monthly event. Does anyone know why GitHub keeps having incidents?
Because it's vibecoded and because it's running on Azure
It’s mentioned on most threads, but their traffic and usage is way up due to AI / agentic programming.
Love it
i see they started letting the github engineers ship code again after that shitshow last month.
"Rewrite GitHub in Rust. Make no mistakes."
That's what you get when the bulk of open source is now depending on Microsoft.
Another case of tactical downtime.
Color me shocked
This is so frustrating. I wish it were easier to migrate away from this tool
holy crap push is a new low
Who even runs GitHub at this point?
According to themselves, 225 million people.
https://github.com/about
I just work here man.
Copilot.
Who's the commanding officer here, soldier?
codefloe.com is still up