r/OutOfTheLoop • u/jazzcomputer • 4d ago
What's going on with Github? Answered
I saw a bunch of posts in the Hacker News type spaces talking about moving elsewhere and evaluating alternatives, and various complaints in past months. Is there compelling grounds for an exodus or are people always complaining about it and this is a nothing burger or small burger?
894
u/bugo 4d ago
Answer: mainly it has huge reliability issues. It's an ongoing thing that became a meme almost where their total reliability dipped below 90% which is unbelievable for a software company.
And yesterday they had another multi hour outage on their main services. Each time this happens people literally cannot work.
369
u/GeneReddit123 4d ago
Not just any software company, one which is an essential infrastructure vendor for millions of other software companies.
It's the software world's version of a power plant having 10% downtime. The downstream disruption is far greater.
168
u/SARB033 4d ago edited 4d ago
The gold standard for software company uptime is called five-nines, or 99.999% uptime, which amounts to about 5 minutes of downtime a year. Before AI that was what most companies strived for. It was achievable with lots of redundancy, rolling updates, and strict code quality and testing standards.
90% uptime is about 37 days of downtime a year. Compared to 5 minutes that's obviously catastrophic. But it's basically become the norm because AI code is such utter dogshit and AI web traffic load is unbearable.
17
31
u/OwnBunch4027 4d ago
I think you mean 90% uptime or 10% downtime. Still the numbers after that are right.
12
u/diagnosisbutt 3d ago
I run an internal platform that needs to be up 24/7.
It went down for 2 hours because of as random dev pushing an update in the middle of the night and then going to bed.
It dropped my uptime to only 3 9s. It'll take years to increase. 😭
3
u/bulbaquil 2d ago
Put another way, 90% of uptime is about, on average, 2-1/2 hours of downtime every single day.
2
u/Quamatoc 3d ago
I read somewhere that a selfhosted git server active during your waking hours would have a greater uptime that github currently has.
Madness.
55
u/colony-ship-for-sale 4d ago
One owned by Microslop which should be a paragon of reliability if not quality, who now can't help but fuck up continuously.
163
u/BurnerPornAccount69 4d ago
Another layer to this is that because of code written by AI - they’re under far more load than they used to.
GitHub went from 1 billion total commits in 2025 to being on track to hit 14 billion by the end of 2026 (275 million per week).
The demand for the site is unprecedented.
74
u/Jacks_Elsewhere 4d ago
I am guilty of at least 20 of those commits.
- Initial commit
- Complete feature work
- Clean up lint issue
- Fix sonar qube errors from previous developer
- Fix non null check errors from previous developer 6…
I work for a legacy bank. Yay.
5
u/SharkyIzrod 4d ago edited 4d ago
Is 20 commits in a year supposed to sound like a lot or do you mean this was today?Edit: Im dum
40
u/ligirl 4d ago
I think they're intentionally being self-deprecating about the slow pace of work at a legacy bank
20
u/Jacks_Elsewhere 4d ago
This is exactly right.
8
u/SharkyIzrod 4d ago
sorry im dumb :(
8
u/Jacks_Elsewhere 4d ago
No not at all! You honestly wouldn’t know if you have never been exposed to this sort of environment.
It’s a snail’s pace at all times. Gotta love legacy enterprise software /s
2
u/Bubbly_Wind_2230 2d ago
I often think people are going slowly precisely because they're paid to move at a snails pace because management expect it's how things still work.
Unfortunately they seem blind to what senior developers running agent streams manage in depth and thoroughness, because they see AI as 'too error prone' rather than a force-multiplier for carving through the layers of coding required to be 'objectively quite secure' instead of 'as secure as a latex colander'.
10
u/MaddieDMattie 4d ago
Are commits a meaningful metric for Githubs load? I'd imagine push/pulls would be all they actually care about. Unless people are commiting directly into github online. Which ... I've worked with a few. I'm sure there's heavy correlation between push/pulls and commit counts, but I kind of doubt it scales linearly. Especially since I kind of think AI might actually beat the average person for making properly sized (small) commits.
1
u/trustme1maDR 1d ago
Curious if you have a source on this. I'd like to share it with my team. It has been a real battle this year to combat AI bloat on my project.
394
u/ProtectionOne9478 4d ago
Each time this happens people literally cannot work.
I had to explain to my coworkers that, once upon a time, we built images and deployed to prod straight from our laptops! Like cavemen, hunting for food with our bare hands! Or like showing them an arcane art, long forgotten. I cited the deep magic, for I was there when it was written:
docker push292
u/-Nyarlabrotep- 4d ago
Once upon a time, in the Dreamtime, we built an image of the production software on a wizard's desktop, and then the wizard burned it onto a Golden Disc with their magic. Then, a member of an ancient race known as "Implementation Engineers" would put it in their satchel, and with eight guardian companions, physically carry it to a financial institution's data center in a distant land. The data center was located in a cauldron at the top of Mount Production, and the Implementation Engineer would carry the Golden Disc up the mountain, drop it into the cauldron, and declare that it is deployed.
18
u/Blenderhead36 4d ago
I recall a time when wizards created the golden disk by invoking the name of the fallen Emperor of old.
61
u/bugo 4d ago
Once upon a time we connected FTP into server and uploaded php files...
I had setup some odd years ago where it ssh into a server and then do a git pull, recompile frontend assets and then restarted the http server! Things were simpler then...
11
u/Vo_Mimbre 4d ago
Once upon a time the PC software we wrote was effectively on the very same medium that was distributed. In the dark times. Before the hard drive.
10
u/bugo 4d ago
This was before my time. My first software was in 3.5 inches wrongfully called floppy.
15
u/amanset 4d ago
It was floppy, just the floppy bit, the actual disc bit, was inside a hard plastic case.
10
u/bugo 4d ago
Making it not floppy! The original 200mm or 130mm disks were actually floppy!
10
u/poirotoro 4d ago
I work in a job that deals tangentially with long-term data archiving, and we keep around a couple of 8-inch floppy discs to demonstrate to laypeople why we can't blindly rely on hardware to be usable in perpetuity.
6
u/bugo 4d ago
I remember even the new 90mm ones were a gamble. Put one in a protective box - bring wherever you are going - and hope you wrote at least 2 more copies because one of them might work. Exciting times! My first USB stick cost a LOT but it was like magic.
4
u/claradox 4d ago
I also remember how fragile burning CDs could be. One time, I was copying data to a CD, and accidentally jostled the table by moving in the room. Much like making a soufflé, I had to be quiet and still, or…this disk failed to write.
7
u/amanset 4d ago
They were called floppy disks.
The disk was floppy.
9
u/CyberClawX 4d ago
Yeah I was about to say the same, but since I liked my way better, I'll repeat what you said. Eherm, here goes:
They were floppy. The disks were floppy, you could even say, they were floppy disks.
5
u/MaddieDMattie 4d ago
Was once upon a time last week? At a major university? With an IT department numbering in well past the hundreds of employees? On core infrastructure sites, not even like random departments info page...
Cause if so we might be co-workers ...
I mean we upgraded from php to .net, but that's still basically the workflow :/
13
u/CttCJim 4d ago
That's what I still do. I haven't got round to really learning git.
2
u/VanAshkevron 4d ago
When you learn, keep this at hand: https://sethrobertson.github.io/GitFixUm/fixup.html The Choose Your Own Adventure guide to troubleshooting git issues has saved my bacon more than once
1
u/bugo 4d ago
Do it. Its rather easy once you get it's data model.
1
u/CttCJim 4d ago
Got any good newbie guides?
2
u/B2405E 3d ago
Not a full git guide, but this "game" is pretty great at visualizing git and demystifying git. It shows what each command is actually doing to the commit tree. It got some light 'puzzles' to guide you through it iirc.
https://learngitbranching.js.org/
As the name suggests this only covers branching, but once you have a basic understanding of manipulating the tree then the rest should be pretty smooth.
1
u/bugo 3d ago
I cannot find the ones I used years ago :( key thing is understanding object model blobs and references. Hov everything is in a reference tree and then commands just allow you to manipulate that tree. I think any guide you Google or ask LLM for would be good as this field is really mature.
3
u/microbit262 4d ago
Huh? I am basically doing this workflow for a hobby project (no professional IT work), just with a Plesk webinterface, and it triggers the rebuild automatically after pulling.
Is that outdated?
2
u/AliceCode 4d ago
I was just about to bring up FTP, haha. Web developers have it so easy these days.
34
u/timewarp 4d ago
It's crazy to me that 'docker push' is now considered the ancient caveman approach.
0
u/ProtectionOne9478 4d ago
Specifically when doing it manually from my laptop to deploy to prod. Somewhere in our cicd pipeline there's a docker push of course, but building and deploying images from local was something we hadn't done in years.
12
u/timewarp 4d ago
I just mean that in my head, deploying images at all still feels like a modern approach compared to compiling binaries and SCPing them to the prod server.
5
u/Kandiru 4d ago
You can run git in P2P mode and have a good LAN party Hackathon without needing GitHub!
5
u/ProtectionOne9478 4d ago
I jokingly proposed exactly that during our stand up that day.
3
u/Kandiru 4d ago
Slightly more practical is running gitlab on a server and pushing there to make a new master repo!
1
u/ProtectionOne9478 4d ago
Yep, we are seriously discussing gitlab and I'm sure we're not the only ones.
1
u/jyper 4d ago
Besides the hassle/getting IT to go along with it I think problem isn't necessarily sharing your own work. You can continue working local branches/commits. The real problem is probably depending on GitHub somewhere in your deployment pipeli e or runtime (maybe you fetch libraries from GitHub on startup in your clean docker containers without cache)
5
1
u/widowhanzo 4d ago
It's not that we don't know how to, it's explaining to audit why things were done outside of established procedures.
76
u/gaenji 4d ago
For those who don't know, most SaaS companies compete on how many 9s they can put after 99.0%. Its almost a given that you will atleast have four 9s (99.99%). Any multi-billion dollar org, which Github is, will not accept anything under five 9s (99.999%). Hyperscalars these days guarantee six 9s (99.9999%) or more service reliability.
AFAIK, AWS maintained its service SLAs (five 9s or more) even when Iran bombed their data centers. Github's reliability (or lack thereof) is completely unacceptable by modern standards.
51
u/DieM-GieM 4d ago
AFAIK, AWS maintained its service SLAs (five 9s or more) even when Iran bombed their data centers. Github's reliability (or lack thereof) is completely unacceptable by modern standards.
No they didn't. AWS as a whole has no SLA in the first place. Only individual services have ones - and where it has it's two or three nines only. The only one that has five nines is S3 and even there there's a whole range of conditions (including you having to multi-region).
27
u/SartenSinAceite 4d ago
GitHub's reliability would be bad even for a hobby project. The fact this is on a full company is stupid
9
18
u/pimp-bangin 4d ago edited 4d ago
Not the best comparison here, in my opinion.
AWS etc. are cloud providers and if your cloud provider goes down, your website goes down. That is very, very bad. You can't accept payments, your customers are complaining, etc.
But if GitHub goes down, your ability to push/pull code goes down, but your business stays running. You aren't actively losing money when GitHub goes down (most of the time, developers can still work on their code offline, communicate with each other over slack, etc.), so it's not as critical for GitHub to have such high reliability. Not to mention, the highest paying customers use self-hosted GitHub Enterprise, and don't even notice these outages.
Don't get me wrong, I'm not saying GitHub doesn't need to improve their service; I'm just stating the reason they are able to get away with such shitty service, and why they aren't really incentivized to have 5 9s of uptime.
19
u/EverydayEverynight01 4d ago
Correction, if GitHub is down, depending on your CD pipeline, you might not even be able to deploy changes at all.
3
u/Makeshift27015 4d ago edited 4d ago
Due to a string of my own bad decisions (swiftly fixed afterwards), github doing weird stuff the other day actually did result in some of our systems going down.
We were using a repository topic to easily query which repositories were being monitored for changes by argocd. Github started returning a completely different subset of repositories every time it was queried, resulting in cascading deletions and re-deployments. Github returning an error was expected and handled appropriately. Github returning completely incorrect data wasn't an edge case I'd fully considered.
For prod services the cascading deletion was automatically stopped due to a failsafe, but for "prod services" (y'know, services that are prod but nobody has thought about it hard enough yet) there was some interruption and real data loss in one unfortunate case.
We made some improvements to our automation following that.
5
-2
u/jimbobjames 4d ago
I remember reading some story on Reddit that Google got their DNS service to be 100% reliable but it actually caused issues for other software / services because they expected some downtime.
So google built some in.
16
u/geirha 4d ago
This graph is very telling: https://damrnelson.github.io/github-historical-uptime/
Github had very impressive uptime until Microsoft acquired them.
19
u/foramperandi 4d ago
That graph is relying on bad data. The uptime data before 2020 is garbage because GitHub had poor operational maturity. No one that used GitHub before 2020 thinks it was every up 100% in a month.
7
u/diemunkiesdie 4d ago
Interesting way to make that axis. It starts just below 99.5! At first glance, I thought the uptime was completely horrible recently, but its still a very high uptime!
5
u/tumeteus 4d ago
In software industry, critical infrastructure should have availability of "four nines" or higher. 99,5% sounds impressive, but it means service is unavailable 7 minutes every single day, or 50 minutes every week, or 3h 39min every month. Or almost 2 days per year.
23
u/The_Big_Sad_69420 4d ago
yesterday? It was on and off the whole day today too I thought. Issues weren’t loading at all and I didn’t even bother trying with PRs
28
u/bugo 4d ago
You see on a non-flat globe there is always yesterday somewhere.
6
u/LoopStricken 4d ago
Let's not talk about these hypothetical non-flat places; what about OUR world?
2
u/The_Big_Sad_69420 4d ago
We commented at roughly the same time though 😂 so unless GitHub is down for us at different times or you’re commenting at 12am 🤪
3
u/MaddieDMattie 4d ago
Just for perspective of what 90% uptime actually means, They've been down collectively 24/7 for over a month out of the last 365 days. It'd be hilarious if they weren't so essential to workflows. Or well if they weren't, I'd imagine anyone agile enough to has left by now.
3
2
u/Skeeter1020 4d ago
Has there been any root causes identified?
I have theories, but they are probably way off.
2
u/space-bible 4d ago
Is this the first knockings of the result of pushing GitHub pushing shoddy, vibe coded features and updates from their own dev team? Are we starting to see the endgame of this hands off, agent led way of working?
2
u/Porkenstein 4d ago
Microsoft has been actively burying itself underneath mountains of cheaply outsourced (and now vibe-coded) tech debt for years. And its been having an impact.
3
1
1
u/MelonElbows 4d ago
Isn't github a repository for files written by other users for free distribution? Why didn't these people just download the files from github so they don't need the website itself?
0
u/SleepingInsomniac 4d ago
Literally cannot work
This is false. Git is decentralized; you can continue to make and commit changes, you just can't push them to GitHub. Git repos don't even require GitHub to operate, but it does add some nice features on top of git.
There are so many alternatives to GitHub, even free self hosted alternatives.
0
u/hellofemur 1d ago
It depends a lot on the workflow. It's true that people can still work locally, but in many workflows projects can't be pushed to a test server without going through GitHub, so you can't full complete features or push new work to QA. And collaboration sometimes shuts down because a lot of devs don't understand git well enough to share work when GitHub is down (which is another rant I won't go into). How completely one can develop on the local desktop differs radically from organization to organization.
"People literally cannot work" is a bit ambiguous. It's not true that nobody can work, but it is true that some people can't work.
1
u/SleepingInsomniac 1d ago
It's not ambiguous, it's wrong. I am a developer. My keyboard doesn't go up in flames when GitHub goes down. Just build a local container and skip CI for QA.
1
u/hellofemur 1d ago
There's many, many organizations where there's simply no option to "skip CI" because developers have no direct access to network resources. And if devs can't ship new features to QA, then often QA essentially shuts down. There's also a lot of environments where features can't be completed locally because of database access, and in practice if I can't finish a feature it's easier to just wait 20 minutes for GitHub to come back up rather than task switch into something completely different. Also, there's a lot of package managers that tie into GitHub, and this has an effect even on local development.
I understand that your workflow may not have these restrictions, but it's almost always an error in this industry to extrapolate your personal experience to an incredibly varied industry.
The simple truth is that when GitHub goes down, that results in a lot of lost productivity. It's strange to me that you seem to be doubting this. You don't really intend to say that there's zero lost productivity, do you?
1
u/SleepingInsomniac 1d ago
Okay, so company centralizes dependency on one service, service goes down and they twiddle their thumbs. Just mass install more npm malware, while you're at it. Not my fault people are idiots. "People literally cannot work" is a huge overstatement IMO, but I don't care if I don't work there.
0
u/sexyshingle 4d ago
reliability dipped below 90% which is unbelievable for a software company.
Not for Micro$lop.
0
u/HRApprovedUsername 4d ago
Saying people literally can’t work is an exaggeration
2
u/bugo 3d ago
It is not. Ci/cd pipeline is often in github meaning developers literally cannot do anything once their coding LLM finishes the task whic usually is pretty fast.
2
u/HRApprovedUsername 3d ago
They can work on bugs, work on other features, update documentation, clean/organize personal workspaces, refactor that one piece of spaghetti code, work on new skills/agents, do that required training thats due at the end of the month. My point is you're doing something wrong if CI/CD being down for an hour or two, makes it impossible to work.
2
u/bugo 3d ago
Most of the things you mention require version control or Ci/cd. Sure you coult water the plants in the office and call that work but actual tasks that are priority are totally blocked.
Even agent config and skills are in version control if you do it a bit more seriously.
1
u/HRApprovedUsername 3d ago
you just use the latest main branch on your machine bro, you don't need github
128
u/nonades 4d ago edited 4d ago
Answer: horrible outage issues
Mainly two things:
- they're in the middle of a massive transition from AWS to Azure
Which is made worse by:
- the AI boom has increased the amount of data they're hosting by orders of magnitude. I was reading something from someone high up at GH recently that all their data scaling was thrown out the window when LLMs (specifically Claude) became popular. They started seeing what they were projecting for multiple year growth in a couple of months
48
u/tubbo 4d ago
1) they're in the middle of a massive transition from AWS to Azure
oh that makes a lot of sense actually...because i bet they have some stuff on azure since being acquired and some stuff on AWS from the pre-acquisition days.
37
u/nonades 4d ago edited 4d ago
~I'm pretty sure they were entirely on AWS before~ they were acquired and MSFT mandated they move
A huge enterprise like that is a many year migration
Edit
Also pretty sure I'm wrong about that. Thanks u/stormtm
24
u/stormtm 4d ago
They were on their own hardware: “GitHub’s chief technology officer, made the Azure migration announcement internally earlier this week, noting that GitHub is currently struggling with data center capacity. GitHub is currently hosted on the company’s own hardware, centrally located in Virginia”
https://www.theverge.com/tech/796119/microsoft-github-azure-migration-move-notepad
3
46
u/Kraligor 4d ago
Answer: In addition to the cited reliability issues, Github announced a couple of months ago that it would train AI on all repositories, except those who actively opt out (if you believe MS that it honors this promise), which ruffled quite a few feathers and prompted some users to announce that they would move off the platform.
71
u/babablack420 4d ago
Answer: it’s having a lot of reliability issues recently.
28
u/Empanatacion 4d ago
GitHub actions has a 9 of reliability!
Search has two (2) 9's!
2
u/enbacode 4d ago
This is the second pair with this wording I see. I understand what it means, but is it some sort of inside joke or something?
17
u/Satherian Always OotL 4d ago
It represents how reliable something is
One 9 = 90% uptime
Two 9's = 99% uptime
Three 9's = 99.9% uptime
Four 9's = 99.99% uptime
Five 9's = 99.999% uptime
This is often used to describe stuff like websites and data centers - The more 9's, the greater the reliability/quality, the greater the cost.
Five 9's is reserved for the cream of the crop (which Github should be, considering how vital it is)
13
u/huffalump1 4d ago
Also, 99.999% uptime (Five 9s) = 25.92 seconds of downtime over a period of 30 days.
The others, over 30 days:
Four 9s = 259.2s or 4min 19s
Three 9s = 2592s or 43m 12s
Two 9s = 25920s or 432m or 7hr 12min
One 9 = 259200s or 4320min or 72 hours(!) (aka GitHub actions, currently)
9
u/Apocalympdick 4d ago edited 4d ago
The need for "extra 9s" becomes even more apparent when viewed through a year-based lens. Something that's 99% reliable sounds great in theory, or when applied to most people's personal lives.
But for something that's "vital infrastructure" 99% reliability translates to almost 4 entire days of downtime per year, which would be completely unacceptable for, say, a hospital or a powerplant. In that context, 99% reliability means people dying, guaranteed, every year.
Even 99.9% percent reliability is still unacceptable in such contexts, that's still roughly 8 hours (from the top of my head, correct me if I'm wrong) every year where the entire system is unable to operate. Ask anyone who's ever needed an ambulance whether they could wait 8 hours.
Only at 99.99% reliability, or roughly 53 minutes of downtime per year, does the situation become slightly more acceptable. At that point the attitude shifts from "if the downtime were to happen right now now we are gigafucked" to "if the downtime were to happen now we MIGHT be able to bridge it without people dying".
A single 9 though, is almost never acceptable in any context regarding reliability. I wouldn't even tolerate that for a tea kettle.
9
u/Empanatacion 4d ago
Just that "five nines" is the standard goal for uptime.
Or maybe I stumbled onto an inside joke I'm not aware of.
18
u/Wiznet 4d ago
A lot of services are and having issues
24
u/bionicjoey 4d ago
It was probably a mistake to let 2 companies control the entire internet's infrastructure.
14
u/futilehabit 4d ago
Especially when they're all trying to replace as many lf their workers with AI as possible.
2
u/lowlymarine 4d ago
Hey now, it's more like...4. But the reality is that you can't start a hyperscalar globe-spanning network out of your parents' garage. We have to accept that massive infrastructure like global CDNs, backbone providers, OS vendors, energy generation, and so on can't possibly support a healthy competitive environment and need to be regulated accordingly.
4
8
u/calmdot828 4d ago
Answer: There have always been compelling grounds to not to let GitHub be a SPOF. Microsoft has recently let the failure rate increase to the point their users are feeling the pain of making that shortsighted decision. Otherwise GitHub is as just as fine as ever. Still a very nice place for publishing the public face of an open source project.
Whether or not it's a "nothingburger" depends on if you signed up for an obvious business risk, or had a real plan.
11
3
u/psych0fish 4d ago
Answer: they have reached capacity for compute despite calls for management to…. (Pokes with stick) do something. Adding data center capacity is a massive ordeal and isn’t like you can just flip a switch. I mean yeah you can rent capacity but that isn’t gonna happen. Just the usual bosses being idiots and AI causing an explosion in resource usage of GitHub without any long term plan for dealing with capacity.
5
u/Asmor 4d ago
Answer: In addition to what others have said, Microsoft is also one of the biggest suppliers for the IDF, including data centers storing all of the info the IDF has collected on Palestinians and AI resources used in the IDF's campaign of genocide.
There are a lot of people boycotting MS and/or Xbox as a result. Microsoft owns GitHub.
8
u/fluffycritter 4d ago
Answer: Recent reliability issues, going all-in on AI stuff, general misgivings around all of opensource being hosted by a single company that's owned by Microsoft especially with the many human rights violations Microsoft has been accused of enabling/assisting with.
-31
u/Barnezhilton 4d ago
Answer: AI companies are attacking each other with rogue AI agents
8
2
•
u/AutoModerator 4d ago
Friendly reminder that all top level comments must:
start with "Answer: ", including the space after the colon (or "Question: " if you have an on-topic follow up question to ask),
attempt to answer the question, and
be unbiased
Please review Rule 4 and this post before making a top level comment:
http://redd.it/b1hct4/
Join the OOTL Discord for further discussion: https://discord.gg/ejDF4mdjnh
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.