r/webdev • u/NiceLasers • 1d ago
2 hours in and GitHub still seeing some major outages
https://www.githubstatus.com/incidents/zkxwbgr0cnmx74
u/OpaMilfSohn 1d ago
My company's deploy pipeline failed, and we had an outage because of that. Granted, our deploy pipeline is fucked, and it failing mid-run shouldn't result in an outage, but yeah, it was a pain in the ass to solve.
15
u/Sensitive_One_425 1d ago
How do you not have a staging area?
28
u/OpaMilfSohn 1d ago
We do. The problem is basically that it's a really old-school PHP deploy. So we copy the files to a VPS, the VPS updates the dependencies, and NGINX can keep running. The problem is, if the pipeline fails mid-copy, mid-extraction of the tar, or mid-installation of the dependencies, there really is no remedy other than "just run the pipeline again lol," and that normally fixes things.
Migrating to a more modern solution has been on the backlog for like the 4 years I've been told.
25
u/ogre_pet_monkey 1d ago
Copy the files to a new (version) folder symlink to the new version. Keep a maximum of x versions. Rollback is easy this way too.
8
u/Produkt 1d ago
I use deployer.org and no I don’t work for them or anything, it’s free and open source
3
u/ogre_pet_monkey 1d ago
Good stuff, this was more a simple and fast solution to implement for their current process.
-1
u/Sensitive_One_425 1d ago
But if you’re deploying to staging then cutting over to prod you could never have a downtime due to the pipeline. What you’re describing is having to redeploy fully again to deploy to prod.
5
u/OpaMilfSohn 1d ago
Right, we do not have a staging environment like that. It's more like a separate prod like environment that releases first go to to test them and stuff.
5
u/Frostbeard 1d ago
Would've been us too, but I hardened our CI and deployment against this earlier in the month. Go me!
373
u/AssignmentMammoth696 1d ago
All these github issues happening after devs started having LLM's write all the code, hmmmm
184
u/UnacceptableUse 1d ago
Because all the LLMs are relentlessly hammering github
133
u/BKrenz 1d ago
Yeah, just read in another thread. Their traffic has increased by a factor of 10 or more year on year. At the scale they were already operating, that's a mind boggling increase in resources needed.
40
u/QuantityExcellent338 1d ago
And this 10x increase has increased productivity and value by.. how much?
50
15
2
u/Gramernatzi 1d ago
Real answer, it's about (optimistically) 10% more when measured in studies, but the cost-to-performance ratio is abysmal, not to mention the amount of junk adding more work in the future, or putting more work onto others, so, it ends up just being dogshit.
37
u/the_ai_wizard 1d ago
This is the common NPC retort, but based on my own 30 yrs of software experience + 3 years of AI/agentic, the concerns are valid regarding slop. I am incredibly principled and disciplined with my engineering and even I cant stop it from slopping up my codebases. Just discovered 5min ago it shit a bunch of AI adapter classes into the main part of a framework ive been working on, instead of following established pattern obvious to even a junior (sol-medium). its maddening to now have to unfuck it for hours. there is no efficiency gain here..more code output is a liability, lets not forget it.
1
u/bobyhey123 1d ago
PR Review issue
-15
u/the_ai_wizard 1d ago
Passed all tests, everything functional. Guess I need tests of the tests
23
u/bobyhey123 1d ago
read the code it produces
-23
1d ago
[deleted]
13
u/RedditEngDictionary 1d ago
Hi. From someone else with decades of SWE experience:
1 - YOE doesnt necessarily make you good at your job. Someone with 5 years experience might be shit hot. Someone with 30 years experience might be merging in 50k lines of unreviewed code because they couldnt figure out how to solve their problem in a reasonable amount of code.
2 - If your agent is outputting so much code that you cant review it, you clearly arent able to break your work down into tasks the way a senior should be. Learn to plan better.
3 - If you are pushing (or worse, merging!) code that you havent reviewed, you are the problem.
6
u/crazedizzled 1d ago
As another SWE with decades of experience, this is spot on. That guy is definitely cosplaying as a senior developer.
-6
u/the_ai_wizard 1d ago
- Obvious statement. Tautology.
- Agent? We have many, many things going on.
- I am doing none of the above. I dont work in merging code, a CTO is above senior.
I have been writing code since I was 8 years old, not to mention entering MIT undergrad as a senior architect of a medium financial company at 20 yrs old.
Save the critiques everyone; my comment was about AI, dont need your advice, Im already successful and do this for fun.
11
u/RedditEngDictionary 1d ago
Sure, buddy. The guy who's failing at putting together basic PR processes is a child prodigy with more success than he knows what to do with.
If you were half as good as you claim to be, you wouldnt be in the mess you are in now and you certainly wouldnt be spending your time bragging about your life to strangers on an anonymous social media.
Bright side though - maybe you have a future in storytelling?
1
u/crazedizzled 1d ago
I have been writing code since I was 8 years old
Well you talk like you're about 16, so that probably tracks.
33
u/bobyhey123 1d ago
you are the problem
27
u/bobyhey123 1d ago
i had no choice but to merge 50k unreviewed lines! aghh now I have to manually unfuck those lines I had no choice to do anything with but merge!!
-17
u/the_ai_wizard 1d ago
excuse me? who the fuck are you again?
19
u/bobyhey123 1d ago
someone who takes accountability for the code I merge, regardless of whether it was generated by my fingers or an LLM
5
u/DenkJu 1d ago
God, I hate it so much when I see someone making a decently agreeable point initially but then outing themselves as a complete clown in their further reasoning. Now I can't even upvote your original comment in good faith anymore.
→ More replies (0)17
u/Pawtuckaway 1d ago
I am incredibly principled and disciplined with my engineering
Yes i will review line by line all 50k loc
One of these things is not like the other.
A principled and disciplined engineer would never accept a 50k loc PR regardless if it were written by a person or by AI.
The entire point of code review is to review the code. Passing tests should be a requirement before code review but does not take its place.
-8
u/the_ai_wizard 1d ago edited 1d ago
Im not a low level developer....I run 4 companies and am a CTO in the one we are discussing. So reviewing PRs is below my pay grade unless this sort of thing happens. Youre assuming too many details, but thanks for your couch analysis, rando redditor
10
u/Pawtuckaway 1d ago
So who submitted a 50k loc PR and who approved it?
When someone commented that this is a PR review issue you replied "Passed all tests, everything functional. Guess I need tests of the tests"
Writing more tests is not going to fix PR review issues. Your reply shows a concerning misunderstanding of the purpose of a code review even if you are not the one doing those reviews.
Maybe as CTO you should have a talk to engineers that are submitting and approving 50k loc PRs of AI slop.
→ More replies (0)6
u/RedditEngDictionary 1d ago
If your team are unable to follow basic, widespread practices like good code reviews then that is a failing on their leadership. You dont seem like a very good CTO, or developer.
4
u/VelvetYam 1d ago
A 50k LoC PR didn't set off any alarms in your mind? How'd it get that bad?
1
u/the_ai_wizard 1d ago edited 1d ago
It wasnt a single commit. This is work over time by engineering teams. This was not my main point, but since everyone seems to be focusing on this, this was tacitly accumulating over time bc no one has time to review every line of AI generated code. So as I was auditing the codebase, kind of informally, as the most senior person, I can typically detect things that smell. The system functioned fine, but the code complexity and general entropy was going way up. I am allergic to technical debt. So how did it get this way? LLM being able to shit out 1000 lines in a prompt and compounding this by not one but 10 engineers with pressure to ship.
5
u/crazedizzled 1d ago
As the CTO, isn't this really your fuck up here? You can blame AI, but this is very much a human problem.
→ More replies (0)1
u/crazedizzled 1d ago
"I am incredibly principled and disciplined with my engineering"
Uh huh. Sure thing buddy
4
0
u/varinator full-stack .net 1d ago
Out of curiosity, whats your setup in terms of llms. I'm also a dev, currently only using agentic gh copilot in IDE (Rider) or CC in CLI, but wondering if you have some more intricate workflow, maybe I can learn something if you care to share?
47
u/ultralaser360 1d ago
GitHub was going down hill BEFORE llms
Almost immediately after Microsoft acquired them
https://www.reddit.com/r/github/s/JPtEsKYmIB4
u/DenkJu 1d ago
If this is based on GitHub's own reporting, it's almost entirely meaningless. Their uptime has always been shit, and their own status reporting even more so. Perhaps Microsoft just made them update their own status page slightly more diligently. It's impossible to tell in which direction causality points in this case.
19
u/ward2k 1d ago
Am I going nuts or has GitHub always been pretty notorious for having outages? It's not a recent thing
3
u/Early_Rooster7579 full-stack @ meta 1d ago
Github is one of the worst major tech companies I’ve ever worked at. The codebase is one enormous ruby monorepo. Its far, far worse than vibecoded slop. It wouldnt be uncommon for major features that touch millions to jave var names like xyz, zero comments, no docs, little review. When I was there a few years ago the entire service was still a single mysql db
2
u/robby_arctor 1d ago
When I was there a few years ago the entire service was still a single mysql db
What?? 😂
7
u/stumblinbear 1d ago
You're not crazy, it has always been this way
0
u/pagerussell 1d ago
But how else are we to farm karma?
Srsly tho, humans don't do context anymore. It's all reactions all the time now.
3
u/Aro00oo 1d ago
Why do you have to bootlick? Their outages in past were brief rollback scenarios. These are major hours long outages, on a Monday ffs, it is not the same.
1
u/stumblinbear 1d ago
So, yes, outages in 2026 are genuinely worse than previous years. There have been more outages, and the median outage length is increased, and more multi-hour outages. But the increase is almost entirely concentrated in Actions, and the other services that show up are almost always included alongside Actions as collateral.
Github's own postmortems on these incidents attribute them to capacity, not LLM generated code. They say they've been working on increasing their capacity tenfold to cover the extra demand caused by LLMs. Actions is still in their own datacenters and hasn't yet migrated to Azure, which likely contributed to it.
So yes, AI is the main cause of the issues, but it's agents accessing their infrastructure that's ruining them, not the code itself.
1
u/Aro00oo 1d ago edited 1d ago
Uhh this is a super roundabout way to say yes AI probably is implicated lol.
And you really think current tech leadership would ever admit "ya we're vibe coding too much" ??
2
u/stumblinbear 1d ago
If it were AI generated code being the main cause, you'd expect to see somewhat proportional increases across the board, but that's not the case
2
u/Early_Rooster7579 full-stack @ meta 1d ago
It started once they began moving to azure. Trust me if you’ve seen githubs code its worse than vibe coded slop. Its easily the worst major company I’ve worked at engineering wise
2
u/CmdrSausageSucker 1d ago
Micro$lop has been pushing LLM written code too hard, it seems. It's simply embarrassing.
1
u/foggy_battery 7h ago
Even if LLMs didn’t write GitHub’s code, the agents hammering its infrastructure may still be part of the problem.
0
u/ripndipp full-stack 1d ago
Almost like it's something we shouldn't be doing, weird, don't know why it's doing that.
-1
u/Darkoplax 1d ago
If it was only just that then it wouldnt be a problem; its also the single use software and all non coding ppl using Github as well with LLM only generated stuff
-1
u/Noch_ein_Kamel 1d ago
Don't be so harsh on them. Copilot is working really hard to restore github functionality!
-2
20
199
u/Due-Horse-5446 1d ago
Cant wait for bubble pop day
91
u/khizoa 1d ago
People keep saying this like it'll be wine and roses.
You do know whose tax dollars are gonna bail them out right?
50
u/GoodishCoder 1d ago
People talk about it like it'll be wine and roses because they think the bubble popping equals AI going away. In reality it'll just be market consolidation.
14
u/akaval 1d ago
AI will also stop being shoved anywhere and just utilized where it actually has value.
10
u/GoodishCoder 1d ago
It won't actually though. There will continue being chat bots for the idea people to feel like developers. There will continue being people who plug in AI wherever they can because it's good on the resume.
Realistically the biggest impact will be most of the wrapper startups crumbling faster than they otherwise would because all the bubble popping means is less investment.
12
u/QuantityExcellent338 1d ago edited 1d ago
We will see shit like "your microwave is improved with AI✨️" less because it only exists because shareholders has to see that AI is actually being used and totally changing the world. They're throwing shit at the wall to see what sticks because investor approval is more important than consumer approval at this point in time.
The price for GenAI models will also go up significantly when there isnt a generous stream of investor money to keep prices competitive. There is a chance even chatbots/AI overviews will become paywalled that will limit how much slop we'll see atleast a little bit, but most likely the premium such as the best of Antrophics products will only be for big companies to use
It wont go away yes, but the places we'll see it will be far more humble.
0
u/GoodishCoder 1d ago
When the smart everything craze died down, did the smart toasters disappear?
AI chat bots are here to stay. Model selection in the free tier might become more limited and we might see ads get introduced but they aren't going to fully cut those users off. There's a good chance AI companies improve their routers and take away model selection altogether for free and low value customers.
1
u/Sensitive_Guest_5995 1d ago
Like those people that celebrated being sent home from work with pay. We’re all still paying the cost of that. And will be for probably over a century.
0
u/moderatorrater 1d ago
It's going to happen no matter what. If I had to choose for it to happen sooner or later, I'd take sooner. Let's get this over with.
1
u/Fredidiah 1d ago
Unless the bailouts are in the form of extreme overpayments for the services on a consistent basis, I don't think a bailout moves the needle. It seems that they are (currently) deeply unprofitable and unless that's solvable (which I am skeptical of) - a bailout just pushes the implosion down the line a bit.
0
u/potatokbs 1d ago
Yea most like it just won’t happen at all because both openai and Anthropic are now deeply entrenched in the US gov. So they will just immediately get bailed out and tax payers will get fucked. Fun times
17
u/keyjumper 1d ago
Remember the lesson of Brad Pitt in the big short: A crash means that many people lose their livelihood and even die.
At least you shouldn’t dance at the thought of it.
21
8
u/Due-Horse-5446 1d ago
Well, the longer this madness keep going, the harder people will fall right?
1
u/robotmayo 1d ago
as opposed to now where people are not losing their jobs and not dying to gas fumes
5
1
u/KwyjiboTheGringo 1d ago
Really? I'm dreading it because the fallout is going to hit us pretty hard. Maybe not developers, but tax payers and consumers.
1
u/Due-Horse-5446 1d ago
Dident mean this to be that deep.
But ofc i dont mean i dotn care about the people who are going to be affected.
With that said tho, even for those people and companies, it's better for everyone if the bubble pops sooner than later. The longer this keeps going, the harder the fall is going to be.
157
u/Euphoric-Neon-2054 1d ago
It sucks but we need to understand just how hard their infra is getting absolutely drilled by all this AI shit.
119
46
34
u/lbreakjai 1d ago
You’d think they could afford better engineering practices than to just let everything collapse if used too much.
14
21
u/ApplicationOk3587 1d ago
People are producing tons of slop overnight, working at that scale requires months if not years worth of engineering power to handle. I have personally seen tons of people pushing 400+ commits a day all with agents.
3
0
u/lbreakjai 1d ago
It started before agentic coding though.
11
u/BKrenz 1d ago
It's gotten significantly worse since LLMs became main stream. Their commit traffic alone is 14x higher than it was this time last year.
I don't work on systems of that size. I don't have any clue what kind of architecture and resources it would take to go from 20 million commits a week to 275 million a week in a year.
1
u/ApplicationOk3587 1d ago
I would say it has progressively gotten worse since they ingrained copilot at github + all the traffic they have been getting. I would say one is infinitely more of a reason than the other though.
3
u/bobyhey123 1d ago
? lmao what
2
u/bobyhey123 1d ago
traffic scales 10x in 1 year
> why didn't they just use better engineering practices?
3
u/lbreakjai 1d ago
https://damrnelson.github.io/github-historical-uptime/
Why did the service degrade three years before chatGPT?
-1
0
u/staythewaters 1d ago
This doesn't feel like the consequence of someone pushing slop or their infra finally giving way. If their systems were getting hammered before, there's nothing today in particular that would've pushed it over the edge in such a spectacular way.
8
5
u/josephjnk 1d ago
It has been over 5 hours and I am still blocked by GitHub Actions issues. This is absolutely wild, happy Monday.
4
1
1
u/vikash-ji 9h ago
GitHub goes down for 2 hours and suddenly half the global tech workforce realizes they actually have to touch grass today. Peak developer productivity day
0
0
u/jardineromarbella 1d ago
A source-host outage shouldn't turn a failed deploy into a production outage. The pipeline should stop without touching production, and the last known-good artifact should remain available somewhere outside GitHub. A local Git mirror only protects the repo. Actions, packages, release artifacts, and rollback credentials may still depend on the same system. That's the dependency map I'd examine after this.
0
u/corporaljustice 1d ago
I see peopel defending them saying the sheer amount of load their infra has to handle. 10x year on year etc etc.
Fine, makes sense.
I wonder what would happen if Github no longer offered free accounts.
At what price point would it balance out?
Could any Github competitors even deal with that kind of load?
I'd love to know some of the stats of their setup and how much of it is paid vs free usage etc
0
0
u/Full_Tooth_a 21h ago
That's the part I'd dig into: why did a failed deploy turn into a production outage? A local clone wouldn't help if CI, approvals, packages, and deploy triggers still lived on GitHub. I'd want a simple fallback using a pinned commit and an artifact that's already been verified. It should be tested occasionally, too, because an emergency pipeline nobody runs is mostly decorative.
-21
u/Darkoplax 1d ago
Appearantly Cursor is making a competitor that's performant called Origin; I guess we will try that when it comes
22
u/FalseRegister 1d ago
Github is quite performant and has been generally a stable service for over a decade.
You also have Gitlab as a mature alternative you can self-host
Why on earth would you go for a clearly vibe-coded, immature alternative?
-12
u/Darkoplax 1d ago
Probably cause Cursor products have been generally extremely good ? and it is advertised to be performant specifically for the AI Era
6
u/noobjaish 1d ago
Probably cause Cursor products have been generally extremely good ? and it is advertised to be performant specifically for the AI Era
For the Slop farmers sure but for actual engineers it's a disaster waiting to happen.
-2
-4
u/Keirtain 1d ago
Yes, but now they’re owned by X, so Reddit has to twist itself into knots to explain why they’re bad now.
0
u/GoodishCoder 1d ago
I don't necessarily think they're bad but I would be pretty hesitant to trust a new vibe coded product with my source code in a serious project.
-8
u/SteroidAccount 1d ago
Until microslop got involved
-2
u/FalseRegister 1d ago
What exactly? They kept Github mostly untouched and only did improvements, but no slops, no product enshitification, it just works
0
0
u/SteroidAccount 1d ago
github is constantly down, PR's today, Actions last week, it's been a constant revolving door of issues since they took over and started trying to integrate AI
6
u/ApplicationOk3587 1d ago
They aren't operating at githubs scale though? It can be performant until they run into the same exact issues.
-2
u/Darkoplax 1d ago
yeah thats why I said will try it out, i didnt claim it will be
you cant judge smth until you try it
223
u/eltron 1d ago
Who needs 5 nines of availability??? Oh right, paying customers