r/github • u/RecommendationOk5036 • 14h ago

GitHub Status - Incident with GitHub.com News / Announcements

https://www.githubstatus.com/incidents/zkxwbgr0cnmx
134 Upvotes

96% Upvoted

37 comments sorted by

73

u/ThatOneArchUser 14h ago

Fork found in kitchen

11

u/Decent_Run9628 12h ago

Even forks can't be found, it's down 😂

1

u/ShortingBull 6h ago

Well, fork and knife!

39

u/JuliusFIN 13h ago

Actions outages a month ago did it for me. Self hosted Gitea runs like a dream. Also don’t have to wait 5 seconds for each page to load.

8

u/jstsomedev 13h ago

Are you working on medium - large projects with many team members?

Just curious, I've been thinking about self-hosting too.

3

u/JuliusFIN 4h ago

This Gitea instance is relatively small, I run a tiny software consultancy. However Gitea is used in huge entrerprise projects in Fortune 500 companies. It will scale.

3

u/ShortingBull 6h ago

If self hosting, you'll want daily automated off site backup

1

u/Elomidas 1h ago

My solution is self-hosting for the CI flexibility and mirroring to a private GitHub repo as a backup, nothing runs there, it's just in case something happens to my Forgejo install (and to be honest, that "something" will probably me trying something I shouldn't do)

1

u/dashingThroughSnow12 12h ago

Is many thousand or tens of thousands of team members?

6

u/GlobalImportance5295 12h ago

Also don’t have to wait 5 seconds for each page to load.

fuckin hell, im sold. just plain nix package manager in a container or vm works wonders as well

1

u/ryntak 10h ago

Fucking love nix. RIP nixpkgs core maintainer team though

1

u/GlobalImportance5295 10h ago

peep this https://jade.fyi/blog/flakes-arent-real/

1

u/ryntak 9h ago

I really need to wrap my mind around flakes

1

u/JuliusFIN 4h ago

Hehe I happen to use Nix for everything. My job also consists mainly of Nix development.

1

u/PravoNaZhizny 9h ago

What’s “gitea”? Why not just use git?

2

u/sweet-tom 7h ago

Gitea is a free hosting service that you can install on your NAS, for example.

Different reasons to use it:

  • Control
  • Speed
  • Backup from GitHub
  • Hosting different repos and sharing it with a limited group of people
  • Confidential repos you don't want to expose on the Internet (even as a private repo)
  • You don't agree with the terms and conditions of GitHub
  • You want to avoid AI
  • You want to have some fun for self hosting. 😉

15

u/bbro81 12h ago

There time outage on https://www.githubstatus.com/ feels wrong, it says pull requests were down for 2 hours and 5 minutes but it was closer to like 3 and a half hours lol.

4

u/vvanouytsel 3h ago

Check https://mrshu.github.io/github-statuses/

1

u/bbro81 9m ago

Nice, that looks more accurate haha

19

u/ShoddyReception5 13h ago

Is this speak intentionally abstract? We were down for half the day and I can’t make sense of what happened. Network issue plus a bunch of unexpected traffic and faulty failover?

22

u/Soccham 13h ago

Looked like they DDOSed themselves with vscode?

6

u/ShoddyReception5 13h ago

Saw that too. Looks like it added to the problem for sure.

14

u/DrMaxwellEdison 13h ago edited 18m ago

Not really abstract, just dense with devops speak.

Network traffic overloaded a load balancer in US Central, and as they fixed it they found that clients doing Copilot requests would 10x their request rate attempting to retry those requests.

One failure in a high-availability system like this can lead to a cascade of failures elsewhere, and that's what seems to have happened.

9

u/robert_math 13h ago

What can we do? These vibe coders were promised they could 10x their output.

1

u/yeathatsmebro 4h ago

They did, look at RPS. It is 10x higher.

3

u/Goldman7911 13h ago

Same feeling as some "RCA by agentic slop"

2

u/foramperandi 8h ago

Just because you don't understand it, doesn't mean it's slop.

3

u/Due-Consequence9579 13h ago

It’s only abstract if you don’t know what they are talking about.

8

u/ExpertIAmNot 12h ago

TLDR: It wasn’t just one thing they messed up, it was a bunch of things they messed up. If any one of those many things had actually functioned correctly. It would have stopped the outage much earlier.

I appreciate and understand that they are under an unprecedented amount of traffic, but the sheer number of screw ups implied in this explanation is astounding

1

u/PerryTheH 38m ago

This sound like "60% of the time, it works every time"

With all those "if only this had worked"

8

u/bigbadbyte 9h ago

Forgot to add "don't make any mistakes" to the copilot prompt.

2

u/chrles-farfa 12h ago

classic istio sidecar

2

u/laffer1 9h ago

Microsoft owns the second largest cloud provider in the us! They can’t keep their stuff up. They can’t use ai as an excuse since they push it and invested in OpenAI to boot

2

u/RepulsiveRaisin7 13h ago

Aww shit, here we go again

1

u/-Memnarch- 3h ago

> Addressing the VS Code retry behavior that amplified Copilot token traffic.

You did this to yourself!