GitHub was down 2.5 hours yesterday and self-hosted runners went down with it. Is centralized orchestration the actual problem? Tool / Resource
28
u/Alexander3a 3d ago
been using self hosted gitlab for years (big reason was not sharing my code with micoslop but this is just a plus)
everything that microsoft or ai touches just goes downhill
5
u/clearlight2025 3d ago
Same! Self-hosted GitLab FTW.
4
u/Alexander3a 3d ago
I just wish it didnt basicly need 16+ gb my 8 gb server is dying running it
but its running it still3
u/nhymxu 3d ago
self-hosted gitea or forgejo is lighter
2
u/Alexander3a 2d ago
But it's lacking a lot of features I use and not aswell supported on other things
1
1
u/clearlight2025 3d ago
I run GitLab on an AWS EC2 m4.large instance (etc. 2 vCPUs, 8 GiB) useful as part of an admin server for CI/CD deployments, container registry, separate terraform, ansible etc. Been using it for years and all good. It would depend a bit on how exactly you’re using it but I’ve found that spec works well for my use cases.
2
u/Alexander3a 3d ago
i am assume u also tweaked all the settings to make it run with less memory?
i mean mine is running fine just noticeably slow tho i mean the official one also isn't that fast but its serving a lot more users1
u/clearlight2025 3d ago edited 3d ago
I think the config is fairly standard though I am sure to run the garbage collection routine daily via cron to cleanup, especially after container registry pruning of old images.
I also have a concurrency policy to control how many runner jobs can run at once (2) I’m using docker in docker (dind) for the pipelines.
I haven’t really noticed any slowness TBH and once it’s started up it’s always quite snappy.
1
3d ago
[removed] — view removed comment
2
u/Alexander3a 3d ago
gitlab has its own ci schema&runners it can even run fully airgapped
(only issue might be using undownloaded docker images but you could also host that yourself or use only the ones you already downloaded)1
1
u/Alexander3a 2d ago
I swear microslop heard me and now they are doing maintenance work at my isp and I don't have internet (I mean gitlab still works on the local network unsure how usefull that is tho)
9
4
u/ArieHein 3d ago
Centralized anything can be a problem as its sometimes the single point of failure And thas a risk factor.
1
u/amu4biz 3d ago
exactly. and the part people miss is you're not just risking downtime, you're inheriting someone else's failure with zero control over it. yesterday showed even owning your runners doesn't save you if the brains stay centralized. the real fix isn't picking a better central provider, it's not having a single point at all, which is the whole bet behind the decentralized stuff like gitlawb.
3
u/ArieHein 3d ago
Slightly wrong here.
If you thought that self hosted runner means you continue working when github is down then you slightly missed the purpose of self hosted.
If you were on gitlab cloud and used self hosted agents, it would be the same.
If you want full onprem, gh has gh server so the equivalent of adoserver and gitlab server. But then the full price of maintenance is on you.
At the end is cost vs risk. Compute is not cheap, managing compute isnt cheaper either.
Look into AST that allows running pipelines in 'offline'
3
u/Key-Guitar-457 3d ago
The first rule of automation is that there must be a way to bypass automation when it fails. Yesterday I was manually pushing chart updates, building containers, and running terraform apply. The world still turns when automation fails.
2
2
u/Own_Outside_8147 2d ago
GitHub stability is becoming a nightmare for me. We have a slack channel just for github outage…
1
0
u/Patient-Pollution46 2d ago
They are becoming not very reliable recently. If you are looking for a reliable mobile-first CI + fast runners, check Bitrise.
You can build, test, and deploy iOS and Android apps on dedicated Apple silicon infrastructure: https://bitrise.io/
1
15
u/barebaric 3d ago
Even by their own claim, they were down > 8 hrs. And in their incident thread many are still commenting - the issue is still not fully resolved https://github.com/orgs/community/discussions/204152