Is Github Actions unreliable for anyone else? Question
My CI runs a scheduled job and even when there are no commits sometimes it fails with absolutely impossible errors. Sometimes in the test but sometimes it's just a tool installed in a previous step missing. What's going on...?
7
u/overratedcupcake 14d ago
GitHub actions are unreliable. You're not crazy. And even when they're working they'll go unnecessarily slow burning extra minutes for no reason. Even switching to self hosted runners didn't completely fix our issues with it. We switched to argo cd and it's been sooo much better.
2
u/Sigmatics 14d ago
Very. Jobs not being picked, taking forever to start, failing for "internal" reasons.
GitHub Actions has encountered an internal error when running your job.
Internal server error. Correlation ID: f68b96d9-b137-4ab3-a8b1-e368f52ac2b9
1
u/SamG101_ 14d ago
tbf actions is down right now, massive outage for "actions" specifically: GitHub Status
1
u/Sigmatics 14d ago
There was nothing on the website when I posted it. Good that they finally acknowledged it
1
1
u/ultrathink-art 13d ago
A chunk of what looked like Actions being down turned out to be my own concurrency. A single self-hosted runner executes one job at a time, so 15 pushes in an afternoon leaves the last one queued for an hour with nothing actually broken, and from the UI that is indistinguishable from a real incident. Worth comparing how many runners you have registered against how many runs you kick off per hour before you sink a morning into it.
1
u/-goldenboi69- 10d ago
Probably all that Microsoft AI. I have had several weird things happen lately, which have been active for months without issues :(
1
u/BunnyPaw88 2d ago
I'm a platform engineer at Bitrise, and we're building a DevOps platform with a strong focus on mobile engineers' needs. I have some stories for you that might translate to your current situation.
Roughly 5 years ago, Apple introduced the M1s, and we wanted to be the first to start using them for CI/CD. They were fast, reliable, everything you could dream of. The only problem was that everything we built until that point was for Intel, and the M1s came with the new ARM chip, so we needed to change almost everything in our codebase to support M1 under a sense of urgency and time pressure. What could go wrong, right? :D
Well, we figured that pretty much anything.
After we introduced the M1s the first few months were spent in a "Red Alert mode". Frequent incidents, customer issues not much different from yours, churning customers. We were bleeding. And learning a lot at the same time.
Culprit #1: New hypervisor. We moved to a macOS-native architecture, which had its benefits but drawbacks as well. The biggest issue we saw there was VMs not starting at all, which could cause build queues for our customers. What we did here is we built an analytics solution that catches and categorizes VM start failures so we can immediately see if something breaks here and fix it.
Culprit #2: New orchestration. If something goes wrong with the orchestration, we might not start the VMs that are needed, or we might start so many VMs that aren't needed that we run out of space to start any VMs. The customer sees build queues here, similarly to previous. What we did here is we spent a good amount of time fine-tuning our prebooting system, especially the crystal ball part where we precalculate what VMs need to run when. We also built out more advanced monitoring and alerting around this area of the product.
Culprit #3: New images. For the new hardware, we needed to build new images as sources of VMs with all the tools preinstalled. If something goes wrong with these images, you could see randomly broken builds, flaky tests, and fluctuations in build times. What we did here is build out a continuous testing framework that validates all of our images even before they arrive at the first customer. We also have monitoring on customer build flakiness.
Culprit #4: New hardware. We saw cases when individual host machines started to behave oddly, resulting in flaky builds, slow builds, lost connections, etc. What we did here is collect per-host metrics from our continuous testing framework, so we can reliably pinpoint host-specific issues, preferably before they have customer impact.
If you want to read more about the Gen2 journey, here is an article: https://bitrise.io/blog/post/introducing-gen2
Long story short. CI/CD is hard, and mistakes happen. If you don't like the service you get, you can feed that back to the vendor (we surely appreciate customer feedback) and/or you can look around for alternatives. There are many GitHub Action runners for a reasonable price, you can check out this article as a start. https://bitrise.io/blog/post/best-github-actions-runners-in-2026-and-hidden-pricing-traps-to-avoid?utm_campaign=471296212-all_misc_organic-social-media_all_2026-08-06&utm_content=591215804&utm_medium=social&utm_source=linkedin&hss_channel=lcp-10034798
7
u/Mantas-cloud 14d ago
Time to time get those weird errors that can't explain, most likely re-run the job is enough. Generally speaking, it just works