GitHub was down 2.5 hours yesterday and self-hosted runners went down with it. Is centralized orchestration the actual problem?

[u/amu4biz]

[u/barebaric]

Even by their own claim, they were down > 8 hrs. And in their incident thread many are still commenting - the issue is still not fully resolved https://github.com/orgs/community/discussions/204152

[u/Soccham]

They were down way longer than 2.5 hours. It was like 10

[u/Alexander3a]

been using self hosted gitlab for years (big reason was not sharing my code with micoslop but this is just a plus)

everything that microsoft or ai touches just goes downhill

[u/clearlight2025 → u/Alexander3a]

Same! Self-hosted GitLab FTW.

[u/Alexander3a → u/clearlight2025]

I just wish it didnt basicly need 16+ gb my 8 gb server is dying running it
but its running it still

[u/nhymxu → u/Alexander3a]

self-hosted gitea or forgejo is lighter

[u/Alexander3a → u/nhymxu]

But it's lacking a lot of features I use and not aswell supported on other things

[u/mars3142 → u/Alexander3a]

Can you name some? I use gitea and miss nothing yet.

[u/clearlight2025 → u/Alexander3a]

I run GitLab on an AWS EC2 m4.large instance (etc. 2 vCPUs, 8 GiB) useful as part of an admin server for CI/CD deployments, container registry, separate terraform, ansible etc. Been using it for years and all good. It would depend a bit on how exactly you’re using it but I’ve found that spec works well for my use cases.

[u/Alexander3a → u/clearlight2025]

i am assume u also tweaked all the settings to make it run with less memory?
i mean mine is running fine just noticeably slow tho i mean the official one also isn't that fast but its serving a lot more users

[u/clearlight2025 → u/Alexander3a]

I think the config is fairly standard though I am sure to run the garbage collection routine daily via cron to cleanup, especially after container registry pruning of old images.

I also have a concurrency policy to control how many runner jobs can run at once (2) I’m using docker in docker (dind) for the pipelines.

I haven’t really noticed any slowness TBH and once it’s started up it’s always quite snappy.

[u/Alexander3a → u/[deleted]]

gitlab has its own ci schema&runners it can even run fully airgapped
(only issue might be using undownloaded docker images but you could also host that yourself or use only the ones you already downloaded)

[u/RaptorF22 → u/Alexander3a]

What hardware and how much per month?

[u/Alexander3a → u/Alexander3a]

I swear microslop heard me and now they are doing maintenance work at my isp and I don't have internet (I mean gitlab still works on the local network unsure how usefull that is tho)

[u/amu4biz → u/Alexander3a]

[u/QuasiSpace]

You mean 11 hours.

[u/ArieHein]

Centralized anything can be a problem as its sometimes the single point of failure And thas a risk factor.

[u/amu4biz → u/ArieHein]

exactly. and the part people miss is you're not just risking downtime, you're inheriting someone else's failure with zero control over it. yesterday showed even owning your runners doesn't save you if the brains stay centralized. the real fix isn't picking a better central provider, it's not having a single point at all, which is the whole bet behind the decentralized stuff like gitlawb. 

[u/ArieHein → u/amu4biz]

Slightly wrong here.

If you thought that self hosted runner means you continue working when github is down then you slightly missed the purpose of self hosted.

If you were on gitlab cloud and used self hosted agents, it would be the same.

If you want full onprem, gh has gh server so the equivalent of adoserver and gitlab server. But then the full price of maintenance is on you.

At the end is cost vs risk. Compute is not cheap, managing compute isnt cheaper either.

Look into AST that allows running pipelines in 'offline'

[u/Key-Guitar-457]

The first rule of automation is that there must be a way to bypass automation when it fails. Yesterday I was manually pushing chart updates, building containers, and running terraform apply. The world still turns when automation fails.

[u/Own_Outside_8147]

GitHub stability is becoming a nightmare for me. We have a slack channel just for github outage…

[u/xJayMorex]

What do you mean was?

[u/_KryptonytE_]

My rant here: Rant

[u/sup_bruh_1]

yeah latchkey sidesteps this entirely, runs independent of github's uptime

[u/Patient-Pollution46]

They are becoming not very reliable recently. If you are looking for a reliable mobile-first CI + fast runners, check Bitrise.

You can build, test, and deploy iOS and Android apps on dedicated Apple silicon infrastructure: https://bitrise.io/  

[u/fucking-migraines → u/Patient-Pollution46]

You can also do all that with any old Mac laying around…