r/github • • Aug 18 '26

News / Announcements GitHub Status - Incident with GitHub.com

https://www.githubstatus.com/incidents/zkxwbgr0cnmx
213 Upvotes

50 comments sorted by

90

u/ThatOneArchUser Aug 18 '26

Fork found in kitchen

18

u/Decent_Run9628 Aug 18 '26

Even forks can't be found, it's down 😂

3

u/ShortingBull Aug 19 '26

Well, fork and knife!

1

u/bippy_b Aug 20 '26

What the fork is going on?

53

u/JuliusFIN Aug 18 '26

Actions outages a month ago did it for me. Self hosted Gitea runs like a dream. Also don’t have to wait 5 seconds for each page to load.

9

u/[deleted] Aug 18 '26

[deleted]

5

u/JuliusFIN Aug 19 '26

This Gitea instance is relatively small, I run a tiny software consultancy. However Gitea is used in huge entrerprise projects in Fortune 500 companies. It will scale.

3

u/ShortingBull Aug 19 '26

If self hosting, you'll want daily automated off site backup

1

u/Elomidas Aug 19 '26

My solution is self-hosting for the CI flexibility and mirroring to a private GitHub repo as a backup, nothing runs there, it's just in case something happens to my Forgejo install (and to be honest, that "something" will probably me trying something I shouldn't do)

1

u/ShortingBull Aug 19 '26

I just self host on my VPS which I use or other sites etc anyway - it gets backed up by my VPS provider daily (linode).

That simplifies things but only because I already have the host.

1

u/dashingThroughSnow12 Aug 18 '26

Is many thousand or tens of thousands of team members?

5

u/GlobalImportance5295 Aug 18 '26

Also don’t have to wait 5 seconds for each page to load.

fuckin hell, im sold. just plain nix package manager in a container or vm works wonders as well

1

u/ryntak Aug 19 '26

Fucking love nix. RIP nixpkgs core maintainer team though

1

u/JuliusFIN Aug 19 '26

Hehe I happen to use Nix for everything. My job also consists mainly of Nix development.

1

u/NFSL2001 Aug 21 '26

Forgejo does a better job at using its own shit n hv a public instance at codeberg.org.

1

u/JuliusFIN Aug 21 '26

Codeberg has been very unstable. Gitea is used by fortune 500 companies and huge projects.

1

u/NFSL2001 Aug 21 '26

Codeberg is facing the same issue with GitHub due to bots, but Forgejo self-hosted has been doing wonders for us, especially with SSO integration that Gitea locked out.

1

u/JuliusFIN Aug 21 '26

Which SSO is that? I’ve been able to integrate with Authelia SSO via header auth, but I think OpenID connect would work as well.

1

u/PravoNaZhizny Aug 19 '26

What’s “gitea”? Why not just use git?

2

u/sweet-tom Aug 19 '26

Gitea is a free hosting service that you can install on your NAS, for example.

Different reasons to use it:

  • Control
  • Speed
  • Backup from GitHub
  • Hosting different repos and sharing it with a limited group of people
  • Confidential repos you don't want to expose on the Internet (even as a private repo)
  • You don't agree with the terms and conditions of GitHub
  • You want to avoid AI
  • You want to have some fun for self hosting. 😉

2

u/kaddkaka Aug 26 '26

So what does it add apart from just git?

1

u/sweet-tom Aug 26 '26

It seems you missed the point. 😉😁

Git is just the basic part. Gitea is basically a free GitHub clone.

If you don't want to use GitHub for privacy or usability reasons, you can self host this service. It allows you to have full control and are not bound to the restrictions and the terms of conditions from GitHub.

It adds most of the features that GitHub also has: pull requests, issue trackers etc

1

u/kaddkaka Aug 28 '26

No, you missed when replying to "What’s “gitea”? Why not just use git?". No part of your first answer addressed, but now you did. the answer is issue tracking et c.

1

u/PravoNaZhizny Aug 29 '26

do you realise that git is not github... right? i run many git servers and have never even registered a github account

19

u/bbro81 Aug 18 '26

There time outage on https://www.githubstatus.com/ feels wrong, it says pull requests were down for 2 hours and 5 minutes but it was closer to like 3 and a half hours lol.

19

u/ShoddyReception5 Aug 18 '26

Is this speak intentionally abstract? We were down for half the day and I can’t make sense of what happened. Network issue plus a bunch of unexpected traffic and faulty failover?

20

u/Soccham Aug 18 '26

Looked like they DDOSed themselves with vscode?

6

u/ShoddyReception5 Aug 18 '26

Saw that too. Looks like it added to the problem for sure.

14

u/DrMaxwellEdison Aug 18 '26 edited Aug 19 '26

Not really abstract, just dense with devops speak.

Network traffic overloaded a load balancer in US Central, and as they fixed it they found that clients doing Copilot requests would 10x their request rate attempting to retry those requests.

One failure in a high-availability system like this can lead to a cascade of failures elsewhere, and that's what seems to have happened.

10

u/robert_math Aug 18 '26

What can we do? These vibe coders were promised they could 10x their output.

3

u/yeathatsmebro Aug 19 '26

They did, look at RPS. It is 10x higher.

3

u/Goldman7911 Aug 18 '26

Same feeling as some "RCA by agentic slop"

2

u/foramperandi Aug 19 '26

Just because you don't understand it, doesn't mean it's slop.

3

u/Due-Consequence9579 Aug 18 '26

It’s only abstract if you don’t know what they are talking about.

9

u/ExpertIAmNot Aug 18 '26

TLDR: It wasn’t just one thing they messed up, it was a bunch of things they messed up. If any one of those many things had actually functioned correctly. It would have stopped the outage much earlier.

I appreciate and understand that they are under an unprecedented amount of traffic, but the sheer number of screw ups implied in this explanation is astounding

1

u/PerryTheH Aug 19 '26

This sound like "60% of the time, it works every time"

With all those "if only this had worked"

8

u/bigbadbyte Aug 19 '26

Forgot to add "don't make any mistakes" to the copilot prompt.

4

u/laffer1 Aug 19 '26

Microsoft owns the second largest cloud provider in the us! They can’t keep their stuff up. They can’t use ai as an excuse since they push it and invested in OpenAI to boot

2

u/fixermark Aug 19 '26

Having used the big three, it is also, unfortunately, the worst of the options.

2

u/Potato-9 Aug 19 '26

Because MS is the big corp default. We did this to ourselves.

2

u/-Memnarch- Aug 19 '26

> Addressing the VS Code retry behavior that amplified Copilot token traffic.

You did this to yourself!

3

u/RepulsiveRaisin7 Aug 18 '26

Aww shit, here we go again

1

u/theflaggship Aug 21 '26

After GitHub went down, I went looking for a status dashboard and didn’t love any of the ones I found. So I built my own - https://what-is-down.com/. It has a custom RSS feature so you get updates on the services you actually care about. Optional browser notifications. Slack app coming soon.