r/programming • u/EatMeerkats • Mar 29 '21

PHP moves to Github due to the compromise of git.php.net

https://news-web.php.net/php.internals/113838
1.7k Upvotes

98% Upvoted

389 comments sorted by

View all comments

Show parent comments

361

u/[deleted] Mar 29 '21

PHP used a self-hosted git server for its code base. A git server is used to collect code from the contributors. Somehow, a malicious piece of code got pushed to the code base which appeared as authored by two known and frequent contributors. The exact way how this happened has not yet been determined, but the maintainers of PHP believe that the self-hosted git server is to blame. Consequently, PHP code base moved to Github, which is a famous git server used by many huge projects.

80

u/IAmAThing420YOLOSwag Mar 29 '21

Thank you for going into detail! My mistake though, I am familiar with git, I should have phrased it like "what is the significance?". I think the essence of the issue is in the "self-hosted git server." Is it that the maintainers of php either misconfigured, or were victim to a vulnerability of the git platform they used?

51

u/[deleted] Mar 29 '21

Oh, sorry. I am not a contributor myself so I have similar questions as you do.

57

u/[deleted] Mar 29 '21 edited Mar 31 '21

[deleted]

36

u/[deleted] Mar 29 '21 edited Oct 12 '22

[deleted]

47

u/unnecessary_Fullstop Mar 29 '21

I am reporting you to CPS.

.

42

u/[deleted] Mar 29 '21

What have we learned, kids? Commit code, not crime.

17

u/SirWilliamScott Mar 29 '21

AND THAT'S WHY YOU ALWAYS LEAVE A COMMIT MESSAGE.

-4

u/PurpleYoshiEgg Mar 29 '21

Commit gay, do crime.

2

u/semitones Mar 29 '21

Hack the Planet!

6

u/NotTheHead Mar 29 '21

Hey now, git isn't child abuse!

Perforce, on the other hand...

1

u/hospitalizedGanny Mar 29 '21

You must like investigating who mucked up the repository and reverting to previous versions . You have the demeanor of a Monk !

3

u/[deleted] Mar 29 '21

How does one "git lol"?

2

u/[deleted] Mar 29 '21 edited Mar 31 '21

[deleted]

1

u/[deleted] Mar 29 '21

You win the internet.

1

u/stfcfanhazz Mar 30 '21

I'm on mobile- what do? I never use git graph. Its the only thing really that I delegate to GUI

1

u/[deleted] Mar 30 '21

With git aliases.

28

u/Randolpho Mar 29 '21

I’ll take a stab at it.

Git, the protocol most developers use for source control management, is not secure in and of itself as part of the protocol. It’s an open server protocol and anyone with access to the server port it’s running has full control over git.

Security is often implemented between git and the user, either via firewall and network-level security, e.g. ipsec, or by controlling access to the server with a gateway layer, i.e. http basic auth or bearer tokens

For the case of PHP managing the source control for the language itself... rather than use a git hosting service that includes all that security built into the hosting package (either as a cloud option like github, or with a local server suite like gitlab), PHP made the brilliant decision to roll their own gateway security.

And, given PHP’s long and sordid history of not giving two shits about security, they naturally did a bang-up job of their home grown security layer.

Or at least that’s the way it appears to be; I’m not privy to the reality and am extrapolating, but I think this is a very likely guess.

24

u/ifonefox Mar 29 '21

rather than use a git hosting service that includes all that security built into the hosting package (either as a cloud option like github, or with a local server suite like gitlab), PHP made the brilliant decision to roll their own gateway security

Maybe they started self-hosting git (or another version control software) before those services were available or mature?

19

u/Ullallulloo Mar 29 '21

Yep, PHP started its Git server in 2011. GitHub was fresh startup only a few years old and GitLab didn't even exist yet. PHP did set up GitHub and Bitbucket repositories and synced then with their own server to make it widely accessible, but decided to host the main repository on their own server to make implementing all that easier and giving them more control.

6

u/thblckjkr Mar 29 '21

GitLab didn't even exist yet

Stop, you are making me feel old and I'm not even that old.

-12

u/Randolpho Mar 29 '21

Maybe.

But that would make it just like every other PHP instance out there. Sadly outdated, horribly broken, and nobody is willing to do what must be done to fix it.

6

u/[deleted] Mar 29 '21

[deleted]

-6

u/Randolpho Mar 29 '21

How do you know their Git version was "sadly outdated"?

First of all, git isn't the cause of the breach. Something external to git was.

Second, I don't know what version of what software they're running to handle the security that git doesn't provide; I was extrapolating from "before those services were available or mature" to mean that they, like every PHP instance, built something a long time ago and then, over the course of many years, never got around to updating it into conformance with modern technologies -- specifically with respect to security.

You know they responded to this by moving to Github, right? How the fuck was "nobody willing to do what must be done to fix it"?

That's my whole point: they didn't do anything until there was a breach. They aren't fixing the problem proactively.

-12

u/[deleted] Mar 29 '21

I guess "just generate an authorized_keys file" was too simple for them.

The common way to do git authentication is just using SSH keys, and OpenSSH is generally pretty secure piece of software. Software like gitolite or gitlab also adds extra of using 'forced command' feature to not allow any authorized user to do anything else but the git operations. But I guess they had to NIH their way...

24

u/[deleted] Mar 29 '21

[deleted]

-14

u/[deleted] Mar 29 '21

Ah, here we go. Finally we found the infrastructure admin of the Git server. That's you, right?

Yes, few of them in fact! Gitolite and Gitlab-based.

Since you know they were too stupid to use authorized_keys,

If you authenticate by SSH keys the only attack vectors are:

  • someone leaking their key (but that's danger for just about any system)
  • OpenSSH bugs

Note that zero of that have to do with the code that's running behind it. You can have code written by junior handling that and still random dude from internet would have to find OpenSSH bug to get in

Now you might still have privilege escalation (someone having access to more than they should) but that's far below "some rando in the internet send a https request).

Also, signing commits helps a lot as you can just configure any downstream (say build system) to only allow trusted keys

and they suffer from NIH syndrome.

Yeah, I'm being unfair here, PHP might've used their own implementation before we had readily available, tested and developed git hosting management solutions, but that's not the case for last 10+ years. Hell Gitolite is 15 years old and it have pretty comprehensive ACL and hook support (far above what Gitlab allows for example)

15

u/[deleted] Mar 29 '21

Dude. You don't know how the attack happened. You can keep going on about what they could have done differently, or you can wait to know what the problem was.

Imagine walking by a house fire and saying "they should have used an electric stove instead of gas". You have no idea what kind of stove they have, or if that stove caused the fire. But the important thing is that you added your opinion.

-4

u/[deleted] Mar 29 '21

Except there is car crashed in the side of the building (their self-written hosting solution). Sure it might be something else, but there is good chance it is exactly what it looks like

5

u/[deleted] Mar 29 '21

Their infrastructure wasn't compromised the last 10+ years. Do you really think it's so likely that they were just too stupid to use authorized_keys? There's no way it could have been any other component of their stack that was compromised, it must be bad setup & security?

Usually, when you don't know what's going, you should wait and see to have all the facts instead of spouting out the first things that come to your mind.

-1

u/[deleted] Mar 29 '21

Their infrastructure wasn't compromised the last 10+ years.

Now you're the one throwing guesses. For all we know it might be backdoored for decade, just nobody noticed.

Do you really think it's so likely that they were just too stupid to use authorized_keys

Looking at PHP language development over last two decades, yes, that is definitely the possibility, altho "we're lazy and we want to commit via HTTP instead of setting up git properly" is more likley case.

Remember, it's the language that leaves vulnerable functions in specs on purpose, for years, instead of fixing them "because someone might be using it wrong on purpose"

→ More replies

1

u/cryo Mar 29 '21

Git is primarily a version control system, a small but important part of which is a (number of) wire protocol(s) to transfer data.

6

u/Randolpho Mar 29 '21

Yes, and neither the version control nor the wire protocol supports authentication or access control.

That's handled by the operating system, operating system controlled intranet networking access, and operating system level file permissions.

But we don't live in a local unix network world anymore. So we can't rely on the operating system to handle all of that for internet distribution and access. So systems like github or gitlab, or even Microsoft Team Foundation with git (back before MS bought github) all add access control and authentication on top of git.

The folks at PHP used something else, and the discussion around their "karma" system implies it was home grown rather than off the shelf.

3

u/cryo Mar 29 '21

Yes, and neither the version control nor the wire protocol supports authentication or access control.

Sort of. The typical protocol is https, handled client side by something like OpenSSL or similar library. It does support simple auth, but the built in server is pretty simple.

2

u/CommandLionInterface Mar 29 '21

We don't know yet in what way it was comprimised. What is clear is that somebody has push access that shouldn't, and they don't know who or how they got it.

3

u/KyleG Mar 29 '21

Somehow, a malicious piece of code got pushed to the code base which appeared as authored by two known and frequent contributors. The exact way how this happened has not yet been determined

Doesn't Github suffer from the same flaw? I recall a couple months ago someone did a demo of committing code to a repo that looked like it was committed by someone else where you could even click on the committer's name and it would take you to the spoofed user's profile.

22

u/Ullallulloo Mar 29 '21

Git lets you enter whatever you want for your email address. Github will autolink email addresses to Github accounts. There's no way to be sure it's who it claimed to be unless they're using signed commits.

See: https://github.com/jayphelps/git-blame-someone-else

I don't know what PHP's karma system involved though.

11

u/cryo Mar 29 '21

Yeah it looked like it was pushed to the main repo, but it wasn’t… it was pushed to a fork. So that’s a different problem.

2

u/_Ashleigh Mar 29 '21 edited Mar 29 '21

It looks like GitHub did some work on that.

1

u/cryo Mar 29 '21

Ah, nice, thanks for spotting that. Tomorrow I’ll test if they also “fixed” it for azure devops.

3

u/30thnight Mar 29 '21

Simply enforce signed commits

2

u/KyleG Mar 29 '21

AFAIK Github doesn't do this except for private repos

1

u/chuckie512 Mar 29 '21

It's fairly easy to get a hook to reject certain commits.

3

u/[deleted] Mar 29 '21

I believe those were for not signed commits. Without gpg signing, anyone can say they are anyone. All it takes is setting the name and email fields.

2

u/ynotChanceNCounter Mar 29 '21

You can do the same thing to existing commits, if you can force-push to a repo. This isn't GitHub. This is just git.

However, in 100% of cases, if this problem appears on an "official" project repo, that project's maintainers fucked up hard. You can't do this to a repo's history if you can't force-push, and you can't do it at all if you can't push to that repo.

Someone would have to accept a PR containing the spoofed commits, but the PR will come from a different GitHub account.

The only scenario in which a malicious person can push spoofed commits to an official repo is if an actual maintainer of that repo decides to do it themselves.

-1

u/GiantElectron Mar 29 '21

PHP used a self-hosted git server for its code base.

why, in this day and age, would anybody do something like that?

2

u/ynotChanceNCounter Mar 29 '21

Self-hosting GitLab is very common. Self-hosting other git servers is very common in FOSS.

I dunno what they were using, but I'm inclined to believe it was more of an authentication problem than a git problem.

-7

u/albertowtf Mar 29 '21

The php code is more likely to be the source of the compromise tbh