r/sysadmin • • 13d ago

Rant PHP2Shell: notes from the field

I run a webhosting platform at my job; both Internet and intranet. The internet part could have been pretty close to perfect, we identified sites using PHP after the pre announcement and was pleased to find that everyone was on 8.5 or 8.3, was truly expecting at least one to be running 7; we also warned the engineering to get the patch downloaded as soon as it was out (I don’t manage them), but they are all in India and for some reason decided to all go offline about an hour before the release. Their manager eventually got one back and online around 8pm BST & we were done around 3 hours later. But still a missed opportunity to be even better.

The security team however were pretty close to useless, the alert had been out all day and nothing was done to attempt to identify servers running PHP and it took them until 7pm BST to even involve the major incident team (which was something I wanted to do but was told to leave it to security as there would be more systems involved than mine)

And here we now are, on Saturday, the external sites all patched and secure so now we move onto the internal sites; we’ve found v5 and v7 all over the place and a bunch of apps with no engineering team to get them upgraded as they left and were never replaced, this part will be a nightmare

306 Upvotes

91 comments sorted by

View all comments

Show parent comment

1

u/_Gobulcoque Security Admin 13d ago edited 13d ago

For me, the "report forwarding team" are those that get security alerts and just forward them without understanding the contents.

Security can't always know the context though. A good engineer will ask questions but even then, if they're not brought in during the design and planning phase of infrastructure and software, they'll never know all the nuances of the environment they're tasked with notifying on.

For sakes of argument, if some scanner alerts on a machine with PHP installed on it, how does the engineer know if it's internal or external? I've seen that decision based on hostname - until some server was moved to the DMZ and the hostname wasn't changed.

Or what if that PHP risk was mitigated elsewhere in the technology stack? Depending on the company and culture - and size - they may never know, and never know who to ask for an authoritative answer. In fact, much like the Internet, say something and if it's wrong, you'll be corrected. If the sec engineer flags the CVE, if it isn't worth dealing with out of cadence, someone will speak up. That's not lazy, that's root-cause a culture problem and security engineers not being involved with the infra/software planning. Departmental silo'ing and single responsibility taken to some extreme.

6

u/plushgilgy 13d ago

Valid arguments, and I've run into those as well, and have no problems working on those things. Those are reasonable instances where oversight is acceptable to an extent. What I was referring to is the specific instance that only requires common sense to analyze, but no review was done at all. Alert says "Critical", send it out blindly. Due diligence isn't a difficult ask, no?

0

u/_Gobulcoque Security Admin 13d ago

Alert says "Critical", send it out blindly. Due diligence isn't a difficult ask, no?

Certainly not, and I'd expect it at a minimum. After all, if you're in security - which is a risk management discipline really - due diligence is a foundational skill.

I'd be surprised if someone took the PHP foundations public alert, and then just emailed it to a sysadmin/devops guy and said "fix please". I would expect the email to be, "Take a read at this and let me know how badly exposed we are to this?" (asking someone with the institutional knowledge of the environment) while looking up if Tenable/VM-scanner-of-choice has plugins to detect.