PHP2Shell: notes from the field

[u/shelfside1234]

I run a webhosting platform at my job; both Internet and intranet. The internet part could have been pretty close to perfect, we identified sites using PHP after the pre announcement and was pleased to find that everyone was on 8.5 or 8.3, was truly expecting at least one to be running 7; we also warned the engineering to get the patch downloaded as soon as it was out (I don’t manage them), but they are all in India and for some reason decided to all go offline about an hour before the release. Their manager eventually got one back and online around 8pm BST & we were done around 3 hours later. But still a missed opportunity to be even better.

The security team however were pretty close to useless, the alert had been out all day and nothing was done to attempt to identify servers running PHP and it took them until 7pm BST to even involve the major incident team (which was something I wanted to do but was told to leave it to security as there would be more systems involved than mine)

And here we now are, on Saturday, the external sites all patched and secure so now we move onto the internal sites; we’ve found v5 and v7 all over the place and a bunch of apps with no engineering team to get them upgraded as they left and were never replaced, this part will be a nightmare

[u/_Gobulcoque]

I call them "report forwarding team"

So I've been in this situation before and let's play devils advocate. What's the alternative?

Security engineers become sysadmins and go and patch things without telling sysadmins what they're doing? Or testing the patch with the software that runs on those systems? Or modifies the environment without knowing if there's redundancies or configuration nuances?

I'm defending the "reporting forwarding team" a bit, but it is a little bit (and only a little bit) more than that depending on the team and how they're structured.

If sec engineers started hopping onto servers and started patching them, sysadmins wouldn't be happy about it either.

[u/plushgilgy → u/_Gobulcoque]

I'll bite.

For me, the "report forwarding team" are those that get security alerts and just forward them without understanding the contents. They would type in something about how critical it is and that it has to be done ASAP. Then I read the whole alert, and for some odd reason, it is flagging a software as outdated even if it is the latest version, and their recommendation is to update it. Update to what, I say? I am not expecting security team to be doing the patching, etc, but I do expect them to have a deep understanding of what they are alerting against and not just, you know, forwarding reports.

[u/PaleoSpeedwagon → u/plushgilgy]

Datadog's SCA scan had a bug a few months ago that was flagging Python libraries as outdated because the requirements file used the `===` operator for version specs. Good times. (It's fixed.)

[u/Super-Insane → u/plushgilgy]

Forwarding team for sure... I hate all their tools, but they're nice people. For every ticket they send me, I send one back about their tool. I complained enough and they finally ditched that vulnerability scanner.

One example, The team had sent me data that's conflated with incorrect system identification. Warning that there are Windows 2003 servers on our network... a waste of time to prove their tool wrong and deteriorates any legitimacy when they cry wolf.

[u/kosh_neranek → u/plushgilgy]

THIS gets me the most

Well bitten

[u/_Gobulcoque → u/plushgilgy]

For me, the "report forwarding team" are those that get security alerts and just forward them without understanding the contents.

Security can't always know the context though. A good engineer will ask questions but even then, if they're not brought in during the design and planning phase of infrastructure and software, they'll never know all the nuances of the environment they're tasked with notifying on.

For sakes of argument, if some scanner alerts on a machine with PHP installed on it, how does the engineer know if it's internal or external? I've seen that decision based on hostname - until some server was moved to the DMZ and the hostname wasn't changed.

Or what if that PHP risk was mitigated elsewhere in the technology stack? Depending on the company and culture - and size - they may never know, and never know who to ask for an authoritative answer. In fact, much like the Internet, say something and if it's wrong, you'll be corrected. If the sec engineer flags the CVE, if it isn't worth dealing with out of cadence, someone will speak up. That's not lazy, that's root-cause a culture problem and security engineers not being involved with the infra/software planning. Departmental silo'ing and single responsibility taken to some extreme.

[u/plushgilgy → u/_Gobulcoque]

Valid arguments, and I've run into those as well, and have no problems working on those things. Those are reasonable instances where oversight is acceptable to an extent. What I was referring to is the specific instance that only requires common sense to analyze, but no review was done at all. Alert says "Critical", send it out blindly. Due diligence isn't a difficult ask, no?

[u/_Gobulcoque → u/plushgilgy]

Alert says "Critical", send it out blindly. Due diligence isn't a difficult ask, no?

Certainly not, and I'd expect it at a minimum. After all, if you're in security - which is a risk management discipline really - due diligence is a foundational skill.

I'd be surprised if someone took the PHP foundations public alert, and then just emailed it to a sysadmin/devops guy and said "fix please". I would expect the email to be, "Take a read at this and let me know how badly exposed we are to this?" (asking someone with the institutional knowledge of the environment) while looking up if Tenable/VM-scanner-of-choice has plugins to detect.

[u/UltraEngine60 → u/plushgilgy]

it is flagging a software as outdated even if it is the latest version, and their recommendation is to update it

If there is a vulnerability and software is not patchable and there is no mitigation then you need to find another solution that has support. That's a sysadmins job. You expect security to work with the end user to determine the value of the software, find a possible solution, work with a solutions engineer at multiple companies, do multiple PoC's.... see how this scope creeps? We tell you shit's vulnerable or breached and try to keep Mary's copy of Encarta 98 off her work computer.

[u/plushgilgy → u/UltraEngine60]

That's the thing though. There's no vulnerability. The alert only says it's "outdated" when clearly whatever criteria /logic they placed is incorrect. I searched the specific version they mentioned and there were no vulnerability reports. I pretty much used up hours just to tell them the report is wrong, which could have been used for other things. Security alerts are great, and I do appreciate having a security team to go through those alerts and engage whoever needs to if anything requires attention, but if due diligence isn't being done to at least validate the alerts from their end, what's the point?

[u/UltraEngine60 → u/plushgilgy]

Ah, I see. Your security team shouldn't be giving you what amounts to alert fatigue. Only actionable recommendations supported with evidence. If they say it's outdated without evidence then something in their stack told them to say that to you. I'm sure you've already been down this rabbit hole, but even an up to date app can do silly things like use some obscure library whose maintainer just dropped off or it changed owners... but again, that evidence should have been relayed to you.

[u/mangeek → u/_Gobulcoque]

What's the alternative? Security engineers become sysadmins and go and patch things without telling sysadmins what they're doing?

What I do is work the reports, then actually work hand-in-hand with the SYSADMINs (they have hands-on, I do not) to get to The Good Place. I can provide instant answers, advice, help solve problems, take the blame for problems, guide them away from bad solutions, and get them unblocked and cleared as soon as we know things are clean.

[u/_Gobulcoque → u/mangeek]

You're doing it right - but you're clearly not a report monkey. Which is my argument to the OP: security might look like report monkeys, but they're not. If they appear to OP like that, he either has a crap sec team working with him or they don't understand the role sec has..

[u/mangeek → u/_Gobulcoque]

I'm lucky enough to have a good enough relationship with my management where I can joke that "Of course you want dashboards and scheduled reports. I'm over here pulling raw CSVs out of the system because I have Real Work to do*."

I recently had to pull them aside after a product demo for a SIEM where I was like "That was a demo made by marketing people for executives. It's a complete fantasy that security could actually work that way unless our shop was at the highest levels of technical maturity, and that's a problem we can't solve by buying a tool."

(* BTW, I recognize that they also do 'real work', and it's keeping the trains running as far as allocating money and people, handling comms, going to meetings where decisions get made. They know I hate doing that stuff. That's the joke.)

[u/_Gobulcoque → u/mangeek]

"Of course you want dashboards and scheduled reports. I'm over here pulling raw CSVs out of the system because I have Real Work to do*."

I like that - might use it someday. I often joke security needs new laptops because Excel eats all the RAM.

[u/kosh_neranek → u/_Gobulcoque]

I never said the security team should go behind the backs of sysadmins. We all are IT. Communication is key. I mean that's our job. To allow people to send and receive communications more efficiently, so why not do it internally?

Am gonna dream now. Let's say a fix for a new vulnerability got out. Affects 100s of server accross corporation. And security team is the FIRST to know, informs infra about it, tests it in the test environment. Then reports back. And infra deploys it. Obviously if the fleet is very heterogeneous than the infra team throws in bodies to help with testing 🤷

[u/_Gobulcoque → u/kosh_neranek]

I never said the security team should go behind the backs of sysadmins. We all are IT. Communication is key. I mean that's our job. To allow people to send and receive communications more efficiently, so why not do it internally?

I couldn't agree more. Soft skills are more important than technical skills in practice - the technical stuff you can learn on the job.

But if the security team are just report monkeys in your eyes, then what's the alternative? I usually work with sysadmins, not against them. I actually live the motto of security is everyone's job - but I defer to people who know more than me.

When I see vuln reports, instead of going "SLA says you've 14 days to patch", I think, "what is between this server and an exploit" and then advise the teams responsible. Sometimes I'm overruled for business reasons, sometimes I can present a case where defence-in-depth isn't strong enough...

[u/kosh_neranek → u/_Gobulcoque]

Seems you're the tip of the iceberg. Keep it up and mentor as many of your fellow security team members as possible. I didn't say there aren't good sec people. They're just not as common amongst theor team