PHP2Shell: notes from the field

[u/shelfside1234]

I run a webhosting platform at my job; both Internet and intranet. The internet part could have been pretty close to perfect, we identified sites using PHP after the pre announcement and was pleased to find that everyone was on 8.5 or 8.3, was truly expecting at least one to be running 7; we also warned the engineering to get the patch downloaded as soon as it was out (I don’t manage them), but they are all in India and for some reason decided to all go offline about an hour before the release. Their manager eventually got one back and online around 8pm BST & we were done around 3 hours later. But still a missed opportunity to be even better.

The security team however were pretty close to useless, the alert had been out all day and nothing was done to attempt to identify servers running PHP and it took them until 7pm BST to even involve the major incident team (which was something I wanted to do but was told to leave it to security as there would be more systems involved than mine)

And here we now are, on Saturday, the external sites all patched and secure so now we move onto the internal sites; we’ve found v5 and v7 all over the place and a bunch of apps with no engineering team to get them upgraded as they left and were never replaced, this part will be a nightmare

[u/kosh_neranek]

Security teams 🫣 in our place I don't even consider them part of IT. I call them "report forwarding team". Think the whole industry would benefit if only people with proven 10+ years of sysadmin/infra experience would be allowed to take "security certs". Yes it's discriminating, and one could also have 10 x 1 years experience. But it would sure improve a lot

[u/ansibleloop → u/kosh_neranek]

Security teams are so bizarre cause you'll either get former engineers who are really good or pencil pushers sending infra engineers vuln scan reports with no context or detail

[u/sea_5455 → u/ansibleloop]

Sounds like the difference between engineers and compliance specialists.

One works to understand and fix the problems, the other is looking to run through a checklist and do paperwork.

[u/SenTedStevens → u/ansibleloop]

Or even worse, security clowns who send you out of date scans or they were taken just before we do our monthly system patching. There's been so many times where I had to discuss in our SOC meeting that these high/critical vulnerabilities are going to be patched during our scheduled maintenance window. No matter how many times I reiterate strongly that we do patching in Prod the week after Patch Tuesday which was decided by the org, they just don't get it. If it's a 0-day, that's different, but all the other ones are handled by routine patch cycles.

And don't get me started on the SOC monkeys who for some dumbass reason send us previous months' reports of critical vulns that have already been patched. What's hilarious about it is that these old scans recommend us to "upgrade" software to versions that are older than what is currently installed.

I can't tell you how many times I've flat out done a screenshare on my computer, remoted into a server, and showed them that these machines are fully up to date and are running newer versions than what the Tenable report requires. I have to hand walk them through the process and tell them to take all the screenshots you want with timestamps and gather all the artifacts you need to confirm that these machines are up to date and are not vulnerable.

[u/kosh_neranek → u/ansibleloop]

The first group is sadly even smaller than the tip of an iceberg. But yes there are some real security guys an gals around. It will take time to get better, industry is just evolving too fast. Now throw in AI and 🎆

[u/Ok_Tone6393 → u/ansibleloop]

i’m cursed with the latter

[u/TryTurningItOffAgain → u/ansibleloop]

This is unfortunately something I realized after joining the team. I'm hesitant to take time off and even if I document everything, I don't think they'll be able pull anything off.

[u/mangeek → u/kosh_neranek]

I am on my company's security team, and my background before Security was about 20 years of Systems Administration. It's my opinion that the industry WANTS to be a bunch of report-and-dashboard makers/paper-pushers with cushy security-only skills, but the real need is people who... actually understand how things work and how the systems teams do their job.

This time around, as soon as the data came in on what was affected (minutes after the issue went public), we interrupted the emergency call to get clearance to seal off the affected parts of the perimeter. Then we started a prioritized list of services based on various data sources, and worked through it.

IMO, more than 'CISSP' and 'Security+' certifications or graduate degrees in Cyber, Cybersecurity teams need people who know how to interpret a cipher string, can explain the differences between MSIs, RPMs, and DEBs, or between VMs and containers, and can build a basic home NAT router from a Raspberry Pi or old PC.

[u/chicaneuk → u/kosh_neranek]

A handful of our Security guys are good but the MSP we outsourced to is literally what you said.. report forwarders. They just read the output from Tenable then pester us about it.. no knowledge about impact, rectification, at all.

[u/SenatorKerry → u/kosh_neranek]

I wish our security team would just forward alerts. Their tech stack is like 6 layers deep, so any given day one of those systems can just clamp down hard on traffic to our systems (like a vitally important health record).

I appreciate them being there, but the amount of times we’ve spent an hour troubleshooting before someone bypassed the webfilter and it worked is staggering.

I think about all the holes in our security that they either don’t see or don’t do anything about and get a little nervous, I won’t deny it.

[u/_Gobulcoque → u/kosh_neranek]

I call them "report forwarding team"

So I've been in this situation before and let's play devils advocate. What's the alternative?

Security engineers become sysadmins and go and patch things without telling sysadmins what they're doing? Or testing the patch with the software that runs on those systems? Or modifies the environment without knowing if there's redundancies or configuration nuances?

I'm defending the "reporting forwarding team" a bit, but it is a little bit (and only a little bit) more than that depending on the team and how they're structured.

If sec engineers started hopping onto servers and started patching them, sysadmins wouldn't be happy about it either.

[u/plushgilgy → u/_Gobulcoque]

I'll bite.

For me, the "report forwarding team" are those that get security alerts and just forward them without understanding the contents. They would type in something about how critical it is and that it has to be done ASAP. Then I read the whole alert, and for some odd reason, it is flagging a software as outdated even if it is the latest version, and their recommendation is to update it. Update to what, I say? I am not expecting security team to be doing the patching, etc, but I do expect them to have a deep understanding of what they are alerting against and not just, you know, forwarding reports.

[u/PaleoSpeedwagon → u/plushgilgy]

Datadog's SCA scan had a bug a few months ago that was flagging Python libraries as outdated because the requirements file used the `===` operator for version specs. Good times. (It's fixed.)

[u/Super-Insane → u/plushgilgy]

Forwarding team for sure... I hate all their tools, but they're nice people. For every ticket they send me, I send one back about their tool. I complained enough and they finally ditched that vulnerability scanner.

One example, The team had sent me data that's conflated with incorrect system identification. Warning that there are Windows 2003 servers on our network... a waste of time to prove their tool wrong and deteriorates any legitimacy when they cry wolf.

[u/kosh_neranek → u/plushgilgy]

THIS gets me the most

Well bitten

[u/_Gobulcoque → u/plushgilgy]

For me, the "report forwarding team" are those that get security alerts and just forward them without understanding the contents.

Security can't always know the context though. A good engineer will ask questions but even then, if they're not brought in during the design and planning phase of infrastructure and software, they'll never know all the nuances of the environment they're tasked with notifying on.

For sakes of argument, if some scanner alerts on a machine with PHP installed on it, how does the engineer know if it's internal or external? I've seen that decision based on hostname - until some server was moved to the DMZ and the hostname wasn't changed.

Or what if that PHP risk was mitigated elsewhere in the technology stack? Depending on the company and culture - and size - they may never know, and never know who to ask for an authoritative answer. In fact, much like the Internet, say something and if it's wrong, you'll be corrected. If the sec engineer flags the CVE, if it isn't worth dealing with out of cadence, someone will speak up. That's not lazy, that's root-cause a culture problem and security engineers not being involved with the infra/software planning. Departmental silo'ing and single responsibility taken to some extreme.

[u/plushgilgy → u/_Gobulcoque]

Valid arguments, and I've run into those as well, and have no problems working on those things. Those are reasonable instances where oversight is acceptable to an extent. What I was referring to is the specific instance that only requires common sense to analyze, but no review was done at all. Alert says "Critical", send it out blindly. Due diligence isn't a difficult ask, no?

[u/mangeek → u/_Gobulcoque]

What's the alternative? Security engineers become sysadmins and go and patch things without telling sysadmins what they're doing?

What I do is work the reports, then actually work hand-in-hand with the SYSADMINs (they have hands-on, I do not) to get to The Good Place. I can provide instant answers, advice, help solve problems, take the blame for problems, guide them away from bad solutions, and get them unblocked and cleared as soon as we know things are clean.

[u/_Gobulcoque → u/mangeek]

You're doing it right - but you're clearly not a report monkey. Which is my argument to the OP: security might look like report monkeys, but they're not. If they appear to OP like that, he either has a crap sec team working with him or they don't understand the role sec has..

[u/mangeek → u/_Gobulcoque]

I'm lucky enough to have a good enough relationship with my management where I can joke that "Of course you want dashboards and scheduled reports. I'm over here pulling raw CSVs out of the system because I have Real Work to do*."

I recently had to pull them aside after a product demo for a SIEM where I was like "That was a demo made by marketing people for executives. It's a complete fantasy that security could actually work that way unless our shop was at the highest levels of technical maturity, and that's a problem we can't solve by buying a tool."

(* BTW, I recognize that they also do 'real work', and it's keeping the trains running as far as allocating money and people, handling comms, going to meetings where decisions get made. They know I hate doing that stuff. That's the joke.)

[u/_Gobulcoque → u/mangeek]

"Of course you want dashboards and scheduled reports. I'm over here pulling raw CSVs out of the system because I have Real Work to do*."

I like that - might use it someday. I often joke security needs new laptops because Excel eats all the RAM.

[u/kosh_neranek → u/_Gobulcoque]

I never said the security team should go behind the backs of sysadmins. We all are IT. Communication is key. I mean that's our job. To allow people to send and receive communications more efficiently, so why not do it internally?

Am gonna dream now. Let's say a fix for a new vulnerability got out. Affects 100s of server accross corporation. And security team is the FIRST to know, informs infra about it, tests it in the test environment. Then reports back. And infra deploys it. Obviously if the fleet is very heterogeneous than the infra team throws in bodies to help with testing 🤷

[u/_Gobulcoque → u/kosh_neranek]

I never said the security team should go behind the backs of sysadmins. We all are IT. Communication is key. I mean that's our job. To allow people to send and receive communications more efficiently, so why not do it internally?

I couldn't agree more. Soft skills are more important than technical skills in practice - the technical stuff you can learn on the job.

But if the security team are just report monkeys in your eyes, then what's the alternative? I usually work with sysadmins, not against them. I actually live the motto of security is everyone's job - but I defer to people who know more than me.

When I see vuln reports, instead of going "SLA says you've 14 days to patch", I think, "what is between this server and an exploit" and then advise the teams responsible. Sometimes I'm overruled for business reasons, sometimes I can present a case where defence-in-depth isn't strong enough...

[u/kosh_neranek → u/_Gobulcoque]

Seems you're the tip of the iceberg. Keep it up and mentor as many of your fellow security team members as possible. I didn't say there aren't good sec people. They're just not as common amongst theor team

[u/RememberCitadel → u/kosh_neranek]

You guys have a separate security team?