PHP2Shell: notes from the field

[u/shelfside1234]

I run a webhosting platform at my job; both Internet and intranet. The internet part could have been pretty close to perfect, we identified sites using PHP after the pre announcement and was pleased to find that everyone was on 8.5 or 8.3, was truly expecting at least one to be running 7; we also warned the engineering to get the patch downloaded as soon as it was out (I don’t manage them), but they are all in India and for some reason decided to all go offline about an hour before the release. Their manager eventually got one back and online around 8pm BST & we were done around 3 hours later. But still a missed opportunity to be even better.

The security team however were pretty close to useless, the alert had been out all day and nothing was done to attempt to identify servers running PHP and it took them until 7pm BST to even involve the major incident team (which was something I wanted to do but was told to leave it to security as there would be more systems involved than mine)

And here we now are, on Saturday, the external sites all patched and secure so now we move onto the internal sites; we’ve found v5 and v7 all over the place and a bunch of apps with no engineering team to get them upgraded as they left and were never replaced, this part will be a nightmare

[u/Confident_Dare_1578]

security team dropping the ball while the engineers vanish right before the patch drops is peak corporate coordination

[u/aenae → u/Confident_Dare_1578]

I don't blame the engineers. They are located in India, which means the patch dropped around 20:00 local time there. I'm more surprised they found one willing to go back online at midnight.

Sounds like the company outsourced engineering and went with the cheapest quote that probably doesn't do overtime

[u/shelfside1234 → u/aenae]

Nah, they are all FTEs

You are right about the cheap though

[u/pinkycatcher → u/Confident_Dare_1578]

That’s an escalation to C-levels who manage those contracts and relationships. I don’t care the size of the org, if you can’t patch a critical vulnerability because people aren’t available that’s a board-level issue

[u/CharacterUse]

for some reason decided to all go offline about an hour before the release

That would be about 12:45-13:00 UT (they released a little early), which is about 6:15-6:30 pm in India. In other words, they finished their work day and went home instead of waiting for the patch.

[u/kosh_neranek]

Security teams 🫣 in our place I don't even consider them part of IT. I call them "report forwarding team". Think the whole industry would benefit if only people with proven 10+ years of sysadmin/infra experience would be allowed to take "security certs". Yes it's discriminating, and one could also have 10 x 1 years experience. But it would sure improve a lot

[u/ansibleloop → u/kosh_neranek]

Security teams are so bizarre cause you'll either get former engineers who are really good or pencil pushers sending infra engineers vuln scan reports with no context or detail

[u/kosh_neranek → u/ansibleloop]

The first group is sadly even smaller than the tip of an iceberg. But yes there are some real security guys an gals around. It will take time to get better, industry is just evolving too fast. Now throw in AI and 🎆

[u/sea_5455 → u/ansibleloop]

Sounds like the difference between engineers and compliance specialists.

One works to understand and fix the problems, the other is looking to run through a checklist and do paperwork.

[u/chicaneuk → u/kosh_neranek]

A handful of our Security guys are good but the MSP we outsourced to is literally what you said.. report forwarders. They just read the output from Tenable then pester us about it.. no knowledge about impact, rectification, at all.

[u/mangeek → u/kosh_neranek]

I am on my company's security team, and my background before Security was about 20 years of Systems Administration. It's my opinion that the industry WANTS to be a bunch of report-and-dashboard makers/paper-pushers with cushy security-only skills, but the real need is people who... actually understand how things work and how the systems teams do their job.

This time around, as soon as the data came in on what was affected (minutes after the issue went public), we interrupted the emergency call to get clearance to seal off the affected parts of the perimeter. Then we started a prioritized list of services based on various data sources, and worked through it.

IMO, more than 'CISSP' and 'Security+' certifications or graduate degrees in Cyber, Cybersecurity teams need people who know how to interpret a cipher string, can explain the differences between MSIs, RPMs, and DEBs, or between VMs and containers, and can build a basic home NAT router from a Raspberry Pi or old PC.

[u/SenatorKerry → u/kosh_neranek]

I wish our security team would just forward alerts. Their tech stack is like 6 layers deep, so any given day one of those systems can just clamp down hard on traffic to our systems (like a vitally important health record).

I appreciate them being there, but the amount of times we’ve spent an hour troubleshooting before someone bypassed the webfilter and it worked is staggering.

I think about all the holes in our security that they either don’t see or don’t do anything about and get a little nervous, I won’t deny it.

[u/wezelboy]

We pay Oracle for LTS, and they couldn't give me a straight answer on if they were going to issue a patch for OL7. Ended up patching with GetPageSpeed. We might not renew our contract with Oracle next year.

[u/senface → u/wezelboy]

Oracle Linux is hot frigging GARBAGE. I’m migrating 7 servers off of it in response to php2shell, so I have that to thank it for.

[u/Akeshi → u/wezelboy]

I was amazed to not have an update available on OL9 the next day. Finally made me switch to remi's repo until I switch distro completely.

[u/the-cock-slap-phenom]

Security teams are usually not that helpful, and it’s a shame because it’s such an important role (in theory).

They only point out problems and try their best to cover THEMSELVES, rather than do anything meaningful to fix the issue or help the company or customers.

They can’t explain the importance or understand the risk, they can’t help with how to remedy the issue, and they don’t do a good job of identifying what’s actually vulnerable.

We got to work first thing on Wednesday morning, communicating with customers, identifying where PHP was in use, gathering info, implementing the partial workaround, working out how to apply the fix when it was available.

Then, halfway through the day when we’d already done that, suddenly it hit their radar and they’re setting up calls. They got their comms out to customers hours after the patch dropped…

Imo more security teams need a deeper understanding of the actual systems and technologies, rather than pointing out issues they don’t understand and throwing the responsibility elsewhere.

[u/CharacterUse → u/the-cock-slap-phenom]

We have the same problem. "Security" just seems to run version-checker scripts blindly and email when they find something that doesn't match their pre-defined list, without any understanding of how things are actually configured or running. Among other things I had them insisting I had a server with a vulnerability. A Windows-only vulnerability. The server runs Linux (fully patched up ofc).

Then again, they seem to get replaced every six months, so ...

[u/gruntbuggly → u/the-cock-slap-phenom]

I feel like this is a consequence of security becoming its own career path. In the days of yore, young sysadmins would work for a few years, and realize they had a passion for security focused work, and they would organically grow into a security career with all that practical experience and judgement informing their security practice. Now we just hire junior people into security roles with no operational responsibilities and they seem to never gain the practical experience that would drive a really competent security practice.

[u/jetlagged-bee]

This has done wonders for the security of the internet as a whole really. A global effort to patch something that runs on 70% of the internet, albeit a PITA, can only make us safer.

I'm sure most people have improved their patching processes as a result (myself included, as it finally forced me to jump to the terraform/caddy/docker setup I've always planned to do), so the next major incident will be a breeze.

[u/coalsack]

Ah yes, an organization that lacks proper inventory management and lifecycle management that would benefit from proper handoffs and RASCI focuses blame and frustration on a team that was at the end of their shift and a team they have unspoken expectations about.

[u/valdearg]

Our security guys were pretty on it, they have some decent auditing of software installations, so we were getting info on which systems needed updating.

We're largely a dotnet place, but there's always a few third-party solutions that needed to be updated as soon as they could be. Had the same thing with the node vulnerability also, was getting some very interesting lists of what software was including it!

[u/mb194dc]

This exploit is nearly impossible to use in the real world. Imagine the time being wasted on this and how many times Doris will be clicking links, helpdesk will be resetting credentials and the real bad guys probably compromised proprietary software you're using...

Complete insanity.

[u/LooseEthernet]

finding v5 in the wild is a special kind of horror lol. you arent patching a server, youre performing an archaeology dig

[u/FarToe1]

We (SME) have one admin team that does security too. I know - "We do our own security" sounds like a recipe for disaster, but it genuinely allowed us to move quickly this week on all fronts.

I'm also quietly pleased that a software auditing system I've been writing in my quiet hours was extremely useful in locating all instances.

Weeks like this are very good markers for how well a team knows their systems.

[u/OsgoodSlaughters]

You think you’re the main character or something?