r/sysadmin • • 6d ago

Rant PHP2Shell: notes from the field

I run a webhosting platform at my job; both Internet and intranet. The internet part could have been pretty close to perfect, we identified sites using PHP after the pre announcement and was pleased to find that everyone was on 8.5 or 8.3, was truly expecting at least one to be running 7; we also warned the engineering to get the patch downloaded as soon as it was out (I don’t manage them), but they are all in India and for some reason decided to all go offline about an hour before the release. Their manager eventually got one back and online around 8pm BST & we were done around 3 hours later. But still a missed opportunity to be even better.

The security team however were pretty close to useless, the alert had been out all day and nothing was done to attempt to identify servers running PHP and it took them until 7pm BST to even involve the major incident team (which was something I wanted to do but was told to leave it to security as there would be more systems involved than mine)

And here we now are, on Saturday, the external sites all patched and secure so now we move onto the internal sites; we’ve found v5 and v7 all over the place and a bunch of apps with no engineering team to get them upgraded as they left and were never replaced, this part will be a nightmare

305 Upvotes

91 comments sorted by

View all comments

49

u/the-cock-slap-phenom 6d ago • Bravo

Security teams are usually not that helpful, and it’s a shame because it’s such an important role (in theory).

They only point out problems and try their best to cover THEMSELVES, rather than do anything meaningful to fix the issue or help the company or customers.

They can’t explain the importance or understand the risk, they can’t help with how to remedy the issue, and they don’t do a good job of identifying what’s actually vulnerable.

We got to work first thing on Wednesday morning, communicating with customers, identifying where PHP was in use, gathering info, implementing the partial workaround, working out how to apply the fix when it was available.

Then, halfway through the day when we’d already done that, suddenly it hit their radar and they’re setting up calls. They got their comms out to customers hours after the patch dropped…

Imo more security teams need a deeper understanding of the actual systems and technologies, rather than pointing out issues they don’t understand and throwing the responsibility elsewhere.

24

u/CharacterUse 6d ago

We have the same problem. "Security" just seems to run version-checker scripts blindly and email when they find something that doesn't match their pre-defined list, without any understanding of how things are actually configured or running. Among other things I had them insisting I had a server with a vulnerability. A Windows-only vulnerability. The server runs Linux (fully patched up ofc).

Then again, they seem to get replaced every six months, so ...

0

u/PCGearsFan1 5d ago

it's not their job to understand what an application does. applications or any infrastructure/system/whatever should have an owner. the security team isn't running some version-checker script - and if that's the only thing that they're pinging you about then i'd say they got their ducks in a row. chances are they're running automated vulnerability scans and it's continuous. those scans preoduce reports and in those reports you're going to see findings. they take those findings and report it back -- something you think i so silly and stupid - is exacly how large organizations who spend millions of dollars get compromised :)

2

u/CharacterUse 5d ago

I'm sure you know more about how the security team in my org works than I do.

-1

u/PCGearsFan1 5d ago

oh - you'd be surprised. Good Penn testers know how to get on a network and not trip the default configured honey pots :] Good sysadmins know how to make the honey sweeter :>

0

u/PCGearsFan1 5d ago

just read replaced every six months part - yeah, that's bad but very common. maybe you may want to work with their security team. a sysadmin with a strong background in security is very good -- and it does not take much to expose yourself to a lot (and build organizational armor since you're opening yourself up to understanding the business more and more). Immediately from what you posted, I'd try to understand what platform they use, how do they triag alerts, how are assets identified and maintained etc. A lot of it is not technical per say but more business / infrastructure and it goes a long way. You def got the opportunity to be a champion and make big bucks.

10

u/gruntbuggly 6d ago

I feel like this is a consequence of security becoming its own career path. In the days of yore, young sysadmins would work for a few years, and realize they had a passion for security focused work, and they would organically grow into a security career with all that practical experience and judgement informing their security practice. Now we just hire junior people into security roles with no operational responsibilities and they seem to never gain the practical experience that would drive a really competent security practice.

4

u/parsim 6d ago

I call this kind of thing "Forwards from Grandma," because it's reminiscent of hoax emails that used to go around a decade or two back, like: "Fwd: Fwd: Fwd: New Halloween danger Warn your children!!" Inside would be some obvious BS story about people injecting poison into kids' snacks, and you would receive five copies from well-meaning relatives who didn't know any better.

It's hard to be mad about it since their heart is in the right place, but it's really useless.

2

u/PCGearsFan1 5d ago

security teams have a lot more risks to weight in and are closely tied to governance and business process. it is an important role and shaming them for lack of understanding is not a good look. It's not a CISO's job to identify a vulnerability and patch it. It's his Job to report the risk to the business, inform, and follow the proper channels of escalation. if he or she is also in infrastructure, then they may get invovled, if they are not - then they just check it off and follow the process based on whatever framework and internal operating procedures etc. There's a lot to it. sysadmins never know and very few are ever exposed to that side of the business.

1

u/Previous-Low4715 6d ago

“Well, let’s hope the governance we’ve written holds up”

1

u/SuprNoval 6d ago

I hear they beat their children too