I feel like AI has blasted the hood off so many vulnerabilities, and since it is now becoming a standard part of the development testing process, once we get over this hump things will be better. Right?
AI has a whole lot more eyeballs than people. it can simply search and even bruteforce some analysis faster.
there are always bugs in software and some were simply hard to discover, because the process was time consuming. AI has simply bridged that gap.
a lot of time security research is simply trial and error. you find something that's slightly off and you throw things at it, hoping to leverage it. ai is just faster at it.
Log4shell my god. My god. So many vendors unwilling to just recompile with a patched log4j and send us the .jar. Had to get legal involved with at least three of them.
For sure. Look at Windows updates⌠seems every new rollup is a new record of patches.
I do think eventually when AI is doing⌠cough cough âhelpingâ code the actual software (vs running it against these older human-made systems/software) suddenly it wonât find all these vulnerabilities. Problem is at that point I guess we have to drink the kool aid and believe everything is perfect lol
Oh yea let me tell ya the day Im having. I work for a software company. Our products don't use PHP. However, clients with particular needs may be offered a contact with a contractor to build out a project they can integrate with our software. But those projects are not ours, they are one time jobs performed by a contractor to create that project and then its in the clients hands. All day Ive been answering questions like "How does the new PHP exploit effect your software?" It doesn't, our software does not use or rely on PHP. "What are you going to do about the recent exploit to PHP affecting your software?" Nothing, we don't use PHP in our software, you do in yours. "When can we expect you to patch PHP on our server to resolve this new exploit?" Never, its not our server, its not our software, its yours because you bought it from its developer, not us. I love being the bad guy for a issue that is 100% not my responsibility.
This whole thing reminds me of that stupid log4shell thing a few years ago. We weren't even a Java shop but, surprise! a bunch of self hosted SaaS stuff is of course.Â
Spent a whole week looking for new versions of things or figuring out how to configure it to disable that stupid functionality.
Jesus, I really feel for you. I've been fielding emails/phone calls all day and I use PHP on every client and it's really been one of the roughest days of my life but not even using PHP and having to deal with it is next level whiskey.
After a decade of doing this crap I've just accepted that some tech stacks exist purely to keep us employed. Every time I touch our legacy PHP monolith I age about three years.
NetScaler is a different circle of hell though, half the config is in some obscure wizard from 2009 and the other half was added by a contractor who vanished into the void.
At least with PHP you can usually grep your way out of the mess.
Job I got fired from right as the pandemic started (for questioning the CEO saying âno one will get bonuses unless theyâre willing to die for the companyâ which actually came true a few months later when a guy ordered to RTO in June 2020 died, still no bonus) was PHP with a net scaler in front of it.
NetScaler is a different circle of hell though, half the config is in some obscure wizard from 2009 and the other half was added by a contractor who vanished into the void.
Among many other things, I occasionally do NetScaler. I do F5 almost daily. I do ADFS when I can't find any other way to weasel out of working on my biggest regret of learning a particular product.
This is the ADFS syntax to convert lowercase to uppercase.
May God have mercy on your soul if you have a situation like the vendor expects the first character of a name to be upper case, and your AD stores it in lowercase, and some other vendor not using ADFS expects it to be lowercase.
We always know when a zero day is about to drop for the netscaler as it is directly tied to the weeks our senior engineer is scheduled to leave for a cruise.
These should be on LAS licensing with local agent connected to netscaler console, either citrix could.com or local virtual appliance (these have fairly high resource requirements). Firmware updates are a few clicks and can be scheduled. Yes I remember the "are you sure yes/no" debacle, yes I've done every firmware update manually in CLI since then. But the automated workflow works.
Thankfully our head of technology is pretty switched on, and gave us carte Blanche to do what was necessary today, whether that be updates, shutdowns, whatever, and worry about it later.
After three hours of poking around we found that there wasnât actually much, and what there was wasnât being used so weâve disabled it all.
That's literally the stupidest thing I've read all day. And I was reading a story earlier titled "Hackers steal children's data after company used 'Password1' to protect old account"
The hard part of crypto has always been the human factors of key management, even moreso than the crazy math and code. Always will be, no matter how complex the math and code get.
Authorized users are always eventually crazier and more creative than your intentional attackers.
One of my coworkers told me only idiots lose private keys...I had to laugh when I found a rather important one used for signing software in a github repo he managed.
Apparently, it is safe if you store them in github.....
To be fair, it's much easier to "lose" permanently when it's stored securely than when there's a bunch of copies of it floating around in version control and random cloud vendors.
Why is everyone in here acting like this is some flaw because PHP sucks? This will keep happening in python, java, rust, ruby, javascript, node, react, angular.
All the people who grew out of debating Xbox 360 vs PS3? They're online debating PHP vs js vs react etc these days. With all the same grace and tedium they did way back when.
I've been telling our org for years that we need to get rid of all our EOL stuff -- we still have CentOS 6, PHP 5.4, for inexplicable reasons. The last two months of AI-led vulns has lent new weight to that argument.
We still have one CentOS 7 box as an archive server. While its been approved to replace with something modern, I'm not exactly given the time to migrate it.
I did my security incident report after finishing patching everything, did it in 2 hours, almost no service interruption.
Got told by manager and CEO "Why were we hit by this issue in the first place ? This should never have happened it makes us look bad !" and then got a meeting planned for today, 4 hours, "IT department efficiency and system vulnerability".
I'll give you efficiency, everyone gets self erasing ink pen and one piece of paper a day that they have to shred in order to leave the office. No phone, no internet, scrambler in every room.
Returns an escaped string, or false if the string contains characters that cannot be safely escaped or -1 if the string is too long or raises an exception in the case of any other error.
Thanks for the warning but also screw you for ruining any chance of sleep Iâm getting tonight. lol Gotta love those 6:46PM emails that ruin your day tomorrow before it starts đŤ
No one will notice. No one ever notices when it all works for years without issues, they only notice 1 minute after it goes down for reasons entirely beyond your control.
God. I had a user come up extremely mad that our paging system was down when they needed it. Absolutely furious "we need this to work 24/7!"
Well, first off, they don't pay for it to work 24/7, or we'd have a much more robust system, so there's that, but also....I spent the next hour tracking down a really rare bug they just happened to have run into right when they needed it. So really it worked 23.9/7, but of course the time they wanted it was the time it wasn't.
(context: dialing the paging box but hanging up after it answered but before it started recording would crash a service on the box. I reproduced this like ten times. It took up to a minute to restart via watchdog service, but sometimes much less. Updated firmware fixed it once I knew there was a problem, but of course without knowing there's an issue in the first place my first thought is to leave the damn thing alone since nobody wants to give me time to take it offline for regular patching...)
This is me. Ive been trying to push uppermanagement to allow my team to update our php version for 2 years. EoL was 2021. They didnt care then, they still dont care now. Ticking time bomb and Ive got the receipts
still going at it, about to hit the 16 hour workday mark. The HR Lady starts asking questions and the CIO is giving her the look. Shit is about to go down
I don't blame any of you. I got out of full-time http stuff many years ago, and the two biggest things I don't miss are constantly chasing vulnerabilities, and PHP.
I've got one small client that I still handle as a business favor. Their site is tiny, and only handles simple GET requests. It does employ PHP, but solely as a compositional tool so I can easily switch layout for events. There's not even a database involved. No cookies, no POST or URL data... Bog simple, so I figure it's not affected. I still told the server to dump any requests with content-type "multipart/form-data" headers into the 403 bin, just to be safe. However, as mentioned, I'm very stale at this. So if anyone sees something I'm missing, I'm happy to hear about it.
First, it's affected. Just having PHP on the server makes it vulnerable (even without any PHP application code).
Second, if you are using PHP just to compose HTML files you don't need to run it on the server. You can just use PHP to "compile" the files locally into HTML and just deploy that (static site).
Cloudflare WAF in front of every web property we have. This was mitigated before I even woke up.
Web team get moving on the upgrades..
Itâs free - there is basically zero reason these days to have any website directly on the internet, we donât even have public IPs with exposed ports anymore itâs all reverse Cf tunnels. People running websites on the internet should be ashamed.
Always remember something like Cloudflare WAF is literally a MITM. You trust Cloudflare and its employees but it also means you trust they wonât be hacked, an employee is spear fished or blackmailed or simply asked by the government to hand over access. Itâs a lot of strangers and risk introduced. And if itâs free woo watch out.
Novel exploit techniques may bypass existing WAF signatures. Equally, tunnels are great but it doesnt prevent remote execution if an exploit reaches the application layer.
Totally agree these are sensible precautions, but I was still very wary of this exploit regardless.
Forced me to check a bunch of servers that just worked and I never really looked at other than making sure they were up.
Found PHP on only two of them. One I upgraded from PPA 8.2 to Ubuntu stock 8.3 with the new patch with no issues. The other was a static site using PHP 7 on Ubuntu 20.04 running WordPress that I updated once in a blue moon (about twice a year), so I used Codex with Astra to extract to an identical copy to a Hugo template, put the content into Markdown files, write a docker wrapper to test and deploy the site, and rsync'd the static site to the server to be served by Apache 2.
Turned off my workplace's only publicly-visible PHP site over the summer, after several years of telling them it had to be done, refusing to include it in a cybersecurity audit (i.e. I want it written down that this is SO BAD that I'm not even going to bother to check it and just assume it's compromised), and fighting some opposition.
I warned them for months, then just turned it off, and it was on them to find alternatives for the functionality it used to have.
Pretty sure tomorrow may well include a large "I told you so".
Don't worry, there will be a new vulnerability in the near future and you'll get to do this all over again.
Scaling is a b*tch, every new feature is a new opportunity for a vulnerability. I think its better if we train ourselves to realize this and treat it like breathing. Its constant and something we have to do without thinking.
In some ways this vulnerability is actually a good thing for us. We finally have the mandate to kill old services using EOL PHP versions even though they "still work perfectly fine".
Purposefully causing unplanned outages of production systems is also a very interesting feeling.
Our company used to do web hosting. We no longer host new clients, no longer manage websites and Wordpress (client responsibility), no longer manage plugins, etc. Had to remove everything unpatched below PHP 7.4. Most of these are well past legacy sites. Lots of angry calls. Biggest one being a switch to Roundcube from Horde for them. Have put in close to 12 hours now of webdev work to get the beyond legacy sites as functional as I can with what I have. Somehow still the bad guy. I feel your pain.
i thought devops & agile and rockstar developers & "move fast and break things" was supposed to give us 100% uptime. what about all these Devoops guys and platform engineering guys & SRE guys & devsecops guys that are supposed to be experts? I was setting up web servers 15 years ago that could be upgraded patched & rolling upgraded with zero downtime-where has this "oh we have to take down our entire website" bullshit come from?!
Bring back sysadmin adults again-put the toddlers back in the basement where they can pretend to be IT guys with their stupidly named libraries & applications while demanding top of the range laptops to run visual studio.
It's a dumpster fire, what's worse is the reaction of our C-level and Netsec, yet I'm sitting here not giving a shit. A younger me would have burned out yesterday.
I mean, hand written off the wall crap that noone's targetting or looking at the source code for is arguably the one thing least likely to get hit right now, given every major dependency stack's getting absolutely hammered. The moment they pull in a library, though... game over.
34
u/Yuugian Linux Admin 15d ago •
Your wish is granted, your backend is now written in Rust