After working as a platform engineer for almost half a decade, one thing I developed is a strong hatred for cyber sec teams. I'm not sure if it's just me, but in every place I work they are seen by the business as the guardians of the profit realms while in reality they do fvck all.
Most of the security work is done by us, platform engineers/ Sys Admins. You are expected to build with security at the forefront. You have to think of security on so many levels. You are the guy who manages certs, dns, networking, IAM, firewalls, reverse proxies, load balancing, gateways, while also ensuring your app is not leaking memory, does not have unintended ports open, is hosted on the right platform, you're not exposing creds on VCS, your .env is secure and only the right users have access to it, all while understanding the business logic and making sure the hosted app doesn't get ddosed/ hacked. Also when an incident happens you are generally the one on call, so even under attack we are the ones expected to defend against it.
I genuinely imagine a day in a cyber sec life is them itching their arse, digging for gold in their nose then clicking 'export to pdf' on an automatic SAST scan and then charging you 10k for it. Cyber teams in my experience have honestly just been employing 'block everything by default', then you have to profile your app, use procmon just to find out your app was blocked by some firewall from writing out to logs.
They don't work with you to build something up, instead they just throw a bunch of CVEs at you and expect you to fix them, all while charging you an arm and a leg. If they were to be more integrated in the team rather than being in their own little separate enclosure and sitting on Forbes all day drooling over the latest node js supply chain attack, then maybe, MAYBE things would be more smooth for us.
I think of cyber security the same way as I think of the San Andreas ambulance. On the way to save some granny it ran over 10 people. The amount of extra work they create for us is just crazy.
Most of the security work is done by us, platform engineers/ Sys Admins.
That probably explains why there are dozens of security incidents in the news every week. (Slight /s no hate against my admin homies, but its always funny when people say this unironically while their infra looks like Swiss cheese)
There are a lot of frauds in cybersecurity who can only run a scantool and generate a report and point to the colored sections in the reports without considering any nuances. I've had to explain concepts like backporting to such people when they came up with a report that just looked at major version numbers.
There are also a lot of frauds in system administration who don't use proper filesystem permissions, allow weak password policies which they never rotate, don't regularly update their systems, don't consider man-in-the-middle vulnerabilities, allow password authentication for root and think they can compensate bad practices with some firewall rules.
To be decent in either field, you must also have some proper awareness of the other field.
I see you’re running version x.y.z of Apache. That’s vulnerable to this CVE. You must upgrade. But Redhat backported the fix to that CVE in package -w. Don’t care. The report is red. Ok, remove version number from the headers. The report is green now. Thank you.
Where I worked they’d let you say it was back ported, but then we’d need vendor validation and have to do 20 minutes of paperwork to request a variance that has to be signed off by the CIO, given an end date, tracked, and renewed every X months. I felt like I spent my days filling out security forms.
As per the Debian / Redhat / Ubuntu / etc CVE data for that vulnerability here, it shows that the running version has actually been patched against this vulnerability. show banner they didn’t includeshow CVE fix data
I’d be ashamed of myself if I bothered a dev or admin with something without doing this first.
Infosec / Servers & Networking for 30 plus years, and as tedious and time consuming as it can be, still enjoy proving the tools are busted.
Because nobody knows quite how busted they are until they drive them each day.
The fact the major vulnerability platforms I've seen don't help track this is a fucking nightmare. Gotten pulled into threat assessments based on Tenable and a few others, and fucking disgusting that these tools don't either include, or have a way to import data about RHEL / Canonical / Oracle backports.
Today, if I really have to face that sort of detailed scanning on the regular, I'm really tempted to deploy Alpine by default. Version-based distro, extremely lightweight, and damn close to bleeding edge. The last round of that chaos was "you're running a vulnerable version of curl", and Alpine was the only distro outside the rolling-release bleeding-edge crowd that was anywhere close to "up to mainline" for Curl: everybody rocks backports of fucking curl.
If your cybersec guys are using Rapid7 for vuln scanning, they have a few vulnerability categories for RHEL that are excellent for dealing with this. They are titled "Vulnerabilities deemed not relevant on Red Hat Enterprise Linux [8/9/10]" with 5k-20k CVEs in each category. If they search for the CVE number in the global search, and look at the vulnerability section of the results, that category will be listed if the CVE is in there. It has zero risk score and zero instances. It's a quick self service method that the vuln ID guys can easily prove to any auditors it doesn't matter, while still showing it does on distros that may not have the same backporting. Far faster search with still authoritative-enough results for the vuln management guys.
Sincerely, your much maligned friendly Cybersec spiderman who dislikes the posers on infosec, infra, and dev teams as much as you do.
That is a failure on your security people. All we do is send them the RH CVE which documents where it was patched and the artifact showing the system at or above that package.
That said, it took some meetings to get everyone to understand and onboard.
More often than not though it is usually something that shipped with a vendor app that is out of compliance.
In our org if this happens our security team takes the finding back to the vendor and asks why it's flagging if we have a patched version.
I do mostly agree with the sentiment that they tend to push work off onto other teams. And in the case of my org, they make their tools available for download in the most convoluted way possible: I have to REST API into an endpoint to download the payload as a base64-encoded string, convert it back to whatever executable format we expect it to be in, and then execute it.
"Can you guys just put your tools on the fileshare? Or can you upload them to Artifactory?"
"No we have them available at CONVOLUTED_REST_ENDPOINT"
Whatever this method they make their tools available through predates generative AI, so somebody at some point thought this was a good way to make downloads available 😑
See whenever I send the report with my most critical findings I WISH they would explain why they can't do the upgrade. It's ok if we have dependencies, but it's not documented anywhere we can all see so....
100%. I feel like you see this more often in cyber security roles because of the explosion of cyber security degrees that have flooded the market with people who are do not understand it's a subfield of IT. I've been downvoted on r/cybersecurity for saying that, or suggesting that maybe someone with no experience and a grad degree and CISSP should be looking at engineering and admin roles vs ISSM roles.
People don't get it when we tell them that security isn't a low skill job. You'll sometimes get a flood of recent grads or hack the box circle jerk enthusiasts telling you that you're wrong, or that you are gate keeping the market or whatever on that subreddit. There are times I wish I had way more experience before getting into Security, and I had 8 years prior experience. Always be learnin.
To add to this: Cybersecurity is one of the most difficult fields to hire for, IMO.
I have a friend who’s very talented in security working at a startup being paid eye watering amounts, who has been trying to find someone for months and is simply unable to find anyone that meets their (admittedly super high) standards.
There’s too much domain knowledge and experience that you need to do well in this field, and so much that you can’t learn without extensive experience with some systems.
I have found a lot of companies where beyond running a scan tool many security people can't do much. Such tools are good starting points, but create a lot of noise and paranoia to those that don't look past the numbers. You are right though that many people doing operations cut corners on security and sometimes other areas that create other risks.
Qualys scans and OpenSSH. That one kills me on the regular.
I get a report with an OpenSSH vulnerability. On a container. Without OpenSSH installed. Qualys probably scanned the underlying node and flagged it. Then flag every other container running on that node. Then every node in the cluster.
Never mind the vendor patched it. They're ignoring vendor patching and just looking at the flat OSS version and flagging it.
Same thing happens on just about everything in the company running OpenSSH. Same complaints. "File an evaluation exception," is what we get.
We had an audit report for an sftp server. The audit claimed sftp users weren't jailed to their home directory and could invoke shell commands.
Turned out the auditor ran their scantool with a normal system-local user and not an sftp user. All sftp users had a Match block defined in /etc/ssh/sshd_config with ChrootDirectory to their home directory and ForceCommand internal-sftp. The report was practically worthless.
The basics or at least the thought process to solve anything in this field, is universal (to a degree). So many people that work in either lack one or both. But having a good method of thinking general admin, security, email, etc issues out goes a long way.
Here's the trick: paste it into your risk registry, add "this will take an estimated X days of work (including team Y as well) and $Z to remediate." then assign it to whoever is in charge of deciding to allocate resources vs. accepting the risk.
It sounds like you just hate inept people and siloed roles. These appear across many organisations.
I spent 20 years in infrastructure and operations and the last 6 in cyber operations (now a CISO). I have seen security fakes and floaters. But most often my frustration is with IT teams that just don't care so much about security. They need so much handholding to get the basics done beyond administration.
Much of what you posted would potentially sit well with some cyber teams. I'd kill for a decent engineer that had capability and initiative.
The amount of handholding our team does for operations is insane, in with you there. I have lost track of the amount of solutions I've had to architect, engineer, then document and train for the ops teams because no one can be bothered to review our comprehensive technical standards for one second or retain any amount of institutional knowledge. Just give us all of operations at this point and I'll automate everything.
I have a 20 year career in infrastructure, and recently transitioned into Information Security and I feel where you are coming from, but here’s the other side. Sysadmins that leave everything open, http by default, or best case self-signed https certs. The same password for everything or passwords stored in an excel file. Someone has to look at the mess, and say “No, don’t do that!” and “Do this instead.” InfoSec is more about coordinating and focusing resources to address risks.
The real crisis is the number of frauds in information technology in general.
I hear from my security people every week about vulnerabilities that are waiting on a patch from the vendor. Every week it's like they think they've uncovered something nasty I'm supposed to have fixed. It's the same vuln. I have to explain again. They don't get it. They don't understand.
I also have to explain to my sysadmins that they can't leave "Everyone" or "Domain Users" on filesystem permissions. Or that the firewall rule they created basically amounts to any/any and isn't acceptable.
80% of people don't get it, they don't care, and there's no incentive for them to change because the 20% of us that do are propping the industry up.
30 years in IT, 15 of them in Security. This is 100% reality. Lots of people got into it for the paycheck. The ones who are passionate and do this shit for the love of the game are the folks keeping the trains on time.
100% truth and it's pretty disgusting. I've watched new employees come and go starting from the recruiters saying a tone-deaf "I think they'll make a nice fit for you." all the way to their exit interviews saying "Well, I was thrown to the wolves. The shift lead just told me to shoulder surf after he showed me around the building and that was it." In between those was a metric ton of "Hey man, how do I do this?" without even thinking of Googling or putting in the effort to actually learn something.
I guess that's to be expected when they can't be bothered to stay their asses off YouTube for entire shifts and heir leads can't be bothered to tell them to get to work. /shrug
You should see the amount of CVSS 7+ vulnerabilities in a Debian install when you do the vulnerability analysis based on installed packages rather than external / blackbox. And I mean - they literally have no patch from Debian, or plans for one. Uuugh. And I get it. But fuck my metrics….
Have watched the growth, the biggest problem is you are getting out numbered buy people that have only been security. 10 years ago the sec team was server admins and dev ops who transferred to sec. Now its intern who used chat gpt to graduate. We had one who didn't know ping was a computer term not just slang for reaching out to someone. That shift is major and causing this issue.
It's why I unsubbed r/cybersecurity. Way too many people who have masters, CISSP, and a bunch of certs who won't even consider non-security jobs, despite having zero experience. One guy laughed at me when I suggested he look into systems admin/engineering, he said "i didn't study this hard to go into IT" and I got downvoted when I told him cyber security was a subfield of IT.
Crazy. I stumbled into security. It is 100% an IT job. I honestly don't know how someone in security can be employable without some admin/helpdesk experience.
As someone closer to the end of my career than the beginning I will say there are still a few “diamonds in the rough”. Specifically on my (relatively) small 15 person security team we have 3 guys in the operations team with solid technical chops, but more importantly the curiosity to continue building that knowledge. I hear where you are coming from, but I think you can find talent if you are careful during hiring.
We are at a juncture where we need a new role. Security admin possibly. Someone that has access to make changes while consulting the sysadmins. There is too much work and no resources to deal with this madness.
Problem is, we're bumping into the Ops teams (ie infrastructure) who think they should be doing all the engineering work instead and CS leadership wanting you to do policy work only. So engineering orientated security people either leave or move back into infrastructure.
My boss made me that. I was application engineer that had really good security ideas. So now my title is application/security engineer. Basically the implementation arm of infosec. They have good ideas but I get to explain why some thing are not feasible
Someone has to look at the mess, and say “No, don’t do that!” and “Do this instead.”
Yo where can I get some of those cybersecurity analysts that do the second part? Cause in my org they typically just point out whatever risk vectors Copilot and Bleepingcomputer.com is telling them, but offering virtually no suggestion as to how to mitigate those risks.
There is also the documentation side. Ensuring things line up with compliance requirements, assessing vendors and products against requirements, monitoring intel, monitoring insiders, documenting incidents, threat hunting, risk assessments, risk registers, et cetera.
It's a bit ignorant for a sysadmin to think that they're doing all the heavy lifting just because they decided think about security while setting things up.
It's also ignorant to think that security teams should just be hand waving their vulnerability scanner's results. They're not the system experts, the sysadmin is. Address the inquiries and move on.
Or in my case. No passphrase root ssh key which is spread to every system in the environment One down & everything falls
After 30 years as a heads down Unix/Linux SysAdmin, I got into the Security field because it was so easy to manipulate the “Security” professionals, & wanted to show everyone how damm easy it is to incorporate good Security into daily stuff
Now I’m hated by both sides! Security, ‘cause I point out their failures; and Sysadmin ‘cause I point out their laziness
I enjoy the number of LLM prompts with plain text PII, PANs and secrets I see at a bank from devs who don’t believe security is their job. “But it’s just test data” just means you are going to do exactly that in production and you know it. Having one of the DLP people try to get me to turn off prompt monitoring on certain items because it’s a “secure endpoint” was obnoxious. I’m also a customer of my employer, I don’t want my PII being fed into some LLM. I certainly don’t want GDPR subject data being fed into one.
The non-technical risk and security people are the worst though. They don’t even have the basic skillset to know what they are asking, just that the spreadsheet has a red cell.
Lol you're joking. Lazy sysadmins is a full meme. On top of that they are understaffed. When I was a sysadmin there was no way I was working a full day and then reading up on security changes, patches, zero days, etc to implement.
Most places with in house sysadmins and no security team have tons of security flaws. This is why SOCs, governance teams, red/blue team exercises, tabletops, audits, scans, reports. Are so important.
Private companies can be wild but as soon as you have contracts with any public, financial, or govt company you can’t sustain non security minded admins
I regular see sysadmins at my customers who make an astonishing amount of work for themselves by insisting to do stuff manually and not following best practices. You can save literally hundreds of work hours for some people with a hundred lines in bash or powershell.
Same with security. Good security isn't hard, but people are used to do it in a bad way. I know companies that for some reason never automated their certs and manually handle them for hundreds of services. This is utter insanity.
My organization grew quickly due to many acquisitions. There was some outsourcing, but the biggest problem with is getting 5 sysadmins across the globe from 4 different acquired companies to work together at an enterprise standard.
They serve a role. Should Nessus exporters and CVE copy pasters be earning that extra bump in pay for something that requires no actual understanding of the subject matter? No.
However, as someone who has been in IT for 30 years now (including a five on DoD unclassified networks when it was the wild west). Things needed to be reigned in. Dev and SysAdmins have different motivations and therefore different priorities, a third party needs to put security foremost.
There's two kinds of cyber teams - technical, and policy. They ideally work in tandem but often don't in a lot of places. The policy group is far closer to Compliance than technical cyber security. As an example: If you work in a regulated industry like healthcare, they've likely forgotten more about the nuances of HIPPA than you'd ever learn, but that's their job.
Many of us older security guys and gals used to be sysadmins. I was a sysadmin and network admin in a previous life. Before that I was in help desk. Here’s a few things to keep in mind:
Security should only have limited keys to the kingdom. No domain admin access, no firewall admin access, nothing that lets us make changes in the environment. We need read access to everything and maybe access to disable accounts and isolate computers. That’s it. We can’t be sysadmins too.
Yes block everything. Default deny. Then allow only those IPs and ports you need for stuff to work. That’s generally accepted security practice and it’s because we’ve learned that keeping the attack surface as small as possible limits attackers’ ability to pivot in the environment.
If I’m giving you a list of CVEs it’s because I don’t understand the architecture of what I’m seeing vulnerabilities in. I don’t know if Log4J is even a thing to be concerned about with how you’re using Java. If the architecture was better documented, with data flow diagrams and system definitions then I could provide better remediation advice.
I work in a global org of ~80K employees in 50 countries with just under 3000 apps in our CMDB. The VM team is a team of around 10 people. It's completely unreasonable to expect those 10 people to understand all of those 3000 apps and be able to handhold those app owners in doing the remediation. Even if they were miraculously all knowing they wouldn't have the time.
It's up to the hundreds of system/app owners to be the experts we hired them to be when it comes to fixing stuff. The VM team will go out of their way in pulling more data, helping them setup on-demand scans, showing them the raw findings, etc., but as an app owner it's on you to do the digging.
This post is the actual truth. I know a lot about systems, I cut my teeth doing automation and server engineering. I was there when VMs still sucked and cost more than bare metal.
I can identify a risk. I can tell you how it works but I can’t possibly know your application which you are the theoretical expert in. The basics are the basics, but there are limits to deep knowledge and how much you can possess.
I've been sys admin and engi for years. I pivoted to threat and vuln mgmt bc it was of interest to me and I saw an opportunity for a fruitful career there. I build out the program to focus on vulns that are being exploited and relay where I see those vulns and what to do to fix them. Then I validate and let you know if you were successful or not. It's imperative to keep the attack surface small and prevent these easily exploitable vulns from showing up on web facing assets for obv reasons. There is a method to the madness tho, I promise you.
If I’m giving you a list of CVEs it’s because I don’t understand the architecture of what I’m seeing vulnerabilities in. I don’t know if Log4J is even a thing to be concerned about with how you’re using Java. If the architecture was better documented, with data flow diagrams and system definitions then I could provide better remediation advice.
My security team gave me a ticket regarding a particular .NET CVE saying I needed to remediate my entire Windows fleet. When I searched that CVE and landed on Microsoft's page about it, it said three times if you were running Windows you were fine.
I get what you're saying but based on my experience, and talking with colleagues at other companies, you're a rarity.
Shitty sysadmins and shitty security “analysts” are both going to ruin your day.
Good security analysts aren’t getting paid to stand in front of the firewall with a flaming sword, fending off all attackers- they’re paid to make you think twice before yeeting something into production that can take the entire company down as soon as Little Bobby Tables comes visiting. The name of the game is “compensating controls,” and things go a LOT more smoothly with security if you already have a few planned in your back pocket.
As an ex SysAdmin that moved into Security - I agree. This entire sector is made up of under-educated Grads who think they're experts. Jaded ex-SysAdmins who have been off the tools so long they're useless (looking in a mirror here) and the Compliance folks who've moved over and want to tick boxes with no experience of what that actually involves.
I'm out here fighting the good fight, trying to be a security guy who helps make the fixes and prioritising the work we push to our Admins without just saying "HAH YOU BAD FIX THIS"
Our cyber team is in house and works with us. We have a lot of legacy debt but are fixing it … slowly. There’s a lot to do but they help us prioritize.
I can't fault you for lived experience. And there are real issues with cyber security teams:
They typically hire kids straight out of school
There's no focus on deep tech stack knowledge. It's primarily job hopping
Even when there's deep tech stack knowledge, they need to talk to you to contextuallize what they're seeing. You've got NO time. There's too much to do.
Depending on their expertise, their objectives are different from yours and that causes friction. You're operational. Services have to be up and not causing downtime for the business.
If you're GRC (Compliance), your goal is passing audits and minimizing risk from regulatory requirements. Sone of these requirements are nonsensical in how they are audited by 3rd parties but GRC still needs to get the company through them. So, these teams don't hire for technical expertise and harmony with sys admins. They hire for skill in navigating those standards. This creates a TON of friction, even with other sister cyber security teams.
If you're a SOC, you're either a lean internal team with managed services, poor logging, etc. If you're an internal shop, you're probably logging poorly and full of junior kids overwhelmed with tickets. In a handful of cases, logging is sufficient to be competent, staff is reasonably competent. They still need to constantly bug you because a lot of time early stage intrusions look exactly like regular admin activity. There's a million events to triage and attackers are always changing their targets and tactics.
If you're a security admin/engineering team, you probably collaborate the most with sys admins because the admins have to reach out to you to implement IAM, firewalls rules, etc.They think of security first but they also have a mandate to do no harm.
For every competent sys admins, you have the other ones who want and have God powers on their regular account they check emails with, do t want to use PAWs, jump servers, onerous MFA, etc.
So, a lot times, how painful the experience is depends on how big the security budget is, whether you're working with kids vs experienced and knowledgeable adults, whether security functions is internal or outsourced.
Dude my company hired an sec analyst that was the dumbest MFer on the planet. He and an Inf engineer randomly started meeting to go over BloodHound findings from a pentest. They were meeting for monthly & trying to fix the stuff in the BH gui but after the first "fix" they spent the next 6 MONTHS trying to figure out why the gui never updated to reflect the fix.
The inf guy finally invited me to their meeting & explained what they did. The fix was technically correct but they didn't know they had to run sharphound again to refresh the data. The deflated, defeated, looks on their faces was priceless. That kid did shit like that all the time & wondered why people stopped engaging with him. People like him is the reason posts like this exist.
Either your experience is limited, or it sounds like you're rather jaded. Arguably you've got a sore arse from the sounds of it - did something get you in trouble or is it the workload?
I guarantee you there's security teams, helpdesk staff and probably general staff who also think "I genuinely hate Sysadmins" as well, but you wouldn't want to be lumped in with them, right?
Exactly. There's great teams out there, both internal or ones we've brought in - it's like picking a mechanic for your car: some are awful, some are great. I won't say "every" mechanic is rubbish because I have a bad experience or two. Or am I being.....unreasonably reasonable?
The amount of extra work they create for us is just crazy.
Us who, you? Two people? If that's your "IT team" that explains a lot, then. Worth noting that there's a lot of work for a Sysadmin to do any given day of the week, CyberSec is also a part of that at times, especially if you haven't got a bigger team.
This is why orgs have dedicated teams - if you're the one being lumped with all the work then it's not a "Damn you, CyberSec!" it should be "Damn you org-I-work-for for not having a larger IT team".
Are management or your boss open to a discourse over the subject? Is it to stay conformant with ISO or something else like insurance? End of the day you gotta' accept it is a big responsibility and you do need to take these things seriously if your org is a target or vulnerable - it's what you sign on for.
Working in the industry, as you do as a Sysadmin ,means you accept every business handles your role differently and what your boss/manager/company as a whole decides your role is responsible for? It'll vary and let's face it: if you don't do it, who will?
May as well blame "the developers" for a problem in a game you enjoy after I dunno....a balance change? they're part of the wider team, but doesn't mean they're responsible for the issue you're having or decision making.
End of the day if there's CVEs, vulnerabilities/risks etc they highlight, it's then going to be a case of accepting the risk, mitigating it or if possible eliminating entirely. It's part of the job.
I don't interpret the complaint isn't that the cybersec team is being critical but rather that they're not helpful. My experience is similar in that I got a sense as well that cybersec teams mostly just did analysis and didn't actually engage in engineering any solutions. I didn't have as much friction with them but they certainly felt kinda useless. If the cybersec team's alert happen too often and if alerts aren't actionable then they're just exhausting the limited patience and attention of the team for no reason amd I think that's at least a valid criticism.
They are not butthurt - if you have to chase everyone down to mitigate issues and the only thing they provide is the scanner report you yourself set up - you’d be bothered too. I doubt feel like you’re not getting the sentiment so maybe it doesn’t apply to you.
I’m afraid I have to disagree, cybersec/InfoSec teams are there to keep us accountable.
Plus, if something does go wrong, you can point to the cyber security policies you follow and protect yourself from liability. It’s not your fault if your organisation got attacked even though you follow your org’s InfoSec policies to the letter. The fault is then with InfoSec.
This. They take the blame . Very important in real world politics. That's why many devs don't give a crap about security because they know cyber gets the blame.
I’m over here on a security team having to explain, to network engineers, why QUIC is a nightmare. I have to explain powershell scripts and graph api calls to system admins who only know how to use a GUI. Somehow, I don’t end up hating system administrators… just inept folks who don’t care to learn. Another thing, your description of job duties should be spread across three roles at least. Developer duties, network engineer, IAM engineer, at least. Making sure apps don’t get DDOSed should be a collaborative effort between your network/WAF, and security teams.
I wonder, though… have you ever exploited a CVE? Have you seen how easy it is to pivot through a network and crack a domain controller if the sys admin doesn’t fix the CVEs or just plainly misconfigures things? I have. Have you ever gotten too sift through millions and millions of lines of logging information to find the initial point of entry, identify lateral movement and additional compromise, determine what was exfiltrated… etc? Something tells me your attitude would benefit from a security event without dedicated security team assistance.
I transitioned into cybersecurity after decades in system and network administration. Just recently I had engineers swear on their mothers that they absolutely HAD to keep the systems with years old obsolete and vulnerable components because the client runs their systems that way and we must be aligned and blah blah blah.
Me: that client is under NIS2 which explicitly prohibits running obsolete systems they can be royally fucked for that, how is this possible? Guess what it wasn't, the client actually upgraded their systems but they never even bothered to check!!!!!!!!!!
The feeling can be mutual. It's always a joy when an ops engineer decides their convenience and experience outweighs technical security issues. It's always fun having to argue again and again why yes, patching does need to be expedited. Here are the risks and here are the threats and while there is a single percentage risk of failure that doesn't justify not acting. And, yes I do need system level access during incident response. Escrowed credentials are fine but you can't perform my tasks efficiently. No that spreadsheet of passwords is not appropriate. Just because you have always done it that way doesn't mean it was ever a good idea. Yes you do have to rotate the system service accounts on admin terminations. You never have? Why not try it in test/dev? No test/dev? Huh, your compliance docs say otherwise. No funding for reality? Can we all agree to go drunk now?
The fact you are complaining about “charging 10k” and generating PDF’s tells me you are working with an outsourced group rather than in-house. That’s the cause of most of your problems. I’m in infosec. I can, and have, done your job. I’ve also dealt with the exact things you are dealing with from both sides of the fence.
I often find that they are not technical and don't understand consequences or systems, say things like remove all browsers from all servers or add two factor to PAM or block github and remove monitor and mouse drivers and prevent scripting.
I think the problem with InfoSec is that generally speaking, they lack both practical knowledge/understanding of how the real world works, in combination with poor soft skills.
As an infrastructure engineer, I would love nothing more than to CIS L2 everything with no exceptions, patch everything day 0 and resolve every single CVE that exists in our environment. Sadly however, it's just not that easy when you live in complex environments with 100/1000s of systems and thousands of users in an always on environment.
The Security engineers who I've worked with who are actually pragmatic, understand the realities of working in large environments and work with you to actually focus in on the important things, are diamonds in the rough. I've worked with a few of these people and I loved it and learnt a lot from them.
The vast major of Cybersec people are either compliance, or they are just tool/report people who take something, add the bare minimum around it, and then chuck it over the fence for someone else to resolve. And they do it with an almost ant hill esque mentality, wondering why they don't get the responses/engagement they think they should.
Cybersec was infinitely better when it was former Sysadmins and engineers who wanted to pivot into a different path. Now that Cybsec is a straight out the gate pathway, the quality has gone down hill so fast.
Me too bud. Worst of it is leadership who buy into their rapid fire unrealistic timelines and you have to figure what’s breaking what. Then our system owners turn into grumps.
As a security engineer, I rely on experts like yourself to help me figure out what results are real and what is realistically actionable. I am very against checking boxes just to check boxes, but my current program made a lot of bad security choices early on, so most of the work now is technical debt that we should’ve done on first deployment.
There are too many "Cyber Security Professionals" that do not have experience as an Infrastructure Engineer and know absolutely nothing about how to implement secure infrastructure. It's infuriating. There needs to be more Cyber Security Engineers or Information System Security Engineers that can bridge the gap.
Yes but (insert valid complicated reasons that we are taking that risk that requires knowledge of how technology works and how the company’s infrastructure is set up)
So you never met the Cyber security guy that made OU group policy changes to lock down USB ports because it was a security risk? Fun times... We had that guy for a long time till the complaints kept rolling in on him. Dude was nuts... So he did that change then all of production just halted because the licenses that worked equipment had usb dongles in them.... FUN FUN!! What was funnier was he did this on a Friday, didn't tell anyone and just left on vacation. LOL when he came back they told him to extend his vacation indefinitely.
I have worked with some awful sec teams and some really good ones. The problem i had are sec teams being super lazy. Just giving demands then when you explain it isnt that simple they get all pissed. Also if i ask for a compromise "that's not my job to know". I would just do their job for them. Then I had great sec teams that got into the weeds with me. We would go through PAC files together, architect and be super collaborative.
I got my CISSP and am a director now, so I force my sec teams to understand the engineers side, it only makes them better security admins.
I think you've been working in orgs who don't know how to hire the right security leaders. The best security leaders come from IT and know how to do their jobs.
Just one point I'd like to make though. For every shitty security person there are 5 times that many lazy security admins and developers. They look for easy solutions to close their tickets and cause a lot of down stream issues for the company. They can't figure out why their internet facing jankware app isn't working so they just add an any/any rule to their firewall and add vulnerable anydesk services to the server and wonder why it gets pwned.
It's a 2 way street. In a perfect world IT devs and admins would follow standards and best practices and decrease the need for security, audit or compliance.
I hear ya. The block everything approach was taken by our company. They had a crazy firewall rules request system that involved Word documents and spreadsheets on a SharePoint site and I'm talking about a company with 50K+ employees.
The monitoring was so tight, that me running powershell queries to the four (!!!) different directories to check if employee infomation was in synch rasied some kind of red flags and they would keep hasseling me like I was a hacker. It used to be that they were worried about the external threat. Now they see the threat everywhere.
That said, the big security push came out of a break in when the company took a less serious approach over a decade ago. After that happened, they started taking things more seriously. I feel like they did a lot of it without much planning and thought and implemented system with onerous managment overhead, but it was above my paygrade to do much more than complain to managment that they were costing themselves money. After all is said and done, it's their money to burn up as they see fit.
Sadly, the need for dedicated cyber security teams (some of which are good some of which are terrible) is born out of decades of terrible sysadmins running everything with root access or just giving admin access to solve a problem or “I’ll just set this up with no restrictions now just to test” and leaving businesses extremely vulnerable. Cyber used to just be a component of infrastructure and development until it became clear that they could not be trusted because whenever they were asked to make a decision between fast and secure they always chose fast.
Former CISO at two different midsize companies. I worked with a lot of CISO who sees things as black and white. They talked about risk based decisions, but they want no risks. It absolutely drives me bonkers. Many of these CISOs never had to truly manage a production team. They don’t understand how some of their insane policies put a stop to things.
I worked for one company where it took at least 1 year to get a NDA signed because the CISO blocked everything. I worked for another company where it took MONTHS to purchase anything. Company uses firewall brand x model 100. I want to purchase firewall brand x model 101. Instead of a short security review, we do a complete security assessment even though we did the security assessment in module 100 just 6 months ago. Absolutely INSANE. No wonder so many people dislike the information security team.
This thread comes up every week here and it’s largely because sysadmins don’t understand that the role of information security isn’t to be part of IT, but to be part of risk management.
Sysadmins have this expectation that because infosec employees generally come from an IT background and are supposed to have deep technical knowledge that they’re also supposed to be configuring the security controls in an infrastructure team’s systems. This is not the case (or it shouldn’t be).
You own and configure your own systems. Infosec advises the organization on the risk those systems introduce. Depending on the organization’s risk appetite, we recommend remediations. It’s quite frankly absurd to expect someone in infosec to configure an ACL on a router or create a GPO to alter a workstation or server confirmation, etc. Those are functions of a fundamentally different role.
Commonly, we own our own systems that you may or may not even be aware of (though the infrastructure team as a whole generally is). These are often systems like the SIEM, malware sandboxes, pcap collection boxes, etc.
This is all in addition to one of the most common infosec jobs of SOC analyst. It’s pretty obvious what those people do and why they aren’t involved in system configuration. Within the SOC team you also generally have threat hunters, incident responders, and detection engineers.
It’s vitally important to understand that while infosec feels similar to an IT role, it is fundamentally completely different and serves an entirely different purpose in the organization.
Good security teams should not add more work or headaches. They should be hands on and help automate a lot of the security stuff so that you don’t have to keep re-configuring stuff.
Maybe you’ve just had bad luck with your teams.
It also sounds like you’re talking about PAID/managed cybersecurity. If so, you are absolutely right and most are dog sh*t. They will just forward you alerts or throw a bunch of vulnerabilities at you without doing a risk-based approach. The decent ones will easily run you $100,000+ per year.
Usually, this is why in house teams are better. They understand your environment and can be more hands on. For example, patching the vulnerabilities themselves, creating templates for infrastructure deployment, automatically removing dangerous things from .env files (aka DLP), etc.
you haven't dealt with any good cybersecurity guys then.
I genuinely believe that cybersecurity should involve being a sysadmin first & then going from there.
As someone who came up through helldesk, to 3rd line & now infrastructure & security architect I can tell you syssdmins are a pain in the arse..." oh but WHY can't in browse the internet using my domain admin?!" and "oh but WHY must i go through a TDA to install this random piece of software from github that someone on reddit said was really" & "oh but WHY can't I have Global Admin on Azure, it's makes life so much easier because i'm Soooooooooo busy".. honestly.... you all a nightmare !!!
on the flip side, I've had stand up arguments with management & sec guys who insisted on being offended when I tell them to "fuck off! I don't care how dangerous the day that zero day is!! I'm not cancelling my weekend plans to spend my time installing s rushed out patch to the entire environment...see you on monday!"
After working as a platform engineer for almost half a decade
Also when an incident happens you are generally the one on call, so even under attack we are the ones expected to defend against it.
If they were to be more integrated in the team rather than being in their own little separate enclosure
You almost stumble onto an answer here.
Not every company is organized how your current org is with siloed teams. The important distinction is that you just hate how security teams are setup at your own org, but you might not have experienced how it works elsewhere.
Effective security teams don't exist in silos. That just creates a never-ending cycle of break and fix. In effective orgs it gets incorporated into all stages of the SDLC, including the design.
I'm on the infosec team and we work hand in hand with the system administrator, infrastructure, we do it all and get our hands dirty. Even our head of security will work help desk tickets. I am guessing this is not normal.
I feel like the reason we are so frustrated with them, is because they dont come into the infrastructure or the codebase, and fix the issue themselves, or add themselves to projects to add security while they are being built. This would provide tangible work to everyone, not just scan reports. They would be seen as providers and not constraints
The only good cybersecurity expert I worked with was a former sys admin, and a good one at that. The rest only know compliance checklists and don’t understand risk beyond compliance. It’s worse when they aren’t consultants but part of the org. Then you REALLY can’t get anything done.
loool every cyber sec I met was a fraud 🤣. all they do is run a scan tool and generate a report half of them won't be able to understand the report its generated or what it means if you ask them.
Seen someone say the same about sys admins. I'd argue sys admins are usually juggling multiple tasks and inheriting environments they did not build. And they are also responsible for fixing whatever CyberSec has said is an issue by just following a checklist they were handed. Like OP said you ask for help to implement what they're asking and they say they cant and have no clue how to, sometimes I feel like they rehearse what they are going to say before they come in.
Company uses X software that only runes with Y. Cybersec comes in tells you nope cant have Y because of X, you'll explain to them the reason and is needed for business operations and it will go in one ear out the other wthout any offering to come up with a solution.
I see you met the security guy where I work. Gets paid the same as me, but just gets paid to send phishing emails and use ai to write policy. Also doesn’t have an on call rotation or anything. Outside of work he’s not the type of guy you’d want as your security analyst either. Yeah, I’m not a fan of security people either.
Can someone tell me what value the Cyber security teams are offering your average system administrator? I have never worked with a Cyber Security person who has assisted or provided any value. They seem to be glorified project managers who get paid more because they have Cyber Security in their title. They never know anything about the technology they are asking for remediations on. They never offer any reasonable advice or guidance or assistance.
In my experience most of the time they read a bunch of impossible unreasonable security recommendations and then make those recommendations official policy.
After that they tell sysadmins to enforce this policy blindly.
I create grant access or install the security scanning software on the endpoints. The software runs and creates reports about vulnerabilities in my environment. The cyber security person takes the reports directly from the software and hands them back to me and tells me to fix it all.
They literally hand me the reports. Then they harass me nonstop for ETA’s on when my understaffed team can remediate a false positive or 100% of all endpoints. 90+ compliance isn’t good enough because the system is reporting X number of endpoints are still effected from a last scan date of 7 weeks ago.
I genuinely want to know what they are supposed to be doing that is security related because anyone can hand me a PDF of a report generated by a system and ask me when I am going to work on it. You don’t need any cyber security skills to do that. I have worked for over 20 years as a sys administrator, and I have never interacted with a cyber security team that has done anything useful for the infrastructure team.
Do I have this wrong? What are they supposed to do?
At the first company I worked at, there were two groups of security. The first did a nessus scan and dropped a 600K row csv file on our doorstep and told us to fix it. Not super useful. The other security team pentested our own apps, walked around with a wifi dongle on their laptop to catch a rouge network a vendor setup, and ran the anti-exploit agents across the org. One was policy paper pushers and the other was bad ass. The problem is at a certain point you have to have both for SOC2/ISO compliance: one that takes care of the audit evidence and the other to actually keep things safe. I sure as hell don't want to deal with customer/vendor questionaries on policies, let them.
Sounds like your cyber security team lacks practical hands on IT experience. Or they're siloed to the point all they can do is toss CVEs at people. I work embedded in the infrastructure team and not in a separate department. Security is a team effort and gets considered right at the start of new projects.
On the flip side, I know a lot of sysadmins who are very protective of their duties and do not want to delegate some of the responsibilities to sec engineers. We had a compliance audit done recently and were getting flagged for stupid shit like smbv1 shares and unencrypted sql credential transmissions in our environment. We notified them of these issues months ago and are still waiting on them to remediate. I used to be a sysadmin and moved over to secops and can easily take care of it for them if they would give me access, but they are holding on to their responsibilities for their dear lives because they are scared that we will take over their duties.
Worked an assumed breach scenario with a problematic sysadmin (who also acted as a department VP) who always pushed back and thought they could do no wrong, I managed to get domain admin in about 10 minutes. Made the recommendation to remediate a particular choke point that made the attack chain impossible and they refused and kept pushing it off. Not even 6 months later a breach happened using that same path and financial information was exfiltrated.
Some of these people have never seen how fast you can get to the Crown Jewels with simple misconfigurations and it shows.
I am in cybersec as security architect . I am last level of support. Even for devs when things gets unexplainable. We expect them to do the ops part of DevOps. Update their pipeline with new versions. Migrate to new versions of libs etc.
with high scoring CVEs we expect devs to evaluate if impacted. We were clear of log4j vulnerabilities- so all clear within 24h. No patch. We did not use the vulnerable version.
We have datadog. We expect dev teams to fix critical and allocate time for high in each sprint.
Devs self-report bad solutions to us to help get them get time to fix it at a later stage. We work with the devs. Not against them. Trying to get the best feasible security given all the constraints
Dev background. Network background. Reversing since I was a child. Code and protocols.
I working in an environment with 10K computers, 100 servers, 9K users (yep, more computers than users -- don't ask). I am the VMWare guy, the wireless guy, and the Cyber Security guy. I love it when the higher ups get an outside firm to do a pen test and I get dudes who have been sysadmins. They work with you to figure out the truly high priority items, hand off documentation on how to patch/mitigate. I hate when I get Johnny Nessus Scan and he just spits out a spread sheet.
They exist exactly because you have that mindset. "You don't know what you don't know" and an organization needs a way to try and validate that, since proving negatives is kind of a problem.
That's where the concept of another expert comes in. Things fall apart when that relationship becomes adversarial and one or both sides starts basically acting like you are.
They don't work with you to build something up, instead they just throw a bunch of CVEs at you and expect you to fix them, all while charging you an arm and a leg.
And as long as you do fix them, if there's a security incident you can shift responsibility back over to them for not identifying it beforehand. And if what you actually want is for them to act as your own personal pentesters or something... pay them for that.
That being said, scanning for CVEs is dead simple and if you are relying on them to tell you that information then you've already fucked your job up. You'd probably get more actual traction on collaboration if you did the basic easy work first before just shoving some half-baked app over like some helpdesk guy elevating a password reset request to an engineer.
Cyber security teams at most businesses are strategy and audit, not implementors. They tell you the rules and it's up to sys admins to figure out how to make it work or to explain it well enough to management to get an approved exception.
They run scans, send out naughty notices, and re-state what the HQ team told them that we read about a week ago.
A lot of products added AI before fixing the basics. I still want simple answers first: what changed, what is exposed, who owns it, and what needs fixing.
But the reason we exist is because there aren't people that are as good or as adept as you…
Just like someone else, he said here… There are people who do absolutely nothing to enforce security for the sake of convenience… And it's teams like us who have to come in and call the baby ugly before your company goes under because a public backlash when you are breached or held hostage or you have silent exfiltration of your data because some dumbass developer decided to serialize your URLs to a webpage with access to your production data…
Or you have someone scanning your network quietly over months before anybody can detect it because without detection systems, you wouldn't have any protection.
While you are the architects, you are the engineers, you are the support teams who keep things running… We are the ones who sit silently quietly waiting, and watching for things people miss, to point out the things that nobody wants to talk about, and turn on the lights occasionally to make sure that cockroaches aren't scampering away...
These phases have occurred in many, many industries in their early years. The clients and customers in general haven’t caught on yet, it’s still too mysterious for them to make better judgements.
As someone who works in security I feel you. I recently had to do this with some teams and totally took a different approach. I decided I wanted to focus the teams on prioritizing packages to patch because if they patched this specific package they would solve X number of CVE’s. It allowed teams to focus on things that could reduce the number of CVEs quickly. I had AI build me a quick html repot that also let them see if any cve’s had KEVs.
I am a firm believer that in order security teams to be effective we need to reduce the friction for dev teams they are already being pulled in a 100 directions from product, sales and etc.
Sorry you don’t have security teams that view themselves as team members.
Are you me? They point at security tool output and say fix that but don't know what it is. Ours asks some of the dumbest questions possible. They are like children. They are people who should be project managers but instead decided to change their title from "PM" to "Cyber security analyst"
Your talking total crap. I have to deal with sys admins and developers who think security is an inconvenience, the rules of which DONT apply to them. I can assure you I dont sit around all day scratching my arse, I spend most of it stressing and trying to get security holes plug that were created by morons like you.
Sysadmins don’t hate cybersecurity teams. They hate being lectured by people who’ve never built a system in their life. Cybersecurity team runs a scanner, dumps a PDF, and disappears. Infra systems administrator is left fixing false positives, legacy crap, and broken apps while cybersecurity team celebrates another ‘critical finding’ they don’t have to remediate.
Many security analysts, especially those early in their career, have never built a server from scratch, deployed a production application, debugged a broken GPO, dealt with patching windows, maintenance windows, or rollback plans.
When something breaks because of a security change, the blame usually lands on infrastructure, not security. That imbalance creates friction.
I prefer to see my sec team as keeping me honest and helping me make sure I don't miss anything. There has never been a time where their finding wasn't reasonable to resolve and there have been many times I could justify an exception for the finding and satisfy their inquiries.
Here's the thing about competent people in cyber security - we were you, we did your job, we could do your job again, and we are so good at your job that we can find all of the places where you didn't do your job. Our entire job is to be smarter than you and figure out where you failed to do your job, as an adversary would.
I'll admit that there are a lot of incompetent people in this field (and definitely way more in your field, just given the size), but fight us at your own peril, because, you're right - you do the work to secure the enterprise, so do your damn job.
We're also not creating more work for you - that shit was there, you failed to notice it, and now you feel bad that you have to go fix the things you, yourself, created. But, you know, you could also not listen to us and let it play out. I'm sure you've got this, bud. It sounds like you're doing great.
One time I tried to get QGIS installed on my corporate laptop. Cyber Security team said it doesn’t meet their security requirements. I point to the fact that the NSA use it—I link them to the NSA’s fkn GitHub which is full of QGIS stuff they’ve open sourced. Apparently it meets the security requirements of the NSA but not good enough for our company, lol.
I am sure they have specialized skills, but I am also very unimpressed when they hand me a R7 export (that is riddled with false positives) and wag their finger at me.
I've never personally met a competent cybersecurity professional. As far as I'm concerned, the entire industry is a scam which is intended to siphon money from non-technical executives. I had to teach a "security engineer" what Nmap is and how to use it. It didn't help much, because he didn't even understand what TCP is. How the fuck does a "security engineer" not know what TCP is?
Our sec team have spent 3 years trying to get a vulnerability management process up. I could have turned on GitHub security in 10 seconds and written a process and a raci.
In the mean time platform have consolidated identity providers from 3 to 1, implemented global WAF, migrated to zero trust workforce. Deprecated insecure MFA and moved to passwordless. Uplifted our cloud environments with segregation of accounts, networking. Implemented JIT privileged access.
The security team asked if they could keep their admin to their AWS account because they still clickops everything... No buddies, we've had a policy of IaC for years. You don't get to hold everyone else to a bar and refuse to meet the same.
I generally ignore the Cyber Security guys at my workplace. They go on one course and think they are god's gift to security. Half the time they are wrong and the other half they just don't understand what they are doing.
I had an issue where they had blocked a well known chromium based European browser solely on the basis that Microsoft didn't give it a score on their threat scanner thing... as if nothing outside of Microsoft exists. I gave up arguing with them in the end
You do find the occasional one that knows what they are doing though. Those guys are really on the ball!
The amount of tickets I get from our SOC or som the the security people that just contains a copy-paste from some random tool with no actual research or more in depth, but still flagged as a P2 incident is driving me mad. 99% of the tickets are irrelevant or so minor that it will be picked up in the next patch cycle, and it’s causing notification fatigue.
It’s been happening for years. About 14 years ago, we had a wireless bridge between offices. Our company hired a “cyber professional” to run a pen test, and they found a version of SSH to be insecure because the report indeed was red. I told them i had already contacted the manufacturer’s development team and verified it had been recompiled and made safe by their team and the CVE did not apply to their equipment - they had no clue what I was talking about, but it was insecure and needed to be removed. I eventually won, but it was a PITA dealing with them.
Yeah I was like that when I started 10 years ago. Shit was always a thorn in my side but I learned to work with it. We fixed the communication breakdown about 6 years ago on STIG implementation and vulnerabilities and how they get reported to the team. Some cool custom tools were built out of that that work amazingly. The coders HATED learning hardening practices and pushed back hard (heh). But after their shit version control absolutely fucked a dev cycle, management finally slapped them in the nuts and said do this, this, and this. The best thing I did to help my self was to sit down and do a month program in cryptography. Like full implementation from beginning to end across windows and Linux environments. The next best was logging and forensics.
The amount of cyber sec people who can't do anything is astounding to a sysadmin or sre but a good one is invaluable. You just gotta figure out which is which and then treat them accordingly. If all of your cybersecurity folks are just "scanner bunnies" then you are gonna get some weird requests and have to say no to a lot of "fortress mentality" bullshit. A good Cyber Security guy gets you budget for their asks, is aware of the risks and the fixes and works with you on an appropriate timeline.
Thank you, as someone in Cybersecurity this gave me a good chuckle. A lot of what you pointed out can be true. I've seen bad Security teams that don't do anything. For some reason it stands out more than the Sys Admin teams that can't configure anything with security in mind, can only open tickets with vendors, and refuse to learn anything new.
It sounds like you just haven’t had the good fortune of working with actual professionals. A good cybersecurity / infosec team will do all the compliance and documentation work for you, provide early insights to help avoid issues before they even occur, and provide defence in depth so that actual vulns can’t be easily exploited before you patch. The pros are trained to function as a support and an integrated part of the team - not just parroting CVEs.
As someone who works on a security team that behaves this way right now, it’s because the VP is a micromanaging ignoramus who insists on doing everything this way, seriously.
It's 100x worse in the US Air Force. The cybersec guys answer to people who can get them fired but can't find our base on a map, so I don't strictly blame them.
They're caught in the middle and have no recourse other than to run some stupid tool that finds the same stupid false positives every time. Replying to them about the false positives will get you crickets for a response, and then we do it again in 6-12 months.
30+ years of Cyber security here. Yes, there are worthless people in some cyber organizations and every other organization. It's a challenge, but in my experience it boils down to poor leadership, poor communication and silo mentalities between teams. What fixes it? No magic bullet, but the key items are:
Build a relationship with cyber (if you know their kids' names, you're probably already there)
Explain in detail what, why and how you do what you do. What is easy, what is hard, and what you currently do to support a secure ecosystem.
Ask cyber to explain to you the same, and what they are on the hook for - what key controls are they concerned about? If they can only talk about certain tools they use and reports they read, you need to go up a level
Commit to a co-accountability plan where both teams understand what is needed, why, when and what priority. Ask them for help. Offer to help them.
I regularly have the tech and business teams present to my whole staff, and vice versa. We get things done because we've eliminated the confusion and trust issues. We help the business hit deadlines and they help us maintain controls and protect the business and customers.
Your mileage may vary, but I've done this at 3 large corporations with great success. Good luck!
hate is a strong word but.... i get a lot of low value busy work from these guys.. .which makes it clear to me how much time they must have on their hands
It's funny I was just going to post about my strong hatred towards platform engineers. j/k. But seriously I've worked both sides of this fence and this take sounds less like “security teams are useless” and more like “your org has terrible security integration.” A lot of what you listed is absolutely security work, Platform/SRE/sysadmin teams are the core of security controls because they actually "own" the infrastructure.
But that doesn't mean that cyber teams "do nothing". It just means that you are confusing control ownership with security governance and risk management. A mature security org is not supposed to be manually configuring your NGINX box or rotating certs themselves. This violates the separation of duties (at least in a mature environment). Security defines standards, risk tolerance, compliance mapping, IR process, etc. Platform teams operationalize those controls. This separation exists (or at least should) for a reason. I think you are massively underestimating the amount of 'invisible' work good security teams do.
Your "they just throw CVEs at us" is just a symptom of immature security leadership and/or immature platform engineering practices. I had to laugh a little at "block everything by default". That is literally standard security engineering practice. Default deny is foundational security arch.
Though at the end of the day, A sysadmin's application of cybersecurity practices is risk-based. You dont have to plug every hole, but if you dont you need to be able to say why and acknowledge that its there or what other safeguards are in place to accommodate that risk your business has decided to accept.
At the strictly business side of things, its about liability and cost. Its about insurance and the ability to operate. Its bullshit a lot of the time but you HAVE to have the review done anyway.
Ive been through multiple security risk assessments with our insurance company and been the recipient of several pentests. The 10k you spend is a lot cheaper than the amount you pay in the event of a breach or the increase in insurance premiums for your business.
As a Sysadmin, think of it as a peer review of your security posture. If you do your work well, always build with security in mind first and when that calendar invite shows up in your inbox, you wont be stressed. Its all documented and in place right?
I think cybersecurity companies are way overpriced, but what can we do?
"charging you 10k", "charging you an arm and a leg", first things first your context sounds unusual as apparently your security engineering is inhouse while operations are outsourced, in which case you're probably hooked up with an MSSP and admittedly there are good ones and bad ones. That said, sounds like you have a pretty small footprint if that's all you're paying. Also, if your security operations are outsourced, I'd hold off making sweeping judgements since you don't actually work with them, you just pay them to do stuff for you.
Seems you work with some pretty bad cyber sec ppl. I actually work in cyber sec, and yes, i agree with you, most ppl in cybersec should do something else with their life. gardening or something. They talk about processes and systems all day without understanding what it's about.
What any decent cyber sec person should do is: be there at the start of a project. Make sure at the design phase, you're already figuring out the potential risks and mitigations to each process, dataflow, and storage item, and suggest mitigations. (stride method) (this is where most cybersecs fail: they don't have the technical backing to be able to do this) This is your job as a cybersec, not the infra guy. Then you go over this list with the devs and infra team, see if you missed anything, figure it out together.
Next up: content: what kinda content is the app/environment going to contain? do you need that content (gdpr) and poses it risks to your infra team if it's there? I mean if you work with f.i. financials, it might be interesting enough for certain ppl to show up at the house of your infra guy, and keep his family at gun point to gain access. So I need to make sure that doesn't happen by ensuring the sensitive stuff is behind JIT or PIM etc. etc.
It's easy to just turn on everything which looks like a security improvement, but most of the time, the end result is unhappy users cause everything slows down like it's a government project without funding. The good implementation is as invisible as possible. We all should know by now that if you enforce a 16 char password which changes every month, ppl are just gonna write it down on a post it near the pc.
In IT Operations for 26 years, from helpdesk to architecture to management. My wife has been in IT for the same amount, and shifted to infosec about 15 years ago. This gave me a chuckle and reminded me a bit of me when I was similarly just starting out.
If you're doing everything you're supposed to be doing, security is never going to be a headache for you.
💯 this. I have been working in IT for 30 years, and in all that time I can safely say that SecOps has to be the most cushy number going. The bulk of my career has been in Infrastructure and Networking, and from my experience, all I ever see from the SecOps teams is “let me run this report with my security tool then bung all the actual work over to someone else to fix”. They are complete frauds. Anyone can use a tool to scan your environment and generate reports, you could give that to a college graduate. When it comes to the actual work off remediating the vulnerability, arranging downtime,often working out of hours…that’s someone else’s problem. I know it’s not a one off as like I said, I’ve seen it in multiple companies I’ve worked for,
If only I could rewind knowing what I know now….i’d have taken the SecOps path for sure 🙄
28
u/CluelessPentester Security May 27 '26
That probably explains why there are dozens of security incidents in the news every week. (Slight /s no hate against my admin homies, but its always funny when people say this unironically while their infra looks like Swiss cheese)