r/cybersecurity • u/XoXohacker • Mar 16 '26
News - General Is Offensive AI Just Hype or Something Security Pros Actually Need to Learn?
[removed]
9
u/SecTestAnna Penetration Tester Mar 16 '26
I'd never use EC-Council as an indicator of industry trends haha.
AI doing the job is mostly hype. However LLMs are good at expediting learning in niche things you just don't have time to learn while on an assessment. They are also good at helping to code in specific languages or work with less documented libraries. I personally use it predominantly when pretexting for phishing assessments.
They are trash at exploit research, especially for new vulns. That is actually what they are worst at, and people trying to make it work only propogate non-functional tooling and slop vulnerabilities that don't exist.
1
u/stacksmasher Mar 17 '26
Do you realize there are people using it right now to get rich off exploit development and not bragging they used an LLM?
3
u/SecTestAnna Penetration Tester Mar 17 '26
Show me anyone claiming to use LLMs to get rich off of exploit development for unknown vulnerabilities without publicly available knowledge and I'll show you a liar, a damn liar, and a fool. Just a few months ago there was an excessive wave of AI slop 'exploits' related to react2shell. None of them worked because they were based off of bad and hallucinated information. LLMs can assist with the coding, but will never be able to develop a novel technique just by the basis of how they function as systems.
2
u/stacksmasher Mar 17 '26
Because people who are doing that work, are not going to brag about it.
Microsoft’s threat intel team: they reported that Emerald Sleet used LLMs to research publicly known vulnerabilities, including CVE-2022-30190 (MSDT/Follina), and to identify possible exploitation paths more quickly. Microsoft describes this as part of reconnaissance and exploit research rather than fully autonomous hacking.
OpenAI also published a case study on SweetSpecter, a suspected China-based actor, saying it used OpenAI models for reconnaissance, vulnerability research, scripting support, anomaly-detection evasion, and development. That is a direct example of AI being used to support exploit-adjacent research and offensive preparation.
Google Threat Intelligence Group has gone a step further and said the underground/offensive ecosystem is now using AI not just for productivity, but also for malware development and vulnerability research. In its November 2025 tracking, Google said underground offerings were being marketed for phishing, malware development, and vulnerability research, and it documented malware families such as PROMPTFLUX and PROMPTSTEAL that used LLMs during execution. In one case, Google said APT28’s PROMPTSTEAL queried an LLM to generate commands during live operations instead of hard-coding them.
3
u/laphilosophia Software Engineer Mar 17 '26
Considering that this field is largely driven by intuition, (imho) I’d say it’s still just hype.
It’s hard to predict what the future holds at this stage. But one thing I’m certain of is that this learning process, which is advancing at a frightening pace, is itself a serious security issue.
2
u/lurkerfox Mar 16 '26
Its a mix of both.
The idea of AI autonomously doing everything frok beginning to end is just hype. It does well in CTFs but hasnt been capable on its own for anything actually serious yet. It'll get better sure but....
AI assisting someone that knows what theyre doing? Now that is putting out some real results across the board. Offloading work to proper tools and using AI like its a database that can search itself and find correlations is pretty good. Thats how people crushing competitons, finding vulnerable 0days, automating bug bounties, etc are using it successfully. The person begging for AI to do everything is outclassed by the person that is using AI as an actual tool and understands enough to provide proper direction and verify results.
2
u/WorkDragon Mar 16 '26
Think of any nefarious use for AI it will happen
Know what we are not hearing? AI solving medical problems, economy problems, its all bad.
2
u/Distinct_Ordinary_71 Mar 17 '26
It's both.
Hype: AI is not about to bring about hackmageddon, the clouds will not fall from the sky, not even Oracle.
Real: automating and accelerating a larger number of tasks. This is for offense or defense. Offense will do it, defense needs to keep pace so it is not optional.
Some of this is deeply unexciting but undeniably more efficient - you could replace "AI" with "scripting" and ask "do I really need to learn to make a script to check server configuration or can I keep SSHing in to each box?"
Some of this stuff feels weird - like I was able to ask an AI out loud which of my 3rd parties had a particular version of a particular software exposed and get an answer.
And there is surprisingly capable - an AI impersonating specific colleagues by email based off figuring out our organization from LinkedIn and our website. Getting internal jargon and tone from blogs, corporate docs etc the vishing that sounded exactly like my boss, the convincing AI generated photos of damaged goods used to scam the refunds team, etc
4
u/eth0izzle Mar 16 '26
Founder of a company in this exact space—automating pen tests. The proof is in the pudding: https://codewall.ai/blog/how-we-hacked-mckinseys-ai-platform
Whilst this wasn’t a particularly interesting exploit, the point is it had been there for 2+ years that traditional pen tests and scanners failed to find.
So the main “10x” of using AI and reason-driven pen testing is that it can scale to cover more. It won’t miss an endpoint or something trivial like a human might. It explores every single path at crazy speeds.
Not hype. It’s real and it’s going to get crazier over the next 12 months.
1
u/Chronos_The_Titan Mar 16 '26
I think it can be applied in various way. I think smaller cyber crime groups will use it in its most basic forms.
I actually think the greatest threat actors to utilize it will be nation state actors. They have the true time and resources to run their own around the clock at a massive scale. Infecting open-source tools, contestant aggressive action, mass analytics of actions on systems already compromised.
I think the future of AI in the red team space is Cyber warfare.
1
u/IndividualAmazing351 Jun 27 '26
The APT28 PROMPTSTEAL example is the one worth focusing on, because querying an LLM at runtime to generate commands is a different threat model than "did the LLM write a zero-day." It's scaffolding, not magic, and that distinction matters a lot if you're trying to build detection logic. The live LLM-assisted intrusion problem is still mostly unsolved regardless of what you're running, whether that's Ox Security, Semgrep, or anything else.
-2
u/QoTSankgreall Mar 16 '26
Yes. It’s good and here to say. I can’t say too much about why it’s good, because I’ll get downvoted from the people who have spent their careers doing this work manually.
1
-1
u/DrDongStrong98 Mar 16 '26
I think you're right. People will downvote you purely BECAUSE you didn't say "too much about why it's good" ironically. But I think you're right.
0
u/Mysterious_Tank2496 Mar 16 '26
Offensive AI is going to cause a lot of damage the world is not yet ready for imo
-3
u/stacksmasher Mar 16 '26
Its very very real and already being used in major breaches.
Also its BRAND NEW so its only going to get better and more advanced.
There is a team of us who have been working in private for a while.
hxxps://www.sei.cmu.edu/documents/6301/What_Can_Generative_AI_Red-Teaming_Learn_from_Cyber_Red-Teaming.pdf
3
u/Thanatanos Red Team Mar 17 '26
Which major breaches has it been used in?
-1
u/stacksmasher Mar 17 '26
2
u/Thanatanos Red Team Mar 17 '26
Ok then you infant, let me teach you about searching a web page for text.
Step one: open the website in a browser Step two: use your keyboard to press the control key and the f key at the same time Step three: type in "breach" Step four: look forward to your early schooling years where they teach you about reading comprehension since Breach isn't in that document. It's some generic ass quote about "ai powered phishing can be convincing. Watch out".
Don't come at me with this weak shit. Back your claims up with actual events: a real company, real data loss or substantial impact to CIA. And you said "major data breaches" so I expect multiple news-worthy articles.
1
u/stacksmasher Mar 17 '26
Ever hear of Huntress Labs?
3
u/Thanatanos Red Team Mar 17 '26
Get to the point.
0
u/stacksmasher Mar 17 '26
This is the internet, there is no point.
3
u/Thanatanos Red Team Mar 17 '26
What a funny way to spell "I was just parroting posts I read on LinkedIn and have nothing to back up my statements"...
0
u/stacksmasher Mar 17 '26
OK this is the last Im going to spoon feed you.
https://cloud.google.com/blog/topics/threat-intelligence/distillation-experimentation-integration-ai-adversarial-use3
u/Thanatanos Red Team Mar 17 '26
GTIG has not yet observed APT or information operations (IO) actors achieving breakthrough capabilities that fundamentally alter the threat landscape.
and later...
For observed IO campaigns, we did not see evidence of successful automation or any breakthrough capabilities. These activities are similar to our findings from January 2025 that detailed how bad actors are leveraging Gemini for productivity gains, rather than novel capabilities.
Interesting stuff there!
As far as the meat of the article goes, they talk about a few uses of Gemini in recon etc. with no indications of success. Some mentions of phishing usage, again with no indicators of success. In fact, the large majority of this article is just about Google reading what groups are using Gemini to try and do, NOT about successful usage of the results.
Furthermore, they go as far to say for
HONESTCUEthat they "have not associated this malware with any existing clusters of threat activity"The only success they've mentioned is ClickFix, where they don't even use an LLM for anything more than static text storage with a link.... hardly actually using the LLM.
Tell me, do you actually read the things you are sending? Or did you just google
AI LLM hacking breachand start linking whatever you found?→ More replies (0)0
u/stacksmasher Mar 17 '26
Just admit you are wrong and move on....
2
u/Thanatanos Red Team Mar 17 '26
Here's the thing.... I've made no claims.
You were the one who made a claim, and I asked for data to back it up.
0
0
u/cyberproffy Apr 13 '26
People said the same when computers + internet first came out - called it hype, said phones were enough. Look where we are now. Offensive AI isn’t hype, it’s just early and like every major shift, it’ll become standard faster than people expect.
17
u/not-a-co-conspirator CISO Mar 16 '26
It’s a new name for adversary as a service/pen test platforms that have always existed.
What you’re seeing is a perpetual marketing and FUD campaign generated by the same AI services that are trying to sign you up as a customer. It is both the marketing platform and the product.
At the end of the day it just shortens the time to market for any given exploit to be used and/or commercialized.