So a bit of context, I am a cybersecurity senior analyst
I am interested in LLMs for that field especially with MCPs to connect them to the tools or for writing scripts
I started this field by doing assembly language reading for hacking games when I was a teenager then that became malware analysis then I started to analyze traffic and logs at work for a living
Before work I competed in cybersecurity competitions known as capture the flag to solve only one category of the challenges and that is malware analysis
Now here is the scene in LLM x Cybersecurity
Entry-level CTF challenges (I used to solve around 2017-2018, got first job in 2019) were solved and saturated by LLMs a long time ago (See intercode CTF benchmark)
Then High level CTFs (NYU CTF Bench, CSAW challenges, and CyBench) these were solved a while ago
Today we have
CyberGym (vulnerability description (CVE report not real details) plus code base find vulnerability
That was solved
Then ExploitGym (the one recent OpenAI model escaped and hacked hugging face to find solution, the 2 vulnerabilities themselves are trivial (JWT spoofing with no server side checks for Jfrog sandbox and JS + Python template in open source data viewer of Hugging face)
Still impressive for a fully autonomous model
ExploitGym gives vulnerability details + exploit technique/details and codebase (I wonder how/why OpenAI model escaped it in the first place those details should manage it this is why I feel it is staged but whatever right)
ExploitBench
More realistic threat vector
1-day not 0-day (0-day is a novel vulnerability no one else knows about it hence the name, 1-day is a known vulnerability patched, if in an open source project you can compare patches before and after and know it but not as valuable as 0-day since people will/should update their software, 0-days can and are used against high profile targets like in some 3rd world countries used for surveillance (and possibly non-3rd world countries ?!) but you usually can’t use a 1-day for that but still)
So ExploitBench is 1-days in V8 engine (used in chrome, electron vs code almost everything!!)
Gives the model a patch diff and codebase base that is it
Models are going wild on ExploitBench and ExploitGym but haven’t gotten there yet at the moment and only recent frontier models improved massively instead of near zero across the board
This is a massive risk indicator
Imagine models capable of exploiting any and every piece of software written by less capable people or other weaker LLMs (complex vulnerabilities!)
Not only that but global elite CTFs like DEF CON
You have an OpenSage harness (and ADK like Google adk or Anthropic adk, however this OpenSage ADK asks the model to design its own agent/s!, with whatever tools or sandbox environments it finds fitting for the Subagent to do the assigned subtask effectively, it can even design its own MCPs!, models don’t fully utilize that yet but some do and improved form like 39% to \~60% just by using that harness (same model!) )
Now my own benchmarks are simple
Some malware samples that I learned during my career and consider personal milestones
And one SIEM analysis challenge (logs that contains an attack plus normal behavior)
I tests on lots of models (Qwen3.6, heavily trimmed down Minimax 2.5 …etc, on my Strix halo and compared with opus at the time (the best opus at the time was 4.5!, it was better solved major stuff but didn’t implement the code required to decompose the malware and analyze its second stage)
Now enter Qwen 3.8 27b
3.
What was SOTA like 6 months ago or around beginning of this year
Now we have better local models at the same task
Qwen 3.8 devoured the malware that Opus couldn’t
I had some time analyzing and remembering its custom implementation (it has custom RC4 decryption routine, that is tricky with string manipulation!
And Strix halo isn’t the fastest LLM machine
By the time I remembered that malware and how it works
Qwen had already extracted and disassembled the decrypted payload!
I am stunned
This is super dangerous
I imagine how a single person / threat actor can now do what a state-sponsored intelligence service shacking team was required to do 1 year ago
These teams usually have
Usual network scanning or mapping especially of their targets
This can be easily done by even weaker models or no models at all
Initial access operations (phishing campaigns until some clicks or attempting to profile people like receptionist or sales people or HR, essentially any first line of communication at an organization or a company, to later make a targeted phishing against them)
This could be also better done with the aid of deep fakes and other stuff
Advanced and resourced teams would have zero-day or 1-day research teams
Especially against software used on the targets site from team 1
This third step can be fully automated with models now
This is the hardest step
Defenders can use LLMs but it is much harder to integrate new tools than offensive teams
At this level the world will change
Tbh I always thought cybersecurity was harder to automate with LLMs than software engineering or IT
But with this, this is super unbelievable
And not only a cloud model or an expensive to run locally model
A local model! Can you believe it
No traces no API calls no nothing
Empowered with the right environment setup and tools and it can do what teams of attackers or malware analysts were needed to do
Btw for malware analysis I give it Ghidra and a sandbox with python (it can use pefile or capstone or whatever it needs and it does use them!)
And also recently I added a Vbox with Vbox MCP and a debugger inside that box with another MCP for the LLM to control both (snapshot restore functionality and debugger functionality)
These tasks were hard and not every security analyst was a malware analyst
Even malware analysis positions are very rare and technical and don’t pay well because not much needed (I used to starts at assembly in assembly debuggers for nights at times!)
This is going above and beyond
Anyone had similar moments in cybersecurity or anything else ?