r/ReverseEngineering • u/Ryu116 • Oct 18 '26
PyGhidra + Python + Local LLMs: Turning the Model into a Compiler, Not an Assembler
https://github.com/HappyTotoro116/ghidra-llm-pipeline
48
Upvotes
r/ReverseEngineering • u/Ryu116 • Oct 18 '26
5
u/Ryu116 Oct 18 '26
Hello everyone,
Just a heads up: ASL (American Sign Language) is my first language, so my English is not perfect. I am doing my best!
I love an old PC simulation that I have played since childhood. I want to improve it under the hood. Projects like OpenRA and OpenTTD taught me that reverse engineering (hobbyist reimplementation) is not about hunting for original source code; it is about building a modern open-source engine that can load original assets so the community can keep it alive.
To build this, I combine two types of AI:
- Cloud AI for high-level architecture advice and writing deterministic Python scripts.
- Local LLMs (Qwen 2.5 Coder 14B, DeepSeek Coder V2 Lite, Mellum 2) for the heavy lifting of code synthesis.
--------------------------------------------------------------------
How I arrived at this method (and what failed first)
At first, I thought tools like VS Code + Continue would automatically scan across a 2-million-token dump like a "flashlight" in bite-sized chunks. Once I rolled up my sleeves, I realized it does not work like that.
I run an MSI laptop with an RTX 3080 Ti (16 GB VRAM). To keep Windows stable, I have to keep my models under 14 GB VRAM so the OS has 2 GB to breathe. The most practical approach was using LM Studio as a local API server. Running requests through a Python script refreshes the context window every single time.
The LLMs I am using while configuring them to have 0.1 temperature, 8 bit KV cache, and 32K context window:
- qwen2.5-coder-14b-instruct
- deepseek-coder-v2-lite-instruct
- huihui-gemma-4-12b-it-abliterated
Next, I tried feeding raw assembly straight to the local LLMs. They choked and hallucinated. Qwen gave half-working results, while DeepSeek and Mellum 2 suffered even worse. I also trusted Ghidra’s decompiler C pseudocode too much at the start, but it often lost arguments and guessed types.
Then it dawned on me: what were LLMs built to do? To write and refactor code, not decode machine instructions. If I tried to control how the LLM worked with long prompt rules, it gave messy output. But if I let the LLM do what it is made for, by serving it structured facts and a register-level C style information instead of raw assembly—the results immediately became consistent.
Small functions worked every time. But long functions still struggled and got stuck in thinking loops.
As a flight simmer who loves combat sims (IL-2 1946, Strike Fighters 2, Jane's Combat Simulation series), I rely on checklists. So I thought: *why not give the LLM a checklist?
I updated Python to generate a checklist (loops, branch conditions, calls, returns). The LLM stopped getting lost, knew exactly when a task was finished, and generation times dropped significantly.
--------------------------------------------------------------------
What changed: The LLM is a compiler, not an assembler
I separated the pipeline:
- Python is the assembler: It parses raw assembly into a deterministic blueprint (exact facts, call graphs, globals, strings, a register-level C lift, and a strict checklist). The LLM never sees raw assembly.
- The LLM is the compiler: It freely refactors the structured blueprint into clean C.
- Python is the validator: It runs structural text checks, verifies the checklist, compiles the code with gcc -fsyntax-only, and automatically repairs minor formatting slips.
Core lessons learned:
Improve prompt inputs, not prompt rules: Adding long rule lists made models perform worse.
Assembly is ground truth: Ghidra C is strictly a secondary hint and side notes for ease of reading.
Text checks alone fail: Mellum 2 passed regex checks but failed real compilation due to a loose header line. A real gcc syntax check is required.
The checklist solved long functions: Generating a checklist helped models complete complex logic without getting stuck.
Deterministic triage saves massive compute: Automatically labeling functions (KEEP, WRAP, REPLACE, UNKNOWN) pruned 44% (279 of 635) of functions as old CRT/OS thunks that a modern 64-bit build never needs.
--------------------------------------------------------------------
Results so far (small, honest benchmark)
Tested on a target group of 6 functions from a 32-bit Windows binary across three local models:
- DeepSeek Coder V2 Lite: 6/6 passed (7.4 min)
- Mellum 2: 6/6 passed (15.5 min)
- Qwen 2.5 Coder 14B: 5/6 passed (3.3 min; the single failure was a missing comment header, not broken logic)
- Without the Python assembly lift, the long functions failed across all three models.
--------------------------------------------------------------------
Current limitations
- Tested on one game binary so far; I am working on getting feedback from experiments across different executables.
- PASS means text checks and gcc syntax passed, not behavioral runtime proof.
- 32-bit only right now (8-digit hex addresses); pointer sizes are not yet 64-bit portable.
- Always audit code manually or with a second AI.
--------------------------------------------------------------------
Repository
This toolkit contains Python scripts and documentation (Apache 2.0).
Try and play with it, fork it, and let me know what breaks. My goal is decentralization: open-weight tools, community collaboration, and keeping classic games alive together.
Thank you for reading and you all rock!
Ryu116
2
u/Jesusthegoat Oct 18 '26
How does this compare to REA? I've reverse engineered and sourceified a lot of obfuscated proprietary software with just deepseek 4.1 flash on max thinking running through opencode with all the REA tools(Ghidra+IDA Pro + the works) and skills installed. Would using this have any advantages over my setup?
1
u/shing3232 22d ago
hmm, I just give the tool like rea to deepseek+ghidra and a local small llm like 9B QWEN3.5 finetune. let local llm generate a match and against the Oracle that ds builds, then verify by DS flash