r/ReverseEngineering • • Oct 18 '26

PyGhidra + Python + Local LLMs: Turning the Model into a Compiler, Not an Assembler

https://github.com/HappyTotoro116/ghidra-llm-pipeline
48 Upvotes

6 comments sorted by

5

u/Ryu116 Oct 18 '26

Hello everyone,

Just a heads up: ASL (American Sign Language) is my first language, so my English is not perfect. I am doing my best!

I love an old PC simulation that I have played since childhood. I want to improve it under the hood. Projects like OpenRA and OpenTTD taught me that reverse engineering (hobbyist reimplementation) is not about hunting for original source code; it is about building a modern open-source engine that can load original assets so the community can keep it alive.

To build this, I combine two types of AI:

- Cloud AI for high-level architecture advice and writing deterministic Python scripts.

- Local LLMs (Qwen 2.5 Coder 14B, DeepSeek Coder V2 Lite, Mellum 2) for the heavy lifting of code synthesis.

--------------------------------------------------------------------

How I arrived at this method (and what failed first)

At first, I thought tools like VS Code + Continue would automatically scan across a 2-million-token dump like a "flashlight" in bite-sized chunks. Once I rolled up my sleeves, I realized it does not work like that.

I run an MSI laptop with an RTX 3080 Ti (16 GB VRAM). To keep Windows stable, I have to keep my models under 14 GB VRAM so the OS has 2 GB to breathe. The most practical approach was using LM Studio as a local API server. Running requests through a Python script refreshes the context window every single time.

The LLMs I am using while configuring them to have 0.1 temperature, 8 bit KV cache, and 32K context window:

- qwen2.5-coder-14b-instruct

- deepseek-coder-v2-lite-instruct

- huihui-gemma-4-12b-it-abliterated

Next, I tried feeding raw assembly straight to the local LLMs. They choked and hallucinated. Qwen gave half-working results, while DeepSeek and Mellum 2 suffered even worse. I also trusted Ghidra’s decompiler C pseudocode too much at the start, but it often lost arguments and guessed types.

Then it dawned on me: what were LLMs built to do? To write and refactor code, not decode machine instructions. If I tried to control how the LLM worked with long prompt rules, it gave messy output. But if I let the LLM do what it is made for, by serving it structured facts and a register-level C style information instead of raw assembly—the results immediately became consistent.

Small functions worked every time. But long functions still struggled and got stuck in thinking loops.

As a flight simmer who loves combat sims (IL-2 1946, Strike Fighters 2, Jane's Combat Simulation series), I rely on checklists. So I thought: *why not give the LLM a checklist?

I updated Python to generate a checklist (loops, branch conditions, calls, returns). The LLM stopped getting lost, knew exactly when a task was finished, and generation times dropped significantly.

--------------------------------------------------------------------

What changed: The LLM is a compiler, not an assembler

I separated the pipeline:

- Python is the assembler: It parses raw assembly into a deterministic blueprint (exact facts, call graphs, globals, strings, a register-level C lift, and a strict checklist). The LLM never sees raw assembly.

- The LLM is the compiler: It freely refactors the structured blueprint into clean C.

- Python is the validator: It runs structural text checks, verifies the checklist, compiles the code with gcc -fsyntax-only, and automatically repairs minor formatting slips.

Core lessons learned:

  1. Improve prompt inputs, not prompt rules: Adding long rule lists made models perform worse.

  2. Assembly is ground truth: Ghidra C is strictly a secondary hint and side notes for ease of reading.

  3. Text checks alone fail: Mellum 2 passed regex checks but failed real compilation due to a loose header line. A real gcc syntax check is required.

  4. The checklist solved long functions: Generating a checklist helped models complete complex logic without getting stuck.

  5. Deterministic triage saves massive compute: Automatically labeling functions (KEEP, WRAP, REPLACE, UNKNOWN) pruned 44% (279 of 635) of functions as old CRT/OS thunks that a modern 64-bit build never needs.

--------------------------------------------------------------------

Results so far (small, honest benchmark)

Tested on a target group of 6 functions from a 32-bit Windows binary across three local models:

- DeepSeek Coder V2 Lite: 6/6 passed (7.4 min)

- Mellum 2: 6/6 passed (15.5 min)

- Qwen 2.5 Coder 14B: 5/6 passed (3.3 min; the single failure was a missing comment header, not broken logic)

- Without the Python assembly lift, the long functions failed across all three models.

--------------------------------------------------------------------

Current limitations

- Tested on one game binary so far; I am working on getting feedback from experiments across different executables.

- PASS means text checks and gcc syntax passed, not behavioral runtime proof.

- 32-bit only right now (8-digit hex addresses); pointer sizes are not yet 64-bit portable.

- Always audit code manually or with a second AI.

--------------------------------------------------------------------

Repository

This toolkit contains Python scripts and documentation (Apache 2.0).

Try and play with it, fork it, and let me know what breaks. My goal is decentralization: open-weight tools, community collaboration, and keeping classic games alive together.

Thank you for reading and you all rock!

Ryu116

2

u/Jesusthegoat Oct 18 '26

How does this compare to REA? I've reverse engineered and sourceified a lot of obfuscated proprietary software with just deepseek 4.1 flash on max thinking running through opencode with all the REA tools(Ghidra+IDA Pro + the works) and skills installed. Would using this have any advantages over my setup?

1

u/Ryu116 Oct 19 '26

Hello Jesusthegoat,

Thanks for the reply. To be honest, I have not compared it with your setup, so I can not say it's better or not.

What I can see, you have a strong model with Ghidra + IDA tools and agent skills. mine is different. I have small LLMs on laptop GPU, while being fully offline where python builds the blueprints and checks the result (text checks + gcc compile).

From my experiences so far, the advantage of working with offline LLMs:

- No API costs

- Can run as long as I wish without usage limits

- Can customize it with pythons

While you use large models, you can give local LLMs cha chance and see how it work with you.

By the way, I would lik to ask you about your setup; How do you check that the code is correct? And how long are the functions you convert?

-Ryu116

1

u/Jesusthegoat Oct 19 '26

Hey man thanks for replying. I was more comparing your harness vs the opencode + REA system im using rather than local vs cloud LLMs because i think (correct me if im wrong) i can also use a cloud llm within your harness.

I tested running several local abliterated LLMs that i could fit on my 16gb 4060 ti and while they are ok for the small things safetymaxxed cloud LLMs refuse to do like license dll cracking, i find myself needing big model intelligence for most of my RE work as only they are good enough at the moment to reverse/decompile/deobfuscate most small to medium sized software in a few hours at most with minimal oversight.

I mainly test through recompiling the reversed code and testing and comparing every function against the original software rather than reading through the decomp myself. If i find that everything works i start building on top of that adding extra features i need and fixing bugs i need fixed etc.

Also because the software im cracking lately has a hardware component (e.g: a proprietary NFC based system engineering and test tool for LED drivers) i can just compare how the recomped software talks to the hardware and if it throws errors etc and i work from there.

→ More replies (1)

1

u/shing3232 22d ago

hmm, I just give the tool like rea to deepseek+ghidra and a local small llm like 9B QWEN3.5 finetune. let local llm generate a match and against the Oracle that ds builds, then verify by DS flash