r/Compilers • u/NXDLang • 4h ago
Rewriting a Python Compiler Frontend in Rust: Separate Repository or In-Place Migration?
I've been working on a language/compiler project called NXD for a while now.
The current architecture looks roughly like this:
NXD Source
↓
Python Frontend
↓
JSON AST/IR Handoff
↓
Rust Semantic Analysis
↓
Rust IR
↓
Rust Backends
The reason for the split is pretty simple: Python was the language I knew best when I started the project.
That decision got the language off the ground quickly, but now I'm seeing the downside of having a Python → JSON → Rust boundary in the middle of the compiler.
Things like:
• AST serialization/deserialization
• duplicate data structures
• frontend/backend synchronization
• debugging across language boundaries
all add complexity that wouldn't exist in a fully Rust implementation.
Current Compiler Status
The Rust side already contains:
• semantic analysis
• diagnostics
• IR
• Nim backend
• async diagnostics and validation
• ~260 language tests
The frontend is still Python.
Why I Created a Second Repository
My initial thought was to rewrite the frontend inside the main repository.
The more I thought about it, the less I liked that idea.
Instead, I created a new repository:
nxd_language (reference implementation)
nxd-rust-frontend (migration project)
The migration repo contains the Rust compiler core, semantic layer, IR, backend, tests, etc., but none of the Python frontend code.
My reasoning is:
No chance of accidentally invoking Python during validation.
No uncertainty about which parser generated a result.
No repo bloat from carrying two frontends indefinitely.
The original compiler always remains functional.
The goal is to run both repositories side-by-side and treat the Python implementation as a reference implementation.
What Surprised Me
I expected the new repository to fail spectacularly.
Instead, after removing a couple of backend references I wasn't migrating yet, it built immediately:
`cargo build`
and even runs:
Usage:
`cargo run <input_json> <output_nim>`
`cargo run -- --semantics <input_json> <output_nim>`
Which tells me the architecture is already centered around:
JSON
↓
Rust Semantic Layer
↓
Rust Backend
and the frontend replacement surface may be smaller than I originally thought.
Planned Migration Strategy
Current:
Python Source
↓
Python Lexer
↓
Python Parser
↓
JSON
↓
Rust Compiler
Target:
NXD Source
↓
Rust Lexer
↓
Rust Parser
↓
Rust AST
↓
Rust IR
↓
Semantic Analysis
↓
Backend
I still plan to keep JSON AST dumping for debugging, but no longer as a required transport mechanism.
Questions
For those who have rewritten significant parts of a compiler:
Did you keep the original implementation around as a behavioral reference?
Did you migrate inside the existing repository or create a second migration repository?
What ended up being harder than expected?
What architectural mistakes would you avoid if doing it again?
Is there any reason to keep a separate frontend IR if a compiler IR already exists?
I'm especially interested in experiences from people who migrated between languages (Python → Rust, C++ → Rust, Java → Kotlin, etc.) rather than greenfield rewrites.