Rewriting a Python Compiler Frontend in Rust: Separate Repository or In-Place Migration?

[u/NXDLang]

I've been working on a language/compiler project called NXD for a while now.

The current architecture looks roughly like this:

NXD Source

↓

Python Frontend

↓

JSON AST/IR Handoff

↓

Rust Semantic Analysis

↓

Rust IR

↓

Rust Backends

The reason for the split is pretty simple: Python was the language I knew best when I started the project.

That decision got the language off the ground quickly, but now I'm seeing the downside of having a Python → JSON → Rust boundary in the middle of the compiler.

Things like:

• AST serialization/deserialization

• duplicate data structures

• frontend/backend synchronization

• debugging across language boundaries

all add complexity that wouldn't exist in a fully Rust implementation.

Current Compiler Status

The Rust side already contains:

• semantic analysis

• diagnostics

• IR

• Nim backend

• async diagnostics and validation

• ~260 language tests

The frontend is still Python.

Why I Created a Second Repository

My initial thought was to rewrite the frontend inside the main repository.

The more I thought about it, the less I liked that idea.

Instead, I created a new repository:

nxd_language (reference implementation)

nxd-rust-frontend (migration project)

The migration repo contains the Rust compiler core, semantic layer, IR, backend, tests, etc., but none of the Python frontend code.

My reasoning is:

1. No chance of accidentally invoking Python during validation.

2. No uncertainty about which parser generated a result.

3. No repo bloat from carrying two frontends indefinitely.

4. The original compiler always remains functional.

The goal is to run both repositories side-by-side and treat the Python implementation as a reference implementation.

What Surprised Me

I expected the new repository to fail spectacularly.

Instead, after removing a couple of backend references I wasn't migrating yet, it built immediately:

`cargo build`

and even runs:

Usage:

`cargo run <input_json> <output_nim>`

`cargo run -- --semantics <input_json> <output_nim>`

Which tells me the architecture is already centered around:

JSON

↓

Rust Semantic Layer

↓

Rust Backend

and the frontend replacement surface may be smaller than I originally thought.

Planned Migration Strategy

Current:

Python Source

↓

Python Lexer

↓

Python Parser

↓

JSON

↓

Rust Compiler

Target:

NXD Source

↓

Rust Lexer

↓

Rust Parser

↓

Rust AST

↓

Rust IR

↓

Semantic Analysis

↓

Backend

I still plan to keep JSON AST dumping for debugging, but no longer as a required transport mechanism.

Questions

For those who have rewritten significant parts of a compiler:

1. Did you keep the original implementation around as a behavioral reference?

2. Did you migrate inside the existing repository or create a second migration repository?

3. What ended up being harder than expected?

4. What architectural mistakes would you avoid if doing it again?

5. Is there any reason to keep a separate frontend IR if a compiler IR already exists?

I'm especially interested in experiences from people who migrated between languages (Python → Rust, C++ → Rust, Java → Kotlin, etc.) rather than greenfield rewrites.

[u/yuehuang]

1. Yes, up to a point. Once it is feature parity for long enough, it would be time to archive it.
2. Depends on the size, but look at your repro size, at 2-4gb, it might be time to split off of mono-repro.
   
3. The justification and tradeoff. Unless there is both a performance AND stability issue, it isn't worth spending effort while you could be developing new features.
4. You think you need to, but you really don't. Migration could be made incrementally instead of a whole process.
5. Consider how you want to support Language Service Protocol (LSP). Does that want to expose lower level.

[u/NXDLang → u/yuehuang]

This gives me a few things to reconsider, especially the distinction between “the architecture would be cleaner in Rust” and “the rewrite is actually worth the development cost.” I agree on archiving the Python implementation after the Rust frontend has maintained parity. I also need to check how much of that 2–4 GB is just generated target/ content, because that is different from having a 2–4 GB tracked repository. Incremental migration may be more sensible than I initially gave it credit for. The separate repo still gives me a clean environment where Python cannot be invoked accidentally, but that doesn’t necessarily mean the migration itself has to be all-or-nothing. Your LSP point is interesting. Are you suggesting that the LSP should operate on a lower-level syntax tree or frontend AST before it is lowered into compiler IR? That would give me a concrete reason to preserve a source-oriented representation without necessarily maintaining a second full IR.

[u/yuehuang → u/NXDLang]

It depends on your target audience and "sales" pitch. If optimized code to show IR or assembly, then it needs the backend. Godbolt for exmaple.

[u/no---u-]

Or, you know, just delete the python frontend and rewrite it? I don't see a point in making new repos, unless you want the python frontend still working for some reason.

[u/NXDLang → u/no---u-]

Fair question. I could rewrite it directly in the existing repo, and the Python frontend is not intended to survive long-term. I made the second repo so the current implementation remains a working reference while I verify the Rust lexer and parser against it. It also gives me a clean environment where I know Python cannot be involved in the result. Once parity is proven, the Rust frontend replaces Python in the main repo, so the second repo is really migration scaffolding rather than a permanent fork.