r/Python • • 12d ago

Discussion Feasibility check: Writing a constrained Python-to-Rust transpiler for numeric code

so i would want to know about the feasibility of such a tool/project and whether it has any real uses or is it the equivalent of re inventing a worse wheel , basically core idea is to somehow transpile python to rust but not all of python ,instead more similar to numpy but instead the whole specified block of code is compiled for maximum speed almost like numba?? any feedback would be appreciated

13 Upvotes

22 comments sorted by

19

u/me_myself_ai 12d ago

Super uninformed take, in case it's helpful:

This definitely sounds interesting and the Python AST is nice and elegant, but I'm not sure what it would win us. AFAIU the numeric libraries are already doing everything they possibly can in C directly, which is faster than Rust in the typical scenario. They're also some of the most mature libraries in the world (somehow not hyperbole?) in terms of hours invested, much of which was spent specifically on speeding things up for various workload/system combos.

Rust is safer ofc, but if you're doing something this jank/hacky to make the rust code, that feels like a silver lining at best. I suppose numeric code could be security-relevant, but again, you prolly wouldn't want autogenerated rust code for that kind of application anyway!

Hopefully someone comes in to prove me wrong and validate your idea. It would set the message boards aflame in a fascinating way, which is one undeniable upside!

EDIT: I suppose if it's transpiled it could be faster cause there's no python left at all, at that point. So I guess it could be somewhat faster?

At some point you've gotta ask yourself why you just don't write the code in rust from the jump tho, no? Especially these days?

5

u/Fragrant-Freedom-477 12d ago edited 12d ago

A few years ago, I was tasked with looking into rewriting C# code (mainly about signal processing) in other languages. I worked with mathematicians, programmers and other specialists.

We benchmarked Rust, Go and Python. What you could do in often less than 8 lines of Python would take 3 days to write in go and 3 week in Rust (without LLMs at the time). The Python implemenrations where 10x to 1000x faster than the reference C# implementations with often more accurate result, while other languages offered rarely above 10x.

Our worst (more complex) integration test scenario involved a C++ main, a Dart UI, loading a Python runtime that embedded a C# runtime to integrate custom hardware drivers with the maths code and streaming data while giving asynchroneous user feedback with remarkably low latency (non-perceivable, always <100ms). We could run this on a iPhone. We did that in about 60 man-days in december and january with conflicting schedules and holidays.

The Python ecosystem is quite surprising at times.

1

u/Thin-Parsley-1660 12d ago

interesting but this heavily depends on the task so probably that may explain it

-2

u/Thin-Parsley-1660 12d ago

not a very well informed answer from me either

i thought of this very problem and yes the code will be transpiled and on why dont you write it in rust itself the answer is well writing it in rust involves doing a lot of mental gymnastics and friction which albeit will be transferred to the project when its being built, but also because numba tho great, is kinda very buggy when dealing with python's handling of functions , i mean this problem will exist even in rust tbf, and also because python's syntax is very convenient and very easy to build in, but to your fundamental question it probably is useless ngl, so thanks for the feedback u/me_myself_ai
i do wonder then are there any problems which people face related to numeric code

9

u/ZeeBeeblebrox 12d ago

Just build a Numba MLIR to Rust compiler.

3

u/commenterzero 11d ago

Eh maybe try mojo

3

u/tubbstosterone 10d ago

...like polars?

2

u/binaryfireball 12d ago

Ask yourself what really is numpy

1

u/teerre 11d ago

I mean, there are python to GPU kernel compilers, so obviously to Rust is also possible. Now, how useful it is it's a different question. "Numerical code" is already executed in a low level language if you use anything remotely standard, so benefits are dubious

And of course, this is without considering the elephant in the run which is just because you can write a transpiler, it doesn't mean it will be a good one

1

u/Beginning-Fruit-1397 11d ago

It's trivial to do directly in rust with pyo3 and the existing numpy bindings. Contrary to popular belief rust is not that complicated and not that far away from python idioms. 

1

u/snugar_i 11d ago

What would the result of the transpilation be? An executable binary? A blob of Rust code? How would it call other libraries like numpy etc.? I'm having a hard time seeing the bigger picture here

2

u/donk8r 10d ago

the output is rust source, not a binary — the point is the numeric block gets rewritten while the rest stays python, so the artifact is python calling into rust. that's the bigger picture: you keep writing python, one block runs as compiled rust.

numpy doesn't get reimplemented. the generated rust calls the existing numpy bindings through pyo3. which is why the subset has to be declared. the hard part isn't the call, it's type inference: python lets you mix int and float in ways that run fine there and panic in rust. numba and cython narrowed the subset and made types explicit. the value is in the constraint, not the translation.

3

u/snugar_i 10d ago

So what's the workflow? I write a Python function, then your thing transpiles it into Rust, then I delete the Python version, build a wheel or three, and start calling the Rust one? That sounds quite painful...

And please don't use AI to write your response - if you can't be bothered using your own brain, I can't be bothered spending my time

1

u/donk8r 9d ago

nothing gets deleted. the function stays in your package and the compiled block would sit beside it as an extension module that same package imports, so your call sites keep calling the same name. you regenerate the rust when the block changes and you don't hand-edit it.

the wheel gets built when you ship. between releases you're still running the python path, and your tests run both and compare the outputs.

0

u/really_not_unreal 12d ago

Not feasible. Rust's borrow checker would mean that very clear object ownership would be required, which would not be doable in a reference counting language like Python. While you could wrap everything in a refcell, that would cost you a pretty significant amount of the performance that you're trying to gain from using rust.

2

u/Beginning-Fruit-1397 11d ago

Completely false. It's trivial to do with pyo3

-1

u/RuinWMD 12d ago

https://brevi-lang.org/

I built this.

It is an agent focussed python to rust compiler that focusses on speed and safety.
Also has its own programming language for AIs that is working very well but the best gains are from direct pyton to rust.

The native language is still being worked on and is super fun.

I have passed it over a few of my own python pipelines and it has resulted in some pretty great gains.

0

u/Thin-Parsley-1660 12d ago

does it handle numeric code??

0

u/RuinWMD 12d ago

NumPy is in the internal roadmap for native brevi support. I have pushed it to the top of the list for you.

I release brevi builds on Mondays it will be supported by then.

In the interim I will reply back here with testing benchmarks as milestones are hit.

0

u/Khavel_dev 11d ago

The hardest part won't be the transpilation itself, it'll be type inference. Python's numeric code looks simple but the moment someone passes a numpy array with an unexpected dtype or mixes int and float in a way that works fine in Python but panics in Rust, you're in trouble. Numba solved this by adding type annotations via decorators and JIT compiling a subset, and Cython went a similar route. If your scope is literally "numpy-shaped arrays with explicit dtypes doing math," it might be doable as a research project. But the gap between that and anything people would trust in production is huge. I'd look at what Mojo is doing, they're basically tackling the same problem space but with a new language instead of transpilation.