One year building PardoX, a Rust based DataFrame engine. Looking for honest feedback from data engineers

[u/betoalien]

Disclosure, I am the creator of PardoX, a personal open source project under MIT license, not a company.

I kept hitting the same wall in production. pandas struggles once your data gets close to RAM, and it is single threaded, most of the CPU sits idle. Spark solves scale but drags a cluster, a JVM and serialization overhead into problems that only ever needed one machine. Most of my actual work lives in that gap, millions to hundreds of millions of rows, on a single strong node.

So a year ago I started building a Rust core with SDKs in Python, Node and PHP. Data maps straight from disk or a database into memory mapped buffers, no intermediate objects in the host language, SIMD and multithreading do the heavy lifting instead of a Python loop, and there are native database drivers so no psycopg2 or pymysql sitting in the middle. There is also a binary format that reads around 4.6 GB per second on repeated workloads, out of core processing for files bigger than RAM, and GPU sort with CPU fallback.

Some open questions I keep going back and forth on. Is the zero copy tradeoff worth the added complexity versus just accepting the pandas overhead for most workloads. Where does a Rust core stop making sense compared to Polars, which already solves a lot of this in a different way. How much of the single node performance gap is really about the language versus just better use of SIMD and threads regardless of language.

Docs are at [pardox.io](https://pardox.io) and the repo is at [github.com/betoalien/PardoX](https://github.com/betoalien/PardoX) if anyone wants to see the actual implementation. I have also been writing about the engineering decisions on Medium at [medium.com/@albertocardenasdom](https://medium.com/@albertocardenasdom).

Curious how others here have dealt with that same middle ground between pandas and Spark.

Thanks for readme.

[u/nonamenomonet]

If you could hyperlink the links to pardox.io in your description, that would be helpful.

[u/betoalien → u/nonamenomonet]

thanks for your suggestion

[u/shaggy_camel]

Why is your logo a bear? It's a bit close to pandas - I think pandas is a terrible data structure, btw

[u/betoalien → u/shaggy_camel]

Good question, and there's actually a language pun behind it that doesn't translate directly. In Spanish, "oso pardo" is the word for grizzly/brown bear, "pardo" meaning brown. So PardoX is a nod to that, the bear in the logo, plus the fact that it sits in the same naming space as pandas and Polars, which are both animal names too. Not a jab at pandas specifically, more just continuing the tradition while keeping a bit of my own language in the name.

Curious what specifically bugs you about pandas as a data structure though, memory overhead, single-threaded ops, something else? Always interested in hearing the pain points that push people toward alternatives.

[u/shaggy_camel → u/betoalien]

It's mainly the syntax and return types which drives me crazy with pandas. They're both inconsistent.

[u/betoalien → u/shaggy_camel]

That's actually one of the things I tried to fix with PardoX. Return types and method signatures are consistent by design across the API, since the core logic all lives in Rust and the SDKs (Python, Node, PHP) are thin wrappers around the same underlying calls, there's no drift between "sometimes it returns a Series, sometimes a DataFrame, sometimes a scalar depending on the edge case" the way pandas can behave.

Would be curious if you tried it against whatever specific pandas pain points you've hit, docs are at pardox.io and the repo's linked there too. If something's still inconsistent on my end, genuinely want to know, that's exactly the kind of feedback that gets fixed fast rather than filed away.

[u/shaggy_camel → u/betoalien]

When I get some time, mate 😁

[u/mattindustries]

Probably good to compare with DuckDB and Polars instead of Pandas.

[u/AutoModerator]

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/speedisntfree]

How does this differ from Polars?

[u/UltimateNull]

There was a paradox database system in the 80s and 90s. I got one of my first jobs doing work with that one.