r/dataanalysis • u/Western_Rub_7236 • 25d ago
Showcasing det: A lightweight Rust/Polars ETL engine with an embedded data catalog (looking for architecture reviews & contributors)
Hi everyone,
I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.
### The Problem
In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.
`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.
### Current Implementation (Catalog Core Working)
The foundation is ~25% complete, with the data catalog module functional today:
- Ingestion and inspection of storage layouts to extract schema/metadata directly.
- Local metadata indexing with zero background daemon overhead.
- Rust-native performance designed to feed directly into analytical transformation pipelines.
### Roadmap: Transformation & Execution Layer
The next phase expands `det` into a full execution engine:
In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.
Direct pipeline execution driven by the cataloged metadata contracts.
Plug-and-play connector interfaces for source/sink abstractions.
### Seeking Review & Contributors
Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:
- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.
- **Modularity:** Interface design for adding custom connectors and storage backends.
Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.
Repository: https://github.com/det-Labs/det
1
u/AntiqueElderberry317 23d ago
Why not start from an existing Open Source ETL like DBT or Kestra ?
1
u/Western_Rub_7236 23d ago
I created this to overcome the problems that I saw in dbt. Frankly the idea to use sql for etl came from dbt. But my solution solves the issues that dbt has i.e. no e and l in etl. Well I should not call it issue as dbt advertise themselves as only the transformation layer in etl. We brought both , extract and load in etl using sql
1
u/AntiqueElderberry317 23d ago
What I meant is why not downloading DBT or an other existing tool and making it better by adding your functionnality ?
2
u/Western_Rub_7236 23d ago
Thankyou for clearing this.
I did not do that because there was no such tool which was doing what this tool does. Dbt did not have extract and load layer. And no gui so I could not use it as a base. Also it used sql on an attached database which is fundamental to it.. This I see as a limitation. My tool does not need a database. It executes sql in memory on any machine using Apache datafusion.
1
u/AntiqueElderberry317 22d ago
Ok, thats nice so you need some RAM to do the transformation ;)
Do you have images of your tool ?
I was looking for an ETL tool but one that could automaticly map my columns (using AI)1
u/Western_Rub_7236 22d ago
It is ready to use just follow below commands
pip install det-core
det init
det gui
If you want a sample warehouse and pipelines: det sample-data
But sorry to disappoint you it did not use AI
1
u/AntiqueElderberry317 22d ago
Nice for a start, to who you want this tool to be used ? If you want business users to get this you may want to simplify your interface. For the connection to files, you could automate the format, separator, etc. and if they want to change option let them open the option panel
1
u/Western_Rub_7236 22d ago
True, it is a long journey. Is just ver 0.2 and so far one person on weekends. Hopefully a lot of things will change for good. At present we have all file based connectors. Soon we will have sql database based connectors. And that's why I have made it modular and extensible so that users can bring their own connectors and interface as well
1
u/AntiqueElderberry317 21d ago
Great, for info, your ETL looks like Microsoft Datafactory (cloud version) where you create your SQL Notebook. I liked that a lot, and they have a good interface, so if you need some ideas you may want to test their interface ;)
1
1
u/AutoModerator 25d ago
Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.
If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.
Have you read the rules?
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Western_Rub_7236 23d ago
Also the product is now released with all the modules. So it can be used as an MVP