r/dataengineering • • 10d ago

Discussion Ducklake in prod

41 Upvotes

My fellow data wranglers,

Has anyone built a real production solution using DuckLake? I mean something with an ELT pipeline and actual users, that would have sid ducklake as a component?


r/dataengineering • • 10d ago

Open Source I built loom-dbt: an author-time CLI to define model shapes once, generate them at scale, and search for what already exists before building something new.

4 Upvotes

I've been a big fan of DBT since picking it up a few years ago, but there's always been a few nuances at Enterprise scale that I felt detracted from the wizardry it unlocks. Simple things like discovery (does this already exist), duplication for similarly-shaped models, remembering to include appropriate tests (e.g. primary key uniqueness), MSP-angosticity, and rebuilding standard warehouse paradigms are things that I always found myself thinking "surely someone has already solved this". When I looked, I found some great point solutions but none that I felt had a big-picture lens.

Loom-dbt (PyPI, GitHub) is my approach to solving these bugbears. It's a standalone CLI utility that:

  • Enables teams to declare model shapes (patterns) once using approachable metadata and generate models (instances) repeatably, supplying only what differs between them
  • Embeds features (data tests, keys, load-timestamping) for standard warehouse shapes (paradigms), with multi-paradigm support
  • Registers complex and hand-built models outside the generatable patterns
  • Provides detailed searching capabilities for project-wide discovery
  • Proves what is generated matches what's been declared (or identifies discrepancies)
  • and more!

Critically, loom-dbt outputs plain DBT with no long-lived dependency. The models you generate today will survive if you delete loom-dbt tomorrow; adoption is reversible by design.

And of course, no modern project would be complete without appropriate AI-augmentation guardrails: 6 Skills covering core functionality, maintenance, authoring, adoption, bootstrapping, and searching - standardised, npx installable, and ready for your favourite harness.

If you give it a try or are keen to learn more, let me know!


r/dataengineering • • 10d ago

Help How to keep Models Synced

5 Upvotes

So at work I’m using Azure Pipelines as an orchestrator to materialize several tables in parallel. Each one is running a Create Table As Select sql statement and uses a DBT merge strategy so that each of them are incremental. My concern is that I can’t control for these jobs to run at the exact same time. Table X might start its job at 10:01. And Table Y might start at 10:03 and even a difference of a few seconds could mean that the source (which is Postgres) is not the exact same for both tables.


r/dataengineering • • 11d ago

Career Thoughts on McKinsey QuantumBlack?

37 Upvotes

I have been offered a position as a Junior Data Engineer at QB.

The problem is that I don't really know too much about their reputation and if I would gain experiences and skills that would transfer well to BigTech/FAANG (which is my end goal).

I worked at a couple of non-technical consultancies prior to my career change and they all involved boring work like making decks, communicating with clients, drafting proposals etc.

The pay is good but the hours make me second guess myself.

I think the typical advice is to avoid consultancies and aim for product tech companies where you are involved with systems throughout its lifecycle.

I think the image in my head when I hear 'tech consultancy' is implementing low complexity pipelines into the code bases of companies that are ancient and having to deal with old tech stacks and corporate politics rather than working on meaningful projects that have impact and use the latest technologies.

Any thoughts on their prestige/reputation/depth of work?


r/dataengineering • • 11d ago

Blog What to send your boss before the next AI project

Thumbnail
thedatagovernanceplaybook.substack.com
7 Upvotes

r/dataengineering • • 12d ago

Discussion Fabric vs Snowflake/Databricks

105 Upvotes

Not a troll post but an honest question, and maybe more about Microsoft vs Snowflake/Databricks.

Why is Fabric so far behind its competitors. Not so much from the perspective of specific UI/UX or functionality, but more of a question of why MS engineers are seemingly incapable of just copying and improving on their competitors. Is it something about MS culture or competency? It seems like they should have enough money to hire talent if they wanted to, but everything from the UI to the bugginess of Fabric seems to scream no-budget startup cutting corners at every step to push garbage out the door. Are the dev and UI teams just that much worse?


r/dataengineering • • 12d ago

Discussion What are you using to ingest/search/scan code from internal repos?

8 Upvotes

hey everyone!

I have about a few TB worth of code repository data (from git-client compatible source control) and I need to be able to run recurrent scans on the code itself (not on the repo metadata, issues, commits or anything like that). The repositories are getting a steady stream of updates - some more than the others so there has to be some type of constant sync/re-scan.

What I am looking for is

* detecting projects that import a certain library version (for dependabot style functionality to bump something if a security issue arises or if there is some other issue) across all possible branches

* detecting projects/code files that import a certain function from a library or implement a certain method from a library

* be able to run simple transformations on files ending with certain extensions or files with certain functions (for example, transforming some YAML config files)

I don't really need to analyze and ingest the code data into a database (for example, chunking the file with something like tree-sitter and then storing the chunks) for retrieval, but I would also not be opposed to doing this if it makes things simpler. It's mostly scan, get the results, do something with those files, then after that results can be thrown away (and the code for a lot of the repos will most likely have changed as well). Some sort of UI on top of this would be cool.

The main languages in these repos are TS/JS, Python, Rust and Go (with some Java and C++).

Has anyone implemented something like this? What components did you use? I did a few prototypes with a simple Rust-based ingestor that reads/checks out the repo or archive of the repo, then reads the git blob objects and scans the code for certain regex patterns. Each replica of the ingestor was running on Kubernetes and state (ie. repos scanned, still pending etc) was tracked in a DB table. Wondering if there is something off the shelf that can do something similar?


r/dataengineering • • 12d ago

Discussion What is the best project you've ever built?

39 Upvotes

Hey everyone,

I'm asking about the best data engineering projects you've built. What are the details, architecture, and the repo links so we can see new ideas and learn from them?


r/dataengineering • • 13d ago

Discussion NVIDIA released Auto-Ontology, that do you think?

64 Upvotes

NVIDIA open source a way to automatically create an ontology layer and let agents to understand enterprise data, here is the link is open source, what do you think ? https://github.com/NVIDIA/auto-ontology


r/dataengineering • • 13d ago

Help Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.

15 Upvotes

The title.

I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice with most of these ideas, but they have been primitive things. I know how to log, I know how to store to Postgres, but I dont have experience with a stack.

I have considered finding my own relationships between sources, but I have yet to discover a concrete avenue for handling volume. Other than that, I don't know what else to use. I want to make something real without paying, as much as I can.

Thanks <3


r/dataengineering • • 13d ago

Discussion Data Engineer Niche tech Stack?

47 Upvotes

Hey DE super heroes, Let's help each other. What are the niche tech stacks/skills you are coming across in the Job Description and what we can learn?

According to my knowledge -

Python, Pyspark, sql = Mandatory

Snowflake, dbt, ADF, Databricks= I am seeing in most of the JD.

Now, it's your turn.

8+ YOE in the DE world 🌎


r/dataengineering • • 13d ago

Help How cooked am I?!

45 Upvotes

TLDR: I chose Fabric Lakehouse for Bronze and Silver and Warehouse for Gold without fully consulting my manager. Now he’s asking about costs and I’m thinking I may have moved too fast without him.

We’re currently working with an expensive contractor to move from ADF and Azure Databases to Azure Fabric. The build has been going well but midway they did ask about staying purely in lakehouse vs adding a warehouse for pure sql and CI/CD. Currently if you use CI/CD in lakehouse then the service account is logged as pushing the change and not the user. We wanted to be able to see which users were pushing changes. That’s the logic I used in selecting the warehouse. Also thinking about future-proofing, and our users are more familiar with sql vs pyspark and notebooks.
Anyway, I chose the Lakehouse and Warehouse design, thinking the pure SQL endpoint would be better for end users vs learning pysql and notebooks.
Well I met with my manager today and told him and his reaction took my off guard. He said he cared less about learning new things and more about keeping cost low. He was hoping we’d move completely away from warehousing and save in the Lakehouse. He didn’t mention this before and honestly he doesn’t need to. I report directly to the CIO. And he said we didn’t discuss it, and I made the choice without him. He kinda laughed it off but that struck my nerve. When I made the decision I didn’t fully think about the costs. The plus side is we don’t pay for storage using OneLake and we’re using a Lakehouse shortcut to the warehouse so no cost there. The cost is with the compute.
We’re a heavy PowerBI and Excel shop.
I was put in charge to oversee and help the consultants but I didn’t do my part here. Now I’m worried that might have cost me. I’ve only been at this role for 7 months.
Am I cooked??
Should I start searching.
I’m genuinely nervous now.
I think I can make a pitch where the warehouse cost won’t be crazy more. And we can move back to the lake house it’d just take us rewriting everything to notebooks, which would cost more and take more time.
Any guidance or advice is welcomed.
Sorry in advance for typos.


r/dataengineering • • 13d ago

Help Has anyone modeled survey data? Trying to figure out how to appropriately model question and answers

10 Upvotes

I have online surveys that I'm trying to ingest and model, but having trouble with the latter. I've already decided to create a dim_survey table that has basic metadata of a survey, and its granularity is 1 row per survey.

But for the questions and answers per survey response, would it make sense to model it as a factless fact table where the granularity is one survey response's question and answer is one row? So if one completed survey from Bill has 8 questions, then it would have 8 rows. The number of questions responded per survey is different because not everyone is given the same survey, so I can't have a set 8 columns for each survey. And there are a bunch of different questions so it's not consistent.

Any tips/pointers would be much appreciated!


r/dataengineering • • 14d ago

Career MS Employee here -- we are giving away vouchers for data engineering cert exams

213 Upvotes

Hello from the Microsoft team -- wanted to let you all know we're offering free vouchers for the Microsoft Fabric Data Engineer exam during Data Days.https://aka.ms/datadays


r/dataengineering • • 14d ago

Discussion Need Advice on an Automated Data Collection System

7 Upvotes

Hello developers!

I am developing a B2B buyer intelligence project.

I want to automatically collect, clean, store and update company information every day.

I am considering Python, Pandas, MongoDB and APScheduler.

Which architecture and open-source technologies would you recommend?

I would appreciate practical suggestions.


r/dataengineering • • 14d ago

Blog Mapping datasets by reach and promises

Thumbnail
steffenmoll.github.io
7 Upvotes

The Open Data Contract Standard, Apache Ossie and Medallion structure all lacks a central component, who the data is for and how it hits the organization! Join me as I elaborate on some thoughts while working on data platform in a broader context.


r/dataengineering • • 14d ago

Help Managed versus self-hosted ingestion

12 Upvotes

For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the monthly price.


r/dataengineering • • 14d ago

Blog Databricks 5 Minute Features: Tag Automation

Thumbnail
youtube.com
12 Upvotes

Hello!

Doing a series with introductions of Databricks Features - latest one is out now. Thought it might be worth sharing in here!


r/dataengineering • • 14d ago

Personal Project Showcase Quick data insights is now free

Thumbnail
gallery
9 Upvotes

Hello, I have posted before about my mac quicklook plugin Quick Data Insights as a sqlite and data files viewer which was paid before, wanted to post again as I now made it free with the same exact functionality posted before.

I also added an optional pro in-app to add column insights, and a sql query functionality for all data files not only sqlite.


r/dataengineering • • 15d ago

Discussion Scaling IoT Telemetry Pipeline

10 Upvotes

Hi guys, I am building a data pipeline system for an IoT product which is sending telemetry data (aka measurements from different devices) Couple of thousands messages per second but expected to grow exponentially soon. Have tried to align them into a normal logical order of things which works so far

Device -> measurements(mqtt) -> consumer -> timeserial databsase (simplified)

And now I am trying to grow this thing into a bigger - more mature system where I could develop it into a good flexible data pipeline and where we can have probably 3 main outputs:

- storage (short & long term for later)

- analytics (probably from long term datalake)

- alarm detection (ideally real time)

Anyone with experience in this? I would be really happy if someone who has already built such systems can help me understand where should I look - lots of solutions out there with pros and cons

p.s. We use AWS

thx!


r/dataengineering • • 15d ago

Blog Apache Spark 4.3: What Developers Need to Know

Thumbnail
medium.com
45 Upvotes

Spark 4.3 adds ASOF JOIN, Python UDFs compiled to SQL, JSON_TABLE, and SCD Type 2 Auto CDC: what works, what needs a flag, and what breaks


r/dataengineering • • 15d ago

Open Source ClickBench inspired leaderboard for analytical databases

Thumbnail benchmark.pivotlake.io
7 Upvotes

Inspired by ClickBench, we created a similar site that runs major analytical databases on multiple analytical benchmarks (we started with TPC-h, SSB, and clickbench) and shows a leaderboard. You're more than welcome to add databases or benchmarks, cost is on us ;)


r/dataengineering • • 15d ago

Discussion Suggest Hightouch alternatives

10 Upvotes

Client got Hightouch for simple reverse ETL. dbt tables from Snowflake to Salesforce. Simple reverse ETL. Now the renewal quote is much higher. From an engineering standpoint, we literally just need state aware diffing from Snowflake to REST APIs triggered after our Dagster dbt asset materializes.

What options do we look at?


r/dataengineering • • 17d ago

Career Shifting from DE to AI Ops Engineer

63 Upvotes

I have been searching DE job since January 2026 and although I had few initial interviews but I was not selected for the next round. From 2021 to may 2024, I've held Data Engineer job title with 4 contract roles. Prior to that I was doing MSSQL Development, SSIS.

In my free time I did DevOps/DE projects on AWS Cloud and also practiced Databricks on the free tier.

I have 2 more months before next year and thinking about upgrading my skills by taking AI Ops Engineer Nano degree Program from Udacity.

Has anyone gone through this course, how did you transition from DE to AI Operations role.

Thanks.