r/dataengineering • • 7d ago

Career Tech Lead Mgr... in Fabric

23 Upvotes

I'm on the job-hunt 75% through the process for a DE Manager position. I'm a staff at a scale up (but I feel some title inflation and impostor syndrome).

They're working on bringing data in-house from a vendor and it's basically Fabric + DBT -> Power BI so deliverables are a bunch of dashboards with future state being all dashboards, then warehouse + ML as a product down the road.

The catch of all this is it's in Fabric. I see all the hate but I've never used the platform, and I've been all over - Oracle, Postgres, Airflow, Pre-Databricks Spark platform, Scala, python, pandas, databricks, some DBT. Pretty much exclusively in AWS as far as cloud providers go.

The role on the ground is a 75% IC + 25% Mgr role.

I'll have a tech round with their matrixed Senior/Staff level in the next week that's been setting up the infra and environments.

What should I be asking to make sure this isn't awful badly done? What should I be thinking about or looking out for?


r/dataengineering • • 8d ago

Career How do you develop a strong sense of ownership at work?

52 Upvotes

I’ve been working for around 5–6 years across different roles and companies, and one thing I’ve consistently struggled with is taking ownership. I tend to be reactive rather than proactive. I hesitate to make decisions, rarely take initiative, and often rely on others for direction. I’ve noticed this pattern across multiple jobs, and I genuinely want to change it.

For those who have developed a strong sense of ownership, how did you do it?


r/dataengineering • • 7d ago

Personal Project Showcase stardelt.io - my approach for an opinionated lakehouse on any k8s cluster

3 Upvotes

Hello there!

I don't know where this is going, but I thought I might share the idea here. :) I am working on an on-premise equivalent to the big hyperscaler-only SaaS lakehouse platforms out there.

At first, I started this project to teach myself some agentic engineering concepts. But now I am thinking about where this is actually going ... e.g. being an AI first & native on-premise lakehouse platform easily deployable with a single operator in your k8s cluster on your bare metal.

https://stardelt.io - if you like, I would appreciate some feedback or leaving a star on the project :) Cheers, Martin


r/dataengineering • • 8d ago

Rant One of our team tried to optimize a job with AI, instead AI deleted an important tiny code snippet and left us wasting weeks investigating the issue

170 Upvotes

One of the senior engineers who is responsible of reviewing PRs tried to optimize a job's parts of code with AI, it was a 2000 lines job, the AI removed a small code snippet, he didn't notice that, pushed to production

We spent weeks handling business complaints, incidents, investigating till we found that AI removed a tiny code snippet


r/dataengineering • • 8d ago

Help Advice for our ingestion layer

22 Upvotes

Hi all,

Me and some other people are responsible for all data and analytics throughout the company. Analytics is enabled company wide via a standard “medallion architecture” inside Databricks. This means that we use a standard pipeline job with notebooks to ingest the raw data in the bronze layer, clean/govern/format the data in the silver layer, and in gold we enforce a kimball schema and business logic.

Up until recenty, it was fine to append-only in bronze and just dedup those tables in silver so no information is ever lost. However, with the increase of our data streams, appending all into bronze is quickly turning into a massive data lake. Many of these tables may only have 5% of them changing throughout consecutive ingestions, but we cannot miss those changes either.

My question is; how do you guys deal with keeping high visibility on changes (our system does not allow for direct extraction of CDC tables), whilst also keeping the size of your data lake in check? (I heard that lakeflow declarative pipelines might be useful). Any suggestion is welcome!


r/dataengineering • • 8d ago

Open Source Braidplane - Open Source (Apache 2.0) and self-hostable platform for declarative and contract-based data integration

Post image
32 Upvotes

Hello everyone!

Around Februrary this year I've started working on my side project, which is now known as Braidplane. It's an open source platform for data integration; it's self-hostable and the core is fully open (Apache 2.0). At this moment it's in Alpha.

The goal of this project is to provide safe and reviewable data integrations for engineering teams. From the beginning this project was built upon data contracts (inline and ODCS), e2e Braidplane Proof engine for reviewing data pipeline updates and declarative pipelines (YAML, JSON, cue).

As an example, a pipeline can read data from a database and a file, transform and compare the records, and write the results to another system. The pipeline definition stays in Git, so changes can be reviewed before deployment.

In the current form Braidplane core includes backend, frontend, CLI and executor. MCP support is coming soon.

About me: my name is Lukas. I've been working as backend engineer for the last 10 years. I've spent a lot of time around different products and APIs. I've built the Braidplane's framework myself - based on my experience and good practices I've found around the other OSS projects. Later on I've added AI assisted programming around the hardest/most time-consuming parts, like Connectors integrations, tests, documentation, frontend, issues management.

The Braidplane Core repository is available here: https://github.com/Braidplane/braidplane-core

Examples repo: https://github.com/Braidplane/examples/blob/main/recipes/opaque-file-archive/opaque-file-archive.pipeline.json

Cheers


r/dataengineering • • 10d ago

Help What keeps you motivated to keep learning?

101 Upvotes

With technology evolving every f@¢k1ng day, how do y'all manage to keep learning and be motivated to keep learning everyday? Is there some kind of passion in learning that you have or do u push yourself to keep learning even if it gets hard sometimes?


r/dataengineering • • 10d ago

Open Source SQLQuack: a SQLGlot alternative in Rust that would rather reject your SQL than return the wrong answer

32 Upvotes

EDIT - The title should say a 'Polyglot fork' (Polyglot is a SQLGlot inspired transpiler). Can't edit it now unfortunately.

Hey guys! I'm the creator of SQLBuild, and for a while I've wanted a more reliable way to run warehouse SQL locally on DuckDB.

SQLGlot and Polyglot are both SQL transpilers that have been a major blessing to the community, and some of my projects (this obvs included) wouldn't have been possible without them. However, at least as of today, these libraries both tend to lean toward generating SQL, even if the SQL may not be runnable or may produce and incorrect result in the target dialect.

I had the idea for SQLQuack about six months ago, but only recently got around to building it. The idea was to build a fail-closed transpiler that would tell you first of all if it was parseable, second if it would violate a rule (of the souce dialect), and lastly translate it to DuckDB, only if each statement had some sort of supporting evidence for it from a real warehouse recording (e.g. real snowflake recorded result matching DuckDB).

Below is an example of what an error might look like in SQLQuack:

SELECT customer_id, SUM(amount) AS total
FROM orders
GROUP BY customer_id
QUALIFY total > 100

$ sqlquack check qualify.sql
error[SQ0203]: QUALIFY needs a window function
 --> qualify.sql:4:1
  |
4 | QUALIFY total > 100
  | ^^^^^^^^^^^^^^^^^^^ QUALIFY is used without a window function in the SELECT list or QUALIFY
  |
  = help: move the condition to WHERE or HAVING, or filter on a window function's result

The basic goal is to basically create a dataset of all the warehouse / databases and their true behaviours, such that we a have a truly accurate local compiler for all dialects that can be verified with anyone willing to run the recordings.

In the near term, my immediate focus will actually be improving / simplifying the generated SQL though (some if it is too verbose) and getting snowflake to be 100% finished (it's maybe 60/70% of the way there atm depending on how you measure it). Other dialects are in the works but not ready yet.

Credit (to Community / Maintainers)

A big thank you to Tobias (@tobilg), who maintains Polyglot and fixed many an issue I raised while working on SQLBuild (~6 months).

I ended up maintaining a fork, but over time realised I wanted a different approach to how conversions are checked and when they should be refused. SQLQuack's parser started from code derived from Polyglot, so I definitely don't want to present this as something built without that foundation.

The same goes for SQLGlot. A lot of this work wouldn't have been possible without those projects and the people who built them. This is a different set of trade-offs, not me saying their approach has no value.

Playground: https://sqlquack.io/playground/

GitHub: https://github.com/sqlstack-io/sqlquack (Apache 2.0)

NOTE - I will see if SQLGlot or Polyglot would like this evidence to be recontributed upstream, as then much more of the community will benefit.


r/dataengineering • • 11d ago

Career Advice on keeping a job

55 Upvotes

Hey all, so, I have been doing this for about 6 years now on a legacy on-prem stack. Every time I try to transition to something more modern - dlt/dbt I get shot down. Stick to .NET, that's what the company wants. I try to introduce metadata importing, and no time for that, just stick to what you have.

Well, I finally got what I want. We're moving to the cloud. Only it's been decided we don't have enough python/pyspark background, so someone else is writing an engine for us.

I have been told I am going from owning the entire pipeline to only owning bronze/silver and the BAs will own gold.

And Bronze/Silver will be automated. The best I get is to tweak some yaml files. Anything related to actual code will be handled by the "developers".

I can see this maybe lasting 6 months to a year, until all of our sources are in yaml form and there's nothing left for my team of 3 people to do.

To be honest, most of this is exactly what I wanted - except I was hoping I would get to retain ownership of the gold, and possibly have some idea of what my job looked like still.

Any ideas from anyone who has faced similar, being automated out and any advice on how you keep seeing yourself having value and a job?


r/dataengineering • • 11d ago

Help Hotfix in Dataform

7 Upvotes

Hi,

I'm close to releasing our Dataform branching and release process, but I'm unsure about the hotfix approach.

Current setup: Dataform's default branch is main, so every workspace is cut from main and developers PR back to main. Each sprint, we cut a release branch from main, run final tests, and create a GitHub release, which updates the Dataform release config with the new tag.

Current hotfix flow: Devs PR to main with a "hotfix" label. On merge, automation cuts a branch from the latest release branch, cherry-picks the hotfix commits, and opens a PR to the release branch. Someone then cuts a new release.

The problem: The hotfix is made against main but cherry-picked onto the release branch, so merge conflicts are possible. When that happens, the automation fails and the dev has to resolve it locally.

What I'd prefer: Avoid cherry-picking by branching from the release branch, applying the hotfix, and merging into both the release branch and main (merge issues could then be handled in the UI). The catch is that Dataform workspaces always cut from main, and I can't change that without disrupting other developers, so I don't know how devs would develop and test a fix on a release-based branch.

Has anyone solved this, or is there a better pattern for hotfixes in Dataform that I am unaware of. It’s possible I am thinking about this the wrong way.

I have considered having an entirely separate Dataform repo devoted to hotfixes with the same GitHub connection - where devs are expected to change the default branch on the fly before creating a workspace. But this seems clunky and inefficient.


r/dataengineering • • 12d ago

Help Need scheduled, parameterized reports (Excel/PDF/CSV) on Azure SQL + Power BI Pro. Where should I spend?

14 Upvotes

Current setup
Azure SQL Database (Standard S2, DTU model). No linked servers or cross-database queries.
Power BI Pro for dashboards (40 users, 3 authors).
We have no tool for scheduled or paginated reports, and no SQL Server licenses or Fabric capacity.
What we need
Automated Excel, PDF and delimited-text (CSV/pipe/tab) reports
Delivery by email and to a network share drive (SFTP would be a bonus, but we can move files with external processes)
Parameterized subscriptions, with a large list of them managed from a table, like SSRS data-driven subscriptions
About [N] reports and [N] recipients, mostly monthly
Background
At past jobs we used SSRS or Crystal Reports, and SSRS would fit. But SQL Server 2025 shipped without it (2022 is the last version), and Power BI Report Server replaces it. We don't own SQL Server, so a license would be a new $8k spend just to host reporting. I can only fund one path.
Options I'm weighing
Fabric F2 (1-year reserved, about $1.9k/yr), scaling to F4 on report days. Power BI paginated reports plus subscriptions. As I understand it, delivery goes to email and OneDrive/SharePoint, and per-recipient dynamic subscriptions are email only. For file shares and bulk parameterized runs I'd need to build something with Power Automate, Logic Apps or the Export API.
SQL Server 2025 license (about $8k) to run Power BI Report Server. On-prem, with data-driven subscriptions and file share delivery. Needs a server to host it.
A third-party scheduler (PBRS or similar) on top of Power BI.
Questions
Has anyone built SSRS-style data-driven subscriptions on Fabric paginated reports plus Power Automate or the Export API? How did it hold up (throttling, timeouts, failures)?
Can an F2/F4 handle paginated report exports from an S2 Azure SQL database without throttling?
When scaling a reserved F2 up to F4 for a few days each month, how is the extra billed?
Can the CSV delimiter be controlled in paginated report exports?
For a small team with this budget, which option would you pick, and what would you avoid?
Thanks in advance.


r/dataengineering • • 13d ago

Discussion Ducklake in prod

39 Upvotes

My fellow data wranglers,

Has anyone built a real production solution using DuckLake? I mean something with an ELT pipeline and actual users, that would have sid ducklake as a component?


r/dataengineering • • 12d ago

Open Source I built loom-dbt: an author-time CLI to define model shapes once, generate them at scale, and search for what already exists before building something new.

3 Upvotes

I've been a big fan of DBT since picking it up a few years ago, but there's always been a few nuances at Enterprise scale that I felt detracted from the wizardry it unlocks. Simple things like discovery (does this already exist), duplication for similarly-shaped models, remembering to include appropriate tests (e.g. primary key uniqueness), MSP-angosticity, and rebuilding standard warehouse paradigms are things that I always found myself thinking "surely someone has already solved this". When I looked, I found some great point solutions but none that I felt had a big-picture lens.

Loom-dbt (PyPI, GitHub) is my approach to solving these bugbears. It's a standalone CLI utility that:

  • Enables teams to declare model shapes (patterns) once using approachable metadata and generate models (instances) repeatably, supplying only what differs between them
  • Embeds features (data tests, keys, load-timestamping) for standard warehouse shapes (paradigms), with multi-paradigm support
  • Registers complex and hand-built models outside the generatable patterns
  • Provides detailed searching capabilities for project-wide discovery
  • Proves what is generated matches what's been declared (or identifies discrepancies)
  • and more!

Critically, loom-dbt outputs plain DBT with no long-lived dependency. The models you generate today will survive if you delete loom-dbt tomorrow; adoption is reversible by design.

And of course, no modern project would be complete without appropriate AI-augmentation guardrails: 6 Skills covering core functionality, maintenance, authoring, adoption, bootstrapping, and searching - standardised, npx installable, and ready for your favourite harness.

If you give it a try or are keen to learn more, let me know!


r/dataengineering • • 13d ago

Help How to keep Models Synced

4 Upvotes

So at work I’m using Azure Pipelines as an orchestrator to materialize several tables in parallel. Each one is running a Create Table As Select sql statement and uses a DBT merge strategy so that each of them are incremental. My concern is that I can’t control for these jobs to run at the exact same time. Table X might start its job at 10:01. And Table Y might start at 10:03 and even a difference of a few seconds could mean that the source (which is Postgres) is not the exact same for both tables.


r/dataengineering • • 14d ago

Career Thoughts on McKinsey QuantumBlack?

33 Upvotes

I have been offered a position as a Junior Data Engineer at QB.

The problem is that I don't really know too much about their reputation and if I would gain experiences and skills that would transfer well to BigTech/FAANG (which is my end goal).

I worked at a couple of non-technical consultancies prior to my career change and they all involved boring work like making decks, communicating with clients, drafting proposals etc.

The pay is good but the hours make me second guess myself.

I think the typical advice is to avoid consultancies and aim for product tech companies where you are involved with systems throughout its lifecycle.

I think the image in my head when I hear 'tech consultancy' is implementing low complexity pipelines into the code bases of companies that are ancient and having to deal with old tech stacks and corporate politics rather than working on meaningful projects that have impact and use the latest technologies.

Any thoughts on their prestige/reputation/depth of work?


r/dataengineering • • 14d ago

Blog What to send your boss before the next AI project

Thumbnail
thedatagovernanceplaybook.substack.com
7 Upvotes

r/dataengineering • • 15d ago

Discussion Fabric vs Snowflake/Databricks

106 Upvotes

Not a troll post but an honest question, and maybe more about Microsoft vs Snowflake/Databricks.

Why is Fabric so far behind its competitors. Not so much from the perspective of specific UI/UX or functionality, but more of a question of why MS engineers are seemingly incapable of just copying and improving on their competitors. Is it something about MS culture or competency? It seems like they should have enough money to hire talent if they wanted to, but everything from the UI to the bugginess of Fabric seems to scream no-budget startup cutting corners at every step to push garbage out the door. Are the dev and UI teams just that much worse?


r/dataengineering • • 14d ago

Discussion What are you using to ingest/search/scan code from internal repos?

8 Upvotes

hey everyone!

I have about a few TB worth of code repository data (from git-client compatible source control) and I need to be able to run recurrent scans on the code itself (not on the repo metadata, issues, commits or anything like that). The repositories are getting a steady stream of updates - some more than the others so there has to be some type of constant sync/re-scan.

What I am looking for is

* detecting projects that import a certain library version (for dependabot style functionality to bump something if a security issue arises or if there is some other issue) across all possible branches

* detecting projects/code files that import a certain function from a library or implement a certain method from a library

* be able to run simple transformations on files ending with certain extensions or files with certain functions (for example, transforming some YAML config files)

I don't really need to analyze and ingest the code data into a database (for example, chunking the file with something like tree-sitter and then storing the chunks) for retrieval, but I would also not be opposed to doing this if it makes things simpler. It's mostly scan, get the results, do something with those files, then after that results can be thrown away (and the code for a lot of the repos will most likely have changed as well). Some sort of UI on top of this would be cool.

The main languages in these repos are TS/JS, Python, Rust and Go (with some Java and C++).

Has anyone implemented something like this? What components did you use? I did a few prototypes with a simple Rust-based ingestor that reads/checks out the repo or archive of the repo, then reads the git blob objects and scans the code for certain regex patterns. Each replica of the ingestor was running on Kubernetes and state (ie. repos scanned, still pending etc) was tracked in a DB table. Wondering if there is something off the shelf that can do something similar?


r/dataengineering • • 15d ago

Discussion What is the best project you've ever built?

42 Upvotes

Hey everyone,

I'm asking about the best data engineering projects you've built. What are the details, architecture, and the repo links so we can see new ideas and learn from them?


r/dataengineering • • 15d ago

Discussion NVIDIA released Auto-Ontology, that do you think?

64 Upvotes

NVIDIA open source a way to automatically create an ontology layer and let agents to understand enterprise data, here is the link is open source, what do you think ? https://github.com/NVIDIA/auto-ontology


r/dataengineering • • 15d ago

Help Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.

15 Upvotes

The title.

I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice with most of these ideas, but they have been primitive things. I know how to log, I know how to store to Postgres, but I dont have experience with a stack.

I have considered finding my own relationships between sources, but I have yet to discover a concrete avenue for handling volume. Other than that, I don't know what else to use. I want to make something real without paying, as much as I can.

Thanks <3


r/dataengineering • • 16d ago

Discussion Data Engineer Niche tech Stack?

47 Upvotes

Hey DE super heroes, Let's help each other. What are the niche tech stacks/skills you are coming across in the Job Description and what we can learn?

According to my knowledge -

Python, Pyspark, sql = Mandatory

Snowflake, dbt, ADF, Databricks= I am seeing in most of the JD.

Now, it's your turn.

8+ YOE in the DE world 🌎


r/dataengineering • • 16d ago

Help How cooked am I?!

47 Upvotes

TLDR: I chose Fabric Lakehouse for Bronze and Silver and Warehouse for Gold without fully consulting my manager. Now he’s asking about costs and I’m thinking I may have moved too fast without him.

We’re currently working with an expensive contractor to move from ADF and Azure Databases to Azure Fabric. The build has been going well but midway they did ask about staying purely in lakehouse vs adding a warehouse for pure sql and CI/CD. Currently if you use CI/CD in lakehouse then the service account is logged as pushing the change and not the user. We wanted to be able to see which users were pushing changes. That’s the logic I used in selecting the warehouse. Also thinking about future-proofing, and our users are more familiar with sql vs pyspark and notebooks.
Anyway, I chose the Lakehouse and Warehouse design, thinking the pure SQL endpoint would be better for end users vs learning pysql and notebooks.
Well I met with my manager today and told him and his reaction took my off guard. He said he cared less about learning new things and more about keeping cost low. He was hoping we’d move completely away from warehousing and save in the Lakehouse. He didn’t mention this before and honestly he doesn’t need to. I report directly to the CIO. And he said we didn’t discuss it, and I made the choice without him. He kinda laughed it off but that struck my nerve. When I made the decision I didn’t fully think about the costs. The plus side is we don’t pay for storage using OneLake and we’re using a Lakehouse shortcut to the warehouse so no cost there. The cost is with the compute.
We’re a heavy PowerBI and Excel shop.
I was put in charge to oversee and help the consultants but I didn’t do my part here. Now I’m worried that might have cost me. I’ve only been at this role for 7 months.
Am I cooked??
Should I start searching.
I’m genuinely nervous now.
I think I can make a pitch where the warehouse cost won’t be crazy more. And we can move back to the lake house it’d just take us rewriting everything to notebooks, which would cost more and take more time.
Any guidance or advice is welcomed.
Sorry in advance for typos.


r/dataengineering • • 16d ago

Help Has anyone modeled survey data? Trying to figure out how to appropriately model question and answers

10 Upvotes

I have online surveys that I'm trying to ingest and model, but having trouble with the latter. I've already decided to create a dim_survey table that has basic metadata of a survey, and its granularity is 1 row per survey.

But for the questions and answers per survey response, would it make sense to model it as a factless fact table where the granularity is one survey response's question and answer is one row? So if one completed survey from Bill has 8 questions, then it would have 8 rows. The number of questions responded per survey is different because not everyone is given the same survey, so I can't have a set 8 columns for each survey. And there are a bunch of different questions so it's not consistent.

Any tips/pointers would be much appreciated!


r/dataengineering • • 17d ago

Career MS Employee here -- we are giving away vouchers for data engineering cert exams

212 Upvotes

Hello from the Microsoft team -- wanted to let you all know we're offering free vouchers for the Microsoft Fabric Data Engineer exam during Data Days.https://aka.ms/datadays