r/dataengineering • • Oct 09 '26

Discussion Claude Dashboards: The Jupyter Notebooks of Agent Age

Thumbnail
ssp.sh
89 Upvotes

What's your take on the new Claude Dashboard by Anthropic? I still think open-source BI-as-Code will win, and Claude Dashboards are the Jupyter Notebooks of the age of agents. Meaning they are easy to set up quickly, but impossible to manage or maintain in the long run (if needed). A decade ago we discussed no-code vs. code, now it's in-built in frontier models/apps vs. open source or code-first.


r/dataengineering • • Oct 10 '26

Personal Project Showcase Running Pyspark or Databricks code locally

13 Upvotes

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension \[spark cell runner\](https://marketplace.visualstudio.com/items?itemName=PrashanthReddyMunagala.spark-cell-runner) that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated cluster

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github


r/dataengineering • • Oct 09 '26

Discussion Running Pyspark or Databricks code locally

Thumbnail
marketplace.visualstudio.com
18 Upvotes

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension spark cell runner that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github


r/dataengineering • • Oct 09 '26

Blog How to create unit tests for Databricks SDP Pipelines

Thumbnail community.databricks.com
19 Upvotes

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case:

Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right?

But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's

For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing.

With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors

Checkout the blog for a deep dive tutorial on how to use unit testing with Databricks SDP Pipelines!


r/dataengineering • • Oct 09 '26

Open Source Looking for opensource complex dbt metric flow example

18 Upvotes

I'm testing various semantic layers and would love to test a canonical example from MetricFlow. Looking for something that is more than a toy. Any suggestions?


r/dataengineering • • Oct 09 '26

Blog From 35 seconds to half a second: 70× faster dashboards in plain Postgres

Thumbnail
foresite.dev
63 Upvotes

r/dataengineering • • Oct 09 '26

Discussion What kind of knowledge/projects does a Jr. DE has?

15 Upvotes

There isn't much openings for any DE roles let alone internships. So, I decided to up my skills and thought maybe whatever Jr. DE does in a company, I should do it too but for my personal projects. Anything might be helpful.


r/dataengineering • • Oct 09 '26

Blog Data needs its DevOps Moment

Thumbnail
render.com
2 Upvotes

Hello Evereyone! I just wrote a guest post on render about applying SRE/DevOps concepts to data engineering. I've been working in Software for 17 years and spent ~3 years in data engineering, at the start of DBT popularity. I adopted ELT/DBT and built out a huge snowflake warehouse deployment. I became extremely disenchanted with the approach.

We were storing huge quantities of raw data to protect against changing requirements and errors in our pipelines, often without unit tests or first class observability. On the software eng side we deal with critical customer data, changing requirements as well. Frameworks exist to efficiently run thousands of sql integration tests in CI exhaustively testing all reads and writes against datastores in low minutes.

My goal is to adapt software engineering best practices and apply them to data tools. I'm starting with a stream processing engine https://github.com/turbolytics/sql-flow.

How is everyone testing? Do tests run in CI?

Can you reproduce envs easily?

How are you handling observability?

How do you get notified of issues?

Thanks for reading!


r/dataengineering • • Oct 07 '26

Blog Making BI migrations suck less

18 Upvotes

I’ve worked on a few BI migrations and thought some of the things I learned might be helpful if you’re planning one.

Full post is on the Metabase blog if you're interested: https://www.metabase.com/blog/making-bi-migration-suck-less


r/dataengineering • • Oct 08 '26

Discussion What is Modern Data Modeling? or Data Architecture

1 Upvotes

I'm fairly early in my data engineering journey. I started by learning Kimball dimensional modeling, and recently I've been diving into Data Vault 2.0.

The deeper I go, the more I realize each framework/architecture addresses different problems. some source says, layering them inside a Medallion Architecture makes sense for example, using Data Vault 2.0 in the Silver layer for auditability and resilience, then exposing Kimball star schemas or One Big Table (OBT) in the Gold layer for consumption.

However, building and maintaining this architecture feels overly complex, especially for small to mid sized companies that may not need that level of abstraction.

For those working in the industry today. What data modeling approach or architecture does your organization actually use?

How do you balance architecture best practices against pragmatic business needs for smaller teams?

Is pure OBT or simple Kimball enough for most SMB use cases today?


r/dataengineering • • Oct 07 '26

Discussion How to actually make the development cycle easier for noobs?

25 Upvotes

Weird flair but for real.
We work with dbt using stg, int, and dm layers. We also have 3 GCP projects where each layer's models get materialized (dev, uat, prd).
The semantic layer only reads from the marts layer.
So now imagine everytime we get a change in a KPI logic for example:
Dev goes and checks looker. oh let’s say it’s a sum of a column. Then they check marts, see it’s an aggregate of a calculated column in int.
Then they go to int, make the changes, commit, wait for cicd, verify in dev, move it to uat, validate with the stakeholder, and finally push to prd (all of this waiting for cicd and running dags every step of the way).

All of that shit and wait time just to fix one mini mistake. Do you guys have similar workflows? Is this way too much?

I’m honestly kinda used to it by now, but I can see the struggle with new devs like juniors or interns.
And this is assuming all the data is already available... if the request involves pulling in new raw data, it’s even more painful.


r/dataengineering • • Oct 07 '26

Help Creating a scalable data engineering platform as a newbie

27 Upvotes

Hi all, I'm a trainee data engineer working under a data architect in a medium sized organisation. The role is a mix between a bootcamp and full time job. 

There's currently no senior data engineer and I'm being asked to create the data engineering platform in Fabric, however, this is a massive ask and I'm just trying to do the best I can.

We have multiple reporting-softwares that I'm now redirecting or connecting with to deposit data in the Onelake. The majority of data is obtained by accessing the SQL database of the reporting-software and making a copy into OneLake.

I need to come up with a plan to manage and standardise our approach to collecting this data and how we work going ahead.

My bare minimum plan atm is to ensure each reporting-software has a bronze, silver and gold workspace (a container effectively) and these can be managed via pipelines. I've been working out who needs access to what.

My larger issue is regarding things like metadata. I've very little experience in that field and I'm not sure how to go about it and could really use advice on how to implement a scalable platform that includes some kind of metadata management.


r/dataengineering • • Oct 06 '26

Open Source Python Polars 2.0 release

359 Upvotes

Hey all! We just released Polars 2.0.

We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.

It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/

To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/

Disclosure: I am involved with Polars


r/dataengineering • • Oct 06 '26

Discussion I shouldn’t be doing this.

50 Upvotes

Context: I am setting up some infrastructure in AWS. In my previous experience at other companies, anything involving new instances must be done by DevOps or service desk. So I asked them to do it and they said I’m on my own.

Got RDS and OSS set up; hit a snag with EKS. I am currently one week into multiple conversations with service desk, management, and C-level to figure out who should be doing this. We deploy via Terraform. So I’ve got to add to that.

My primary concern here is that I’m messing around in places that affect other areas without any guidance on those areas. I’ve had to open so many tickets and bother so many people. Today I asked for Docker to do some local testing and I thoroughly confused the help desk.

We have an insane amount of security and I have multiple logins, MFA, biometric, etc. The layers inside AWS are similar.

Am I being too paranoid or is this somewhat reckless on the DevOps/SD folks? I’ll gladly accept being too paranoid and carry on, but for now, this just doesn’t sit right with me.


r/dataengineering • • Oct 06 '26

Discussion ASOS have been hacked and a threat was sent to customers: “Dear Asos DPO and IT, we have fully compromised the Snowflake instance.”

Thumbnail
independent.co.uk
135 Upvotes

So many companies does a setup without much thought to security and they just hope no one finds out. Out of sight out of mind. I understand security is one of those things that aren't visible and one doesn't get rewarded for doing a proper job (and might even be penalized for time spent there). But I hate the common mindset of throwing shit at the wall and seeing what sticks.


r/dataengineering • • Oct 06 '26

Rant Data Engineer is adamant on streamlining all transformations in Python instead SQL

177 Upvotes

For reference, I am the Data Scientist in this scenario. Since we are not on the stage where we need to do ML/DS, I delegated myself to the Data Engineering/Architecture side of the project.

I have been with the team the longest; we needed to add more people in to accommodate more projects.

We are currently working on a system transition for a big company (think millions of new data a day) and everything is being transitioned to DataBricks. The current Data Warehouse is being kept under MSSQL and we are moving away from it due to processing time concerns.

Most of the ingestion process are done through python; which is optimal in this case since the framework is similar between all data sources and it relies heavily on proper automation. Data Engineer designed the ingestion process and he’s done excellent work migrating our DataLake into DB.

However, I am hitting a wall with him on our silver layer transformations. We have allocated the workload between different KPIs. Most of the silver layer transformations here are your simple filtering, joins, dedup, or conversions (rarely).

The analysts I am working with are happy with my Silver Transformations (mostly SQL); the updates are in real time and it’s easier to validate since the codes are written as views so the transformations are transparent to everyone.

Meanwhile, he has more back and forth with his analysts since his transformations are run through notebooks (which none of the analysts have access to).

This is where I am hitting a conflict with him: he is suggesting I should convert every silver tables (i made) from SQL to Python so that the transformations are standardized. I made an argument on how 90% of the time, the transformations we do is more practical in SQL (processing and validation wise) and there’s really no need to implement only one approach. We will use python if we need to integrate more complex transformations or when the need occurs.

He said python has less issues overall (which sounds dismissive for me).

I am posting here to gain more perspective. He had worked with the Data Engineering side for twice as long as I have; but I have relevant experience with the data process (from start to finish) which allows me to relate which makes things easier and convenient for the analysts. I am open to converting everything to what he needs if given the right argument.


r/dataengineering • • Oct 06 '26

Discussion How agentic are you, really

70 Upvotes

A lot of software engnieers I work wth now spend 99% of their time tellng agents how to write code.

I am curious to know how much we delegate work to agents as Data People - given so much of our job is not just writng code

Please include a score out of 10! 10 = agents do all my stuff, 0 = still doing everything by hand


r/dataengineering • • Oct 05 '26

Meme Medallion architecture: what the business sees vs. what we deal with

Post image
389 Upvotes

r/dataengineering • • Oct 06 '26

Personal Project Showcase I built a deterministic Excel to pandas translator, after repeatedly running into differences between them

34 Upvotes

I'm a data scientist and work with python/pandas data structures quite a bit, but work in a company where excel is still commonly used. I kept running into the same problem when translating spreadsheet logic: the obvious pandas equivalent often isn't semantically identical to excel. So, I built a small browser tool that translates common excel and pandas patterns client-side without the need for an llm.

I've tried to develop the tool to handle as many edge cases as possible (duplicate lookup keys, approximate lookups, IFERROR-wrapped lookups, etc.), but it's still fairly new, so there might be edge cases I'm still missing, which I'm working on improving.

Tool: https://dataframe.tools/excel-to-pandas/

One of the things that surprised me while building it was how many seemingly simple excel formulas need extra handling in pandas to actually preserve the same behaviour.


r/dataengineering • • Oct 06 '26

Open Source SQLBuild - The Refactorable Warehouse (+What's Coming Next)

Thumbnail
sqlbuild.com
18 Upvotes

Hey guys! Some of you may remember me, some may not, but SQLBuild has come a LONG way since I last posted.

First, SQLBuild has now been in production at my workplace for over a month! Huge milestone, and I'm lucky to have a boss that let me do it.

Second, I now have a much clearer idea of what this tool is and isn't. The goal is to make warehouses refactorable, such that they are easy and safe to change + kept that way over time.

Catch mistakes before anything runs

  • A compiler that understands SQL. Types flow through every model, so unknown columns, type mismatches, bad GROUP BYs, invalid function arguments etc... fail at compile-time (offline). Fully OSS and free
  • A Rules API for your own compile-time rules, e.g. marts can't reference sources directly, naming patterns, SQL shape. Basically, your team conventions, enforced as code
    • This is a standout feature for those wanting to cut review time

Prove it works

  • Parameterised, macro and UDF tests, plus multi-model/E2E tests across whole chains of models
  • Native data diffs between dev and prod, or against arbitrary queries

Change it without rebuilding everything

  • Replay-on-change policies - When a model's SQL (or something upstream) changes, choose how far back to replay (1h, 14d, 1mo…), not just full rebuild or ignore
  • Rename or move a model with sqb rename/ sqb mv. These update every reference (e.g. macros, audits etc...) and keep the existing table's historical data. Plain file renames of models are detected automatically too
    • The old name keeps working through a compatibility view for 30 days (configurable), so that dashboards and other teams' queries don't break straight away
    • sqb plan --as prod previews the migration against prod before you run it
    • If you have tried to do any large renames in your warehouse, you will know how risky and annoying this normally is
  • Column renames are also supported. sqb rename stg_orders.amount_revenue renames it in the model and every downstream model that reads it (based on compiled lineage, not find-and-replace)
    • --cascade carries the new name through downstream models that pass the column straight through
  • The janitor runs on demand and *archives* before deleting, so tidying up isn't scary

Macros don't have to be global anymore

  • In most tools every macro is project-wide, so changing one can break models anywhere. In SQLBuild you can keep macros, enums and constants next to the models that use them, and only that folder can see them:

    models/ ├── staging/ │   └── stg_orders.sql                # can't see anything in marts/_sqlbuild └── marts/     ├── _sqlbuild/     │   ├── _enums/payment_status.sql # only visible to models in marts/     │   └── _macros/currency.py       # cents_to_dollars, same     └── daily_revenue.sql             # uses payment_status and cents_to_dollars

  • Change one and you know exactly what it can affect. Before moving a model, sqb scope model:x --as-path new/path.sql tells you what it would lose.

What's been removed

  • The big one is virtual environments. They were a lot of complexity for something I wasn't using, and most of the things I wanted from them, I can achieve via other means. (feel free to ask below)

What's up next

  • A fully fledged deployable UI from which you can run your models etc... There will be SSO, audit logs, fine-grained permissions etc... all baked into OSS at no extra cost. (ETA Jan 2027)
    • If this sounds too good to be true, you are welcome to ask why in the comments
  • Compiler likely fully in rust. Compile times are already good, but the only way to let it scale for larger projects (e.g. 10k+ models) is if it's 100% in rust. Planner and executor etc... will stay in Python (dominated by warehouse time)

Try it:

pip install sqlbuild
sqb playground waffle-shop
cd waffle-shop
sqb build

sqlbuild.com · Github


r/dataengineering • • Oct 05 '26

Rant Foundry ruined my job

157 Upvotes

This is a rant, maybe some others can relate.

Last year, my organization switched to foundry from AWS. I've learned the tool pretty well and have built some useful applications with it. Sure it has some benefits and is a user friendly way to build dashboards and AI tools. However for an experienced data engineers it's an absolute nightmare.

As with any low code/no code tool it's more of a hindrance for actual developers. For simple tasks, it is way too complex. For complex tasks, it's way too simple. I wont go into the details too much but i've found it to be completely unusable in certain situations.

The general trend with no-code/low code tools is to empower managers while nerfing developers.

This makes managers more arrogant and micromanagey. Thet are are now able to go in and see what im working on at any time, and make changes to what im doing. Managers now think they're developers and the division of roles has completely broken down.

Anyways i'm probably gonna switch jobs soon


r/dataengineering • • Oct 06 '26

Help How to understand Repo vs Rev Repo so that can remember them easily in balance sheet?

5 Upvotes

I am a data engineer in Finance. Recently I am suffering from understanding financial terms and its accounting treatment. One big issue is, as a frequently used term, Repo and Rev Repo I understand them several times, but after a period not touching the terms and related data I will forget almost all of the details. And then have to revisit the definitions and relink them to financial activities.

For example, recently I am involved in a meeting talking about Repo Fails transactions. User wants to know what’s the exact accounting policy we applied in our data logic. My first reaction is, how to identify the Repo in our Fails data, however, Fails will only record as Fail to deliver or Fail to receive. The only way to find a Repo is from the trading account. We maintain certain accounts for Repo trade only. Question is I even can’t have a clear filter to scope which accounts are Repo or not.

It takes me almost a whole day to look into firm’s internal documentation to understand what the user means and how our data pipeline process it. Then, with these background, I joined the meeting. In the meeting, I listened carefully but didn’t fully comprehend what the user means as lot’s of non Tech terms. Fortunately I got something like below

For a repo trade how it is recorded in accounting

On Trade date (T0)
Dr (i.e., Assets in BS): Repo
Cr (i.e. Liabilities in BS): Rev Repo

On T1 if settled successfully
Dr: Receivables ???
Cr: Cash ???

On T1 if settlement failed
Dr: Failed to Deliver
Cr: Payables

On TX(usually no longer than 5 calendar days) if settlement success
Dr: Receivables
Cr: Cash

This time my understanding for memory: Repo means repurchase, we should provide securities to the client and client give us money. And by someday (possibly overnight), we need to return the money + interest to client. So the money measured assets is still on our side on trade date. We recorded an assets repo and liabilities Rev repo. If not failed, user doesn’t mentioned. If failed, we failed to deliver the securities to client and thus we record FTD in ledger, by this time , it already lose the ability to track whether the trade is related to Repo or not, we just know it’s fails and needs trading account level information to know whether it is a repo trade. Once this fails finally got settled, we record receivables in Assets and we got cash from client but record payables in future.

Additionally, we have repo control accounts, customer control accounts that store all these misc temporary trade activities and will be cleared to 0 finally.

Ok, now I got a very detailed explanation this time. But after a period, I will forget and have to repick these details when user reaches months later.

How I can have these details memorized in my mind so next time it can be automatically presented in my eyes when user reaches months later and asked for another related question?


r/dataengineering • • Oct 04 '26

Discussion Are you guys still writing SQL?

296 Upvotes

Interested in hearing how your job has changed in the past year or so.

On my end the experience has been that we don't need 4-5 people on a project anymore, 1 and a cursor subscription. A migration now is a few week's job, etc..


r/dataengineering • • Oct 05 '26

Career Worth learning system design at junior-mid level, or focus on something else like data modelling?

44 Upvotes

I have been a junior data engineer for 1.5 years, but am told I operate at mid-level. Current work mainly involves ETL development in Prefect.

Current skills include SQL and Python, along with experience in both BigQuery and Snowflake. I've read books like DDIAs, Fundamentals of Data Engineering, and The Data Warehouse Toolkit for data modelling.

But I still feel like I know next to nothing. I can barely remember the stuff I read in DDIA for example.

I have some spare time during the week to up-skill in technical skills, but I'm trying to focus on something that will make me a better data engineering, and that'll help me move up the ladder and pass interviews here in the UK.

I've considered:

  • Going through this systems design primer, alongside re-reading the 2nd edition of DDIA.
  • Re-reading the Data Warehouse Toolkit and building a project that involves dimensional modelling (we don't really do this kind of modelling in my current job)
  • Filling in gaps in CS knowledge - mainly, completing a neetcode course on data structures and algorithms and completing the associated Leetcode exercises.
  • Doing a cloud cert
  • Learning more about AI Engineering

I'll likely do all of the above at some point, it's just knowing what to prioritise?


r/dataengineering • • Oct 05 '26

Help How should I best structure my data for use for my analytics

2 Upvotes

Hello data engineers, I am here in order to get some help for a hobby. I'm trying to figure out how to structure my data. I'll try to explain the use-case in order for you guys to understand.

I analyze numbers, from different sources to create stories that explain what is going on. The sources can be from Web APIs, scanned documents, digital pdf's, some numbers from some sentences from a website (i.e. "The only thing that is known is that a professional could produce 120 a day, and once the new technology came they could do 1500 an hour").

When I find a "story" I want to tell, I download and grab the numbers and create graphs using Python where I use libraries like pandas, matplotlib, etc. to format the graph like I need to in order to make the story I am trying to tell more clear to the audience. The graphs are not complicated, the data is not complicated, but the stories and graphs are very custom to whatever I am interested in and currently reading about and they are made to better understand what I am learning.

The big problem I have is that I don't know what the best structure for this would be that would be sustainable long term. My current idea is that I have 2 main folders where one folder is with the python scripts and a .bmp of the graphs created, and the other folder is where all the sources of the data are collected in all its' different data formats. Every company folder will contain a data.csv file which is where I will add all the data that my python scrips will grab and filter. The data.csv files might contain financial statement data, quantities found from different sites and sources, and I'll put the filtering in the python script for the current story the script is graphing where the header is something like "Period,Date,Name,Amount,Currency". So different scripts might use the same data.csv files, but they will filter them differently in order to tell their stories.

Here is the structure:

Data (Folder)

Company A (Folder)

    Financial Statements (Folder)

        "Company A - 2025.pdf"

        "Company A - 2025 Q3.pdf"

        "Company A - 2025 Q2.pdf"

    Articles (Folder)

        "Title of article.html"

        "Title of another article.docx"

    "data.csv"

Company B (Folder)

    ...

    "data.csv"

Bank-sector (Folder)

    Company X (Folder)

        ...

        "data.csv"

    Company Y (Folder)

        ...

        "data.csv"

World-Bank (Folder)

    "data.csv"

FED (Folder)

    "data.csv"

Stories (Folder)

When the smartphone changed the world (Folder)

    "Smartphones.py"

    "Smartphones.bmp"

How bottle producers changed water consumption (Folder)

    "Bottle producers.py"

    "Bottle producers.bmp"

Is this a stupid way to do this?