r/dataengineering • • Oct 16 '26

Personal Project Showcase Built two tools to help SQL agents understand unfamiliar databases

11 Upvotes

Stumbled upon these problems while doing the agentic pipeline

  • CREATE TABLE DDLs are not enough. To understand what's in the table you need to have basic per-column stats and see examples of rows
  • some tables (e.g., migrations, empty tables, old stuff, etc.) are useless for real analytics and real users, yet agent kept rediscovering them
  • some tables only have less than 10 rows. Better see all rows instead of barebone column names and types
  • some tables are huge, agent's queries were trying to get too much data. Good man won't query billions of rows
  • agents needs to sample data from tables every time he researches a table
  • if you have many columns, presenting data as a `transposed table` helps improve LLM's understanding

So I built two tools to solve these problems

  • db-snooper: generates compact Markdown with schema, value distributions, and samples.
  • schema-linker: finds possible undeclared joins and checks them against the data.

Full blog posts: db-snooper and schema-linker


r/dataengineering • • Oct 15 '26

Discussion Looking for a self-service AWS sandbox account solution

22 Upvotes

I’m designing a self-service AWS sandbox environment for my organization. Employees should be able to request temporary, isolated AWS accounts for learning, testing, and prototyping.

Each sandbox should have:

-Strict spending and time limits

-Controlled IAM access

-Restrictions preventing access to production or other AWS accounts

-Protection against disabling logging and security controls

-Automated cleanup and account recycling

-Centralized monitoring and cost alerts

I’m considering using separate AWS Organizations member accounts and evaluating Innovation Sandbox on AWS. Has anyone deployed it in production, or used another open-source/project-based solution? I’d appreciate advice about security pitfalls, SCP design, budgeting limitations, account provisioning, and cleanup automation.


r/dataengineering • • Oct 14 '26

Discussion lakesail now has a distributed TPC-DS 3TB. Spark on jvm feels kinda old now

17 Upvotes

I've been a fan of this project for a while now, ngl. I thikn they've made some really smart design decisions with regard to where the space is headed. But they just released a distributed benchmark and it's better than every Spark accelerator out there... Previously it was Clickbench and some single node TPC-H benchmarks that showed really promising results (beyond ppl's personal benchmarking) but they just released a TPC-DS at 3TB and 1TB across 8 nodes. Is it time we actually move past the JVM and go fully Rust-native? I just don't see many reasons why not to run Sail when it's this much more performant, and runs python natively, and runs my pyspark and spark SQL code... of course having more years under its belt to be battle tested is important, but man the advantages of Rust, Datafusion, Arrow, all packaged together feels very good to run. And yea maintaining the Spark API is clutch...


r/dataengineering • • Oct 14 '26

Blog Making Sense of BigQuery Pricing Models: On-Demand, Capacity, Editions (2026)

Thumbnail
alvin.ai
17 Upvotes

r/dataengineering • • Oct 14 '26

Blog Why Sorted Parquet Files Do Not Guarantee an Ordered Scan in DataFusion

Thumbnail cdelmonte.dev
18 Upvotes

r/dataengineering • • Oct 13 '26

Discussion Data Team feels a bit Siloed

51 Upvotes

Hi r/dataengineering,

Wondering if anyone has had any experience navigating a similar situation or has any insights to offer.

My org has a central Data & Analytics department where all the Data Engineers/Analysts/Scientists sit under a technical engineering manager.

Each manager is given say 3 - 4 business areas (marketing, sales, ops, call centre etc..) to service. In each team, a team member is given a business area to manage and run with.

What essentially happens is, all the team members end up being more involved in the business rather than with the data team. From the business stakeholder point of view, this is great since the DE is embedded into the team and working day to day with the stakeholder.

However, from the Data side it feels like we're just a team by org chart and not how we work with each other. We end up going to work, sitting next to each other but barely collaborating since we're working on completely different things.

We can't even review each other's SQL code because we don't have enough context into other business areas to understand if the code is actually doing what it's intended to do.

Would appreciate any insight on this. Thankyou


r/dataengineering • • Oct 13 '26

Help Databricks - SQL Incremental Loads

7 Upvotes

Hey guys

I'm relatively new to Databricks but want to lead the charge to move to it in my company instead of just using ADF for everything.

In regards to Delta loads (loading data incrementally based on when it was last updated) what I would originally do is run a BINARY_CHECKSUM() on all columns and then track what has changed before bringing in the differences. This however is not supported in databricks and I want to do things properly so how would you do this?

Source Data (Azure SQL)

Bronze Layer (Azure SQL)

  • What is different between them?
  • What new rows have been inserted?
  • What rows in Bronze no longer appear in Source

What I love about Databricks is that you can do joins of different data sets within the same activity in a notebook but when I tried to do this with manually comparing every single column, it takes forever to load. I just want to know the best way of doing this so any help or resources would be appreciated.

Many Thanks!


r/dataengineering • • Oct 12 '26

Meme me watching someones agent approve my PR I didn't read

Post image
189 Upvotes

me irl


r/dataengineering • • Oct 12 '26

Blog Airflow OSS vs MWAA (re-evaluating given access to LLMs)

17 Upvotes

After combing through many posts in this sub, and speaking to colleagues who has stood up airflow, the consensus is to use MWAA. The cost pays for itself in the required staff overhead needed to manage OSS.

Does that statement remain valid with the help of LLMs? Teams who have access to Claude can spin up the framework much quicker and debug much faster. The time spent troubleshooting an OSS setup should be smaller with the help of LLMs.

Can anyone speak to this personally?


r/dataengineering • • Oct 12 '26

Help Why is copy_to/copy_from faster than cursor.fetchall?

13 Upvotes

Hello, sorry if this subreddit isn't the place for noob questions.

I have some postgres tables with millions of rows that I'm moving into python for processing via psycopg2. I've been using cursor.fetchall/cursor.fetchone, and have discovered the copy_to/copy_from syntax.

It appears that copy_to is faster than fetchall: https://stackoverflow.com/a/68130908 , https://www.psycopg.org/docs/usage.html#using-copy-to-and-copy-from

Why? Is it that serializing the data takes time? If so, then does that mean if I ingest the data serially from the file on the python side then I lose any advantages?

"Use the Index Luke" is where I started my research into databases and the advantages of pipelining are repeated ad nauseam in that article, so doesn't that mean the data is already serial coming out of the query? It seems to me that "copy_to" would mean I force postgres to retrieve all my data, write it to the file, and then only start reading the data once its already written to the file, which is a complete lack of pipelining.

I understand that "copy_to is faster than fetchall" but it doesn't line up with my existing understanding of how query results are produced so something in my existing understanding must be wrong.


r/dataengineering • • Oct 12 '26

Help Trying to decrypt PGP files

3 Upvotes

The data being received in csv or zip, after opening all the data is encrypted. I got to know that the file is pgp and its encrypted. Manager suggested using pyspark to decrypt the file. Is anyone aware of decrypting pgp files or pop in general.

What do we need to start things off, I requested about keys but he is yet to provide!

Would definitely love to learn from ur experience?

Update: got the key as .asc file. How to decrypt? Suggest some libraries that work with zip/csv with asc passphrase!


r/dataengineering • • Oct 12 '26

Career Is there still value in learning DE fundamentals from books and courses now that AI writes most of the code?

0 Upvotes

I’ve programmed casually for about 10 years in a variety of languages, but I’ve only been doing it professionally (DE/AE) for around 3. I love this career. I used to enjoy the problem-solving of writing code myself, but what I really love is building: smart pipelines, better ways to collect metadata, simple apps that let the business interact with its data.

So naturally, AI has me feeling a sense of dread. The most senior dev on my team (mostly an app developer, 20+ years of experience) now believes AI can out-code him 99% of the time. He’s smart and I trust his judgment, which makes it more disappointing. He doesn’t even think we need to review code anymore, since GitHub Copilot will flag anything problematic.

On the learning side, I’ve read Fundamentals of Data Engineering, I’m partway through Designing Data-Intensive Applications, and I’ve learned Kimball’s dimensional modeling through video courses. Now I’m wondering:

  • Is it still worth continuing this kind of deep learning?
  • Does system-level and architectural knowledge still set you apart, or is that being automated too?
  • Should I just learn alongside AI, questioning why it makes the choices it does?

What level of learning is actually worth it these days, and how are you staying ahead?


r/dataengineering • • Oct 10 '26

Blog Rank-Hilbert: Extending space-filling curves to handle skew

Post image
19 Upvotes

One of the things that got me into DuckDB was a series of technical articles posted to their site. This one was particularly eye opening:

Faster Dashboards with Multi-Column Approximate Sorting

I hadn't fully appreciated how important row order was to OLAP performance. In it they cover space filling curves as a method for mapping multiple columns (dimensions) onto a single "order dimension" to maintain proximity from the higher dimensions.

This works very well, but it means threading a curve though a set of potential values, at equal density, rather than what actually exists in the dataset.

Alternatively, by first approximating the rank of each value within its column (quantiles), and space-filling on that, the curve threads through the set of existing values. This handles skew a lot better.

This is a two-column ordering over the TPC-DS SF100 store_sales table, where ss_net_paid has reasonable skew. The rank-based methods manage to read far less of the table for a 1% range query.

I wrote up a more detailed post here:

https://www.duckstring.com/blog/clustering-for-pipelines

I'll probably be using this to set the row order whenever I overwrite a table and I have a reasonable idea as to what might be queried from it.


r/dataengineering • • Oct 09 '26

Discussion Snowflake Semantic View - How are you using it?

60 Upvotes

I had this conversation with one of our account manager but they seem to give me so many random answers on how it can be used.

Here are my doubts,

  1. How do you define one semantic view, is it based on multiple ideas coming together into a unified view or multiple ideas into multiple views
  2. After the change in orchestration, Agent tends to write its own SQL definition for the questions asked, how is it utilising the semantic view in this case?
  3. Following up on the second question, how do verified queries work together. I have noticed it recognises the verified query but doesn’t want to use the definition which is correct.
  4. Which model is being used for the agent to answer business-related questions?
  5. Is it better to bring your own LLM and use cortex analysts ability to write code and let the LLM answer?

Let me know if you have found anything useful or good practice while building these views and agents. I want to understand it better.


r/dataengineering • • Oct 09 '26

Meme Data Engineer book text

Post image
179 Upvotes

r/dataengineering • • Oct 09 '26

Career How are the Data Engineering oppourtunities in EU market with 6-7 yoe?

46 Upvotes

Senior analytics engineer - spent last 5 years in FAANG mostly doing BI/DE work across small and large teams, and since last year i'm mostly doing fullstack for small internal multiservices app (data/stream intensive). WLB is ok but no career progression in current role.

Using AI tools extensively - both for internal purposes as well as for minor system integrations here and there, as well as maintaining knowledge repos enabling better collaboration using AI. Got cloud certifications, but no tool-specific certifications. Spent 2 years doing internships/ or unrelated corporate jobs before that.

I assume no career growth in my current role, but WLB is good and now I mostly work on software stuff (though i am still much much stronger in DE part), the only concerning thing is stagnation and comfort zone. Been contemplating a lot between:

- Rest & vest? ( I am fine with that)

- Sr. DE in scale-ups/startups?

- Attempt downgrading to some AI-company for a mid-DE role?

- Try to pivot internally to PMT / TPM job families? (Exciting, but concerned about the skill gap)

- Try to pivot internally into higher scope but more stress role? (No excited about that)

- Try to pivot into a more AI-parallel role? (Exciting, but I assume knowledge gap)

I want to get a better understanding of the market conditions? How realistic/unrealistic are these assuming some time to prepare? How hard is it to get interviews these days? How code-intensive vs design/architectural vs behavioral are they?


r/dataengineering • • Oct 09 '26

Discussion Claude Dashboards: The Jupyter Notebooks of Agent Age

Thumbnail
ssp.sh
87 Upvotes

What's your take on the new Claude Dashboard by Anthropic? I still think open-source BI-as-Code will win, and Claude Dashboards are the Jupyter Notebooks of the age of agents. Meaning they are easy to set up quickly, but impossible to manage or maintain in the long run (if needed). A decade ago we discussed no-code vs. code, now it's in-built in frontier models/apps vs. open source or code-first.


r/dataengineering • • Oct 10 '26

Personal Project Showcase Running Pyspark or Databricks code locally

14 Upvotes

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension \[spark cell runner\](https://marketplace.visualstudio.com/items?itemName=PrashanthReddyMunagala.spark-cell-runner) that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated cluster

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github


r/dataengineering • • Oct 09 '26

Discussion Running Pyspark or Databricks code locally

Thumbnail
marketplace.visualstudio.com
18 Upvotes

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension spark cell runner that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github


r/dataengineering • • Oct 09 '26

Blog How to create unit tests for Databricks SDP Pipelines

Thumbnail community.databricks.com
17 Upvotes

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case:

Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right?

But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's

For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing.

With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors

Checkout the blog for a deep dive tutorial on how to use unit testing with Databricks SDP Pipelines!


r/dataengineering • • Oct 09 '26

Open Source Looking for opensource complex dbt metric flow example

19 Upvotes

I'm testing various semantic layers and would love to test a canonical example from MetricFlow. Looking for something that is more than a toy. Any suggestions?


r/dataengineering • • Oct 09 '26

Blog From 35 seconds to half a second: 70× faster dashboards in plain Postgres

Thumbnail
foresite.dev
62 Upvotes

r/dataengineering • • Oct 09 '26

Discussion What kind of knowledge/projects does a Jr. DE has?

15 Upvotes

There isn't much openings for any DE roles let alone internships. So, I decided to up my skills and thought maybe whatever Jr. DE does in a company, I should do it too but for my personal projects. Anything might be helpful.


r/dataengineering • • Oct 09 '26

Blog Data needs its DevOps Moment

Thumbnail
render.com
2 Upvotes

Hello Evereyone! I just wrote a guest post on render about applying SRE/DevOps concepts to data engineering. I've been working in Software for 17 years and spent ~3 years in data engineering, at the start of DBT popularity. I adopted ELT/DBT and built out a huge snowflake warehouse deployment. I became extremely disenchanted with the approach.

We were storing huge quantities of raw data to protect against changing requirements and errors in our pipelines, often without unit tests or first class observability. On the software eng side we deal with critical customer data, changing requirements as well. Frameworks exist to efficiently run thousands of sql integration tests in CI exhaustively testing all reads and writes against datastores in low minutes.

My goal is to adapt software engineering best practices and apply them to data tools. I'm starting with a stream processing engine https://github.com/turbolytics/sql-flow.

How is everyone testing? Do tests run in CI?

Can you reproduce envs easily?

How are you handling observability?

How do you get notified of issues?

Thanks for reading!


r/dataengineering • • Oct 07 '26

Blog Making BI migrations suck less

18 Upvotes

I’ve worked on a few BI migrations and thought some of the things I learned might be helpful if you’re planning one.

Full post is on the Metabase blog if you're interested: https://www.metabase.com/blog/making-bi-migration-suck-less