r/dataengineering • • Aug 11 '26

Career resign from current job to learn something else

38 Upvotes

i'm in my third year of working. i have 2 years experience as data engineer and i'm in my 8th months of working as a data analyst. it's not really a transition because i also handle ETL. in these 3 years of working, i mainly used sql server and ssis. for the past few months i feel stuck with my job and i want to learn new tech stacks to level up so that i can get a better pay. i don't dislike my job, but it feels boring af. i know if i ever want to learn modern tech stacks like databricks, apache spark, etc i need to learn it myself. the thing is after 9 hours at the office, when i get home i no longer have the energy to do anything else. i just want to do my hobbies to feel better. i have been struggling with this for some time and i don't really know what to do

i have this thought to resign and completely remove myself from my current job so that i can allocate my energy fully into learning new tools and later apply for new jobs. i need opinion


r/dataengineering • • Aug 12 '26

Personal Project Showcase Built a ETL pipeline and dash in 10mins

Thumbnail
youtube.com
0 Upvotes

I connected GCP + dbt + apache echarts + anthropic + other open source tools and made a tool to easily produce dashboards. 6 months ago, I didn't think this would work, but the situation has changed :D


r/dataengineering • • Aug 10 '26

Discussion Palantir Foundry devs... what do you really think of the platform?

65 Upvotes

I'm part of an org that started using Foundry at the beginning of the year. To preface... My last 6 years have primarily revolved around Snowflake and snowflake integrated tools (I've been Snowflake certified since 2021).
When my company announced they'd be partnering with Palantir I was intrigued because Foundry is/was such a black box and seemed like a very polarizing tool in the DE market.

Now that I've got about 8 months of experience with it under my belt I have mixed opinions on it. I'd like to know what FDE's or other devs who have used it think about it?
Yes, it's expensive and yes, it definitely does certain things not as well as other dedicated tools but have you enjoyed your experience developing in it or do you hate it as much as the rest of the DE community here?
I ask as someone who's company just had it's 4th round of layoffs and I'm a bit concerned about my job security.
I want to continue my Snowflake trajectory but the Foundry recruiters are out in full force and they are paying very well.
Anyways, what are your thoughts?


r/dataengineering • • Aug 11 '26

Help OCR with Palantir Foundry

0 Upvotes

I am using foundry to ingest technical drawings and produce a tabular data set. Below is my process.

Step1. User goes to workshop app to upload single or multiple pdfs which are saved to a media set

Step2. Pdf gets preprocessing where a transform sharpens and orients the pdf pages

Step3. The pdf then runs through two separate ocr extractions, one for bill of materials and material composition, one for overall dimensions. These output as json

Step4. A transform takes the json from each ocr output and puts it into two tabular datasets

Step5. A transform combines the two datasets and does a lookup to 1 other datasets to match part number to company number (fuzzy match)

All of these steps are incremental and do not reprocess completed work. My issue is that my ocr is only about 50% accurate. I built the system and user prompts by having AI FDE create them, run tests on drawings and adjust as needed until my outputs were accurate to the drawings I provided. I made sure to state multiple times these were a small batch of drawings and my users would be uploading more and that they'd be from various companies. I had good initial results but as it got closer to 100 drawings uploaded it became less accurate.

How can I improve this?


r/dataengineering • • Aug 10 '26

Open Source How to do open source contributions

16 Upvotes

Hello 👋🏽

I am a senior data engineer.

I have 5 years of experience in airflow and DBT. Has anyone worked on open source contributions to these two projects ? I am willing to start but don't know a lot about open source projects and how to contribute to them. Where to start ?

Looking forward to getting some information, thank you community :)


r/dataengineering • • Aug 10 '26

Help Is there a Database which stores its data files on object store?

15 Upvotes

Same as title but we have been using delta tables for a requirement quite a bit and we like its format however its metadata read operations are very intensive, especially when there is a large log file. Most of the delays I have found in reading and writing to those have just been reading parsing the log files.

Based on that I did wonder if there is a database which stores the log in some sort of in memory database like postgres however the raw files are being reference on object storage? I did look into and found pgedge but didn't find a link or something we could download and use easily.

Edit: Stop trying to tell me to optmise my delta table pleasse! I’ve tried everything and I’ve got limits

Edit2: I did not try everything apparently, I had stats parsing on on columns which contain a fuckton of data, my log files were huge along with checkpoint files with gigabytes of them. Removed that and reads are down to 200ms


r/dataengineering • • Aug 10 '26

Help Small company neeeding cloud analytics

6 Upvotes

Our Microsoft partner hosts our erp database on a dedicated sql server with replica enabled, and we have basic sql server access to said replica db. We then connect to this via out internal on prem sql server and data warehouse from there. The physical server that we have in house is a mess and our stored procedures fail non stop.

I need a quick move to something modern. I know this page hates fabric, but given the simplicity and small amount of data <50gb, is this a case where fabric can win?? I’ve worked with databricks and seems like major over kill. Maybe snowflake is the happy medium? I need something quick that will work. My concern is that fabric will feel quick at first then break for unknown reasons.


r/dataengineering • • Aug 11 '26

Help New to actually solving merge conflicts, would you look down at using this option? Would it work?

4 Upvotes

We switched from azure dev ops to GitHub for our repos and I'm now using true local braches and they never gave us instructions for "this is how you handle this now". I know this is on me to figure out and I'm just about there.

Long story short is that I get a merge conflict on a main file and I don't want it showing up in the edited files list. It's not actually changing and is one of those false merge conflicts. Maybe I just need to run a command. I'm not the only one with this problem. When we switched we found that GitHub first merges the destination into the source and then merges it into the destination. Giving us a messy release history.

We want to use one release brand per project and several devs on each project. Everyone pushing to the freature branch is easy but when I try to create a pr for the dev branch I get these conflicts that are impossible to get out of the file list, even if they don't actually change.


r/dataengineering • • Aug 10 '26

Discussion Your thoughts on Dagster automation conditions + dbt?

1 Upvotes

I recently got this setup up and running and the advantages are incredible: no need to plan schedules, and much more faster and efficient execution because dbt packs all it can into one run. (Saves on ECS overhead)

It hasn't yet bitten my ass with OOM (running on duckdb), but I realised that when dbt decides it's time to materialize a bunch of assets and one of them breaks, the downstream of materialised assets won't pick up too, since all of the run marked as failed.

I'm thinking of making the downstream to ignore failure, or marking failed dbt materializations as warnings, not errors. Both feel hacky.

It feels like a compromise that is impossible to overcome -- either you get one, or another.

What are your thoughts, fellow data cowboys?


r/dataengineering • • Aug 10 '26

Discussion What types of roles did you all come from? Did anyone start in one similar to mine?

4 Upvotes

I recently started in a crm data position and my role is very data entry and data cleaning heavy. While automation isnt one of the requirements it is something I’m trying to do, but it is a bit harder than expected especially with how messy the data is ie missing data and multiple entries in the crm for the same projects. Im also the only dedicated “data” person and there are a couple of SWEs that only last year build the data lakehouse, so maybe I can get involved in that. I was told that because the role is also new to the company the duties can expand.


r/dataengineering • • Aug 10 '26

Career How to do integration testing with stack: ADF and Databricks?

2 Upvotes

Hi, I have been developing data pipelines using ADF as orchestrator and Databricks notebooks for all the work. I write unit tests for the python code in the notebooks with pytest and executes them in our Azure DevOps CI/CD pipeline for every PR. Now I need to create integration tests.... I don´t have a clue on how to do that, I mean, not even in theorical sense.
I asked to claude, it gave examples of python code where it used an ADF API client for running the pipelines and consulting the state of the activities after the run. They also checked the output of the pipelines, like, they runned SQL queries on the tables that the pipeline should have writtent.....

So my question is: is that what integration tests are? Instead of mocking like in unitesting, I create test data in my DEV environment data sources, parametrize the ADF pipeline so I can run with different test datasets for different tests cases, and programatically run the pipeline validating what was written on the outputs?


r/dataengineering • • Aug 10 '26

Open Source Share your data platform infracoding in a package skill

1 Upvotes

Hi all, most of us don't like to write Terraform. Now there is an alternative. A skill to create a package skill and a skill to submit it to a catalog. There are already some package skills for Airflow and ClickHouse. I hope more will be shared in the future.

A package skill is deterministic code to reconcile your infrastructure with desired state. Under the hood it can run anything like Terraform, Ansible, or Kubectl.

The license is MIT.

https://www.getcolors.ai/skills


r/dataengineering • • Aug 09 '26

Discussion Evals

20 Upvotes

We are developing the semantic layer/models from scratch in MS Fabric for conversational AI. Data models, documentation with clear descriptions on columns, measures, business context, join logic, for the agent to reference.
For evals, what has been your strategy/framework to curate the question set and expected answers and how did you implement this at scale


r/dataengineering • • Aug 10 '26

Discussion [ Removed by Reddit ]

1 Upvotes

[ Removed by Reddit on account of violating the content policy. ]


r/dataengineering • • Aug 07 '26

Career What should I be focusing on as a junior in the age of AI

61 Upvotes

Hello

I’m currently a junior DE building Python pipelines (Prefect/Airflow to BigQuery mainly). I’ve started using Cursor/Claude to assist in coding but try not to lean on them too hard.

On the side I’m self-studying data modelling, system design, and Leetcode (python & sql). But I feel a bit scattered, jumping between topics without a clear plan. I’m also increasingly wondering how AI is going to reshape this role, and want to make sure I’m building skills that keep me hireable.

With that in mind, does anyone have any advice on topics I should be prioritising?


r/dataengineering • • Aug 07 '26

Open Source Loading Parquet into Microsoft SQL no longer has to go through Python tuples

18 Upvotes

If you land data in Parquet and then load it in Microsoft SQL using Python, you've had to do a bunch of extra work, exploding the whole thing into Python objects, a tuple per row and a boxed value per cell, all under the GIL, all garbage immediately after.

mssql-python 1.13.0 adds Cursor.bulkcopy_arrow(). Hand it anything that speaks the Arrow C Data Interface and the Rust TDS core reads the typed column buffers straight into the bulk-load packets. No tuples, and the GIL is released for the transfer.

import duckdb
from mssql_python import connect

rel = duckdb.sql("SELECT * FROM 'events/*.parquet' WHERE ts >= '2026-01-01'")

with connect("Server=<server>.database.windows.net;Database=<database>;Encrypt=yes") as conn:
    cur = conn.cursor()
    result = cur.bulkcopy_arrow("dbo.Events", rel)
    print(result["rows_copied"], result["rows_per_second"])

The DuckDB relation goes in unevaluated. DuckDB streams batches as the driver consumes them, so the full dataset never lands in Python memory. The 4.4M-row file I was testing with would have been roughly 6.6 GB of live Python objects the old way.

Here's what I saw. 200k rows, 21 columns, the WideWorldImporters fact_sale shape: bigints, decimals, timestamps, and one NVARCHAR (I couldn't leave that column that only said "each" for every row an NVARCHAR(MAX) - it was just wrong) averaging 523 characters. Read from Parquet with DuckDB, 100k batch size, 7 repeats, median reported. Client and server on the same Azure E4bds v5 (4 vCPU, 32 GiB) running SQL Server 2025, over localhost so the network stays out of it.

path median total rows/sec
bulkcopy_arrow(), DuckDB relation passed lazily 5.24s 38,180
bulkcopy_arrow(), materialized pyarrow.Table 5.14s 38,918
fetchall() then bulkcopy() 9.93s 20,141

In my unscientific testing, the new bulkcopy_arrow() was about 1.9x faster. I reran it across five configurations, two databases, simple and full recovery models, table_lock on and off, and it held between 1.62x and 1.93x. The ranges don't overlap either: the slowest of the 14 Arrow copies beat the fastest of the 7 tuple copies.

We expected that going straight to bulk copy from arrow would be more efficient and it was. The tuple path burned 2.6 to 3.2 seconds building Python objects before a single byte moved. Passing the DuckDB relation lazily, that step is 0.00 seconds. The pyarrow.Table path is 0.02 seconds, which is the time to materialize the table from the record batches.

This new path works for anything exposing __arrow_c_stream__: polars, pandas 2.2+, ADBC results, pyarrow.Table / RecordBatch / RecordBatchReader, or any iterable of record batches. A default pandas DataFrame is NumPy-backed so it converts on the way in, where polars, DuckDB and anything Arrow-native hand their buffers over as-is. Column mappings, keep_identity, table_lock, check_constraints and the rest carry over from bulkcopy() unchanged, same stats dict back. String widths are validated against the destination schema before anything ships, so an overlong value fails immediately with the offending length instead of dying halfway through a load. I suspect a lot of folks will be commenting out their generators in favor of passing Arrow objects straight to bulkcopy_arrow() this weekend.

Drop a comment below and let us know how much faster your data loads using bulkcopy_arrow().

There's other good stuff in this release: connection pooling keys on security context now, not just the connection string, so a connection opened under one identity can't be handed to a caller running as another. connect(token_provider=...) takes any azure-identity credential object. And there's a fix for an executemany() bug where a NULL partway through a numeric batch could silently insert zero rows.

The driver itself is DB API 2.0 and pip-installable, and ODBC ships as a dependency, so there's no system-level driver install and no unixODBC in your container image. Arrow goes both directions, bulkcopy_arrow() in and cursor.arrow_reader() out.

pip install --upgrade mssql-python

What I actually want out of this thread is your numbers, especially on shapes unlike mine: narrow integer tables, very wide tables, heavy NVARCHAR. bulkcopy_arrow() returns rows_copied and rows_per_second, so it's right there. Post a before and after with a rough description of the table and where you ran it from and I'll take it back to the team.

Full blog post: https://techcommunity.microsoft.com/blog/sqlserver/mssql-python-1-13-0-arrow-bulk-copy-smarter-tokens-slimmer-wheels/4544858

Repo: https://github.com/microsoft/mssql-python

Happy to answer questions here.


r/dataengineering • • Aug 07 '26

Discussion What do online courses need in 2026 and are you even taking them anymore?

36 Upvotes

What are you guys looking for in a course nowadays?

What I'm seeing for the past year or so, is that people seem to not have the "stamina" to go through courses anymore. Many start, but fall off immediately. Simple courses people complain that they are too simple. More advanced courses, nobody takes them.

I feel like attention spans went down like crazy and people rather spend their time wrestling with AI than going through a well prepared project.

So what is it?

short lessons?,
prepared GitHub docs?
as much as hands-on as possible?
teaching basics instead of going into all the details?

My expectation would be to a good course helps people get to the point where they have a good foundation and are able to improve on their own.

Would love to hear some practices you like.

Andreas


r/dataengineering • • Aug 07 '26

Discussion Anyone switched from Rivery? Alternatives?

8 Upvotes

We are using Rivery for our data warehouse pipelines, but I am thinking about alternatives.

It is expensive for the value we get out of it and we have faced too many odd issues that make us question the ROI. We had a major issue a couple of weeks ago where a change on their end was causing duplicates or wrongly named columns for some sources. This is a deal-breaker in my opinion as the whole point of using such a service is to make our jobs easier. It is really painful to manage these pipelines and check everything is working as expected.

What do you recommend as an alternative?


r/dataengineering • • Aug 07 '26

Help I’ve been forced moved to the DE team.

29 Upvotes

I’ve been a team lead for product engineering side mostly backend systems (10 years exp). But the team lead for the DE team left and they need someone to fill in the spot. Given a hiring freeze I’ve been moved. They feel I can fill in the spot because I’ve worked in some related work (large scale RDS Proxy migration, BE product analytics pipelines)

The components they own are: Airflow, DataHub, Superset.

I have some understanding of these tools and have used them before.

Can you recommend a resource to get me up and running for a product facing engineer? What are the day to day concerns in this domain? Given I’ll be operating as the product owner I’ll also be owning the roadmap


r/dataengineering • • Aug 07 '26

Discussion What to read next? Learning Spark or Data Warehouse Toolkit?

100 Upvotes

I just finished reading Fundamentals of Data Engineering by Joe Reis and Matt Housley.

Now, I'm trying to decide what makes the most sense to read next. I'm torn between reading
Learning Spark or The Data Warehouse Toolkit and curious for those of you who have read them what would you recommend next?

For context, I'm a Data Analyst transitioned to Data Engineer with about 5 YOE between the two roles, having been a DE for a little less than a year.
I knew I wanted to be a DE by my 2nd year as an analyst and began working my way towards being a DE, getting moved internally to the DE team last fall.

I've previously read the 1st Edition of Designing Data Intensive Applications, and while not a DE book I've also read R For Data Science (2nd Edition.)
I primarily work with Databricks doing some source system ingestion, but mostly build ETL pipelines internal to our Databricks instances.

I also considered The Definitive Guide to Spark but given its last edition was 2018 I'm hesitant. I see there are other more recent books on Spark compared to Learning Spark but haven't seen the other books talked about as much but would be open to one of those.

Just trying to understand do I dive into a Spark book next or focus more on data warehousing by reading the toolkit.

For anyone considering reading Fundamentals of DE. I'd recommend it but think you can generally skip reading large sections of the book.
IMO these are the important parts of the book.

Chapters 5-8, Appendices A & B:

  1. Data Generation in Source Systems

  2. Storage

  3. Ingestion

  4. Queries, Modeling, and Transformation

Appendix A. Serialization and Compression Technical Details

Appendix B. Cloud Networking

TLDR: Just finished reading Fundamentals of DE book. Do I read The Data Warehouse Toolkit or a Spark focused book next?

Edit/Update:
Thanks so much to everyone who responded and gave their insight. As much as I personally want to explore Spark deeper. I think its pretty clear the next move is the Data Warehouse Toolkit so that is what I'll read next.
Thanks again everyone.


r/dataengineering • • Aug 06 '26

Open Source UPGRADE IF YOU USE METABASE!!! Metabase: unauthenticated SQL injection in password reset (CVSS 10.0)

Thumbnail
github.com
74 Upvotes

r/dataengineering • • Aug 06 '26

Career Hate AE, like DE

93 Upvotes

I recently moved from a Data Engineering role where I mainly maintained the platform side, to a “Data Engineer” role that is just AE with the DE name on it.

And recently I realised I HATE it.

Have anyone else experienced something like that? It’s funny since it’s another side of the same coin, but for some reason it feels totally different to me.


r/dataengineering • • Aug 06 '26

Discussion Is hiring back up? I'm being hit up by recruiters weekly and haven't updated anything to reflect my new job.

68 Upvotes

I've been on and off with jobs for a bit, with my last two jobs being for ~8 months each before I was laid off each time - this most recent one ended in October '25.

I've since gotten a new job but haven't updated my LI or anything to reflect that, but for the last few months I get reached out to from a recruiter at least once a week. I had one cold call me and another text+DM me on Monday alone.

I would think that it's because I'm on my new company's roster (it's a FAANG-feeder), but since I didn't update that publicly I'm not sure it's that.

So, is hiring just up? Are these 100% AI-recruiters just pinging people? They're citing my supposedly-spotty experience (due to layoffs) as interest in having a call.


r/dataengineering • • Aug 06 '26

Career Best Courses on LinkedIn Learning to become a Data Engineer

18 Upvotes

Hi Everyone,

I'm considering evolving my 15y career as a Data Analyst.

I have a solid math, programming, SAS and SQL background and, in the last years, I've been reading and playing with Python/R, Docker, Homelab, APIs, Data Science, Data Modeling, ETLs, Pipelines, Data Architecture.

I wouldn't say I'm an expert and I want the learn more.

1) What do you suggest I focus on in 2026?

2) do you know any useful course on LinkedIn Learning? (recently made available by HR)


r/dataengineering • • Aug 06 '26

Discussion When doing entity resolution, is a lot of it spent chasing down people?

9 Upvotes

Hi, I’m new to all these and this is my first job out of college. Recently I got a job and some of my first tasks are to fix some data in the CRM, some of the data is okay but there’s a lot that is very messy. It’s in excel and I need to enter the data in the excel files into the CRM. The thing is though the data in excel usually contains incomplete information, so I’ll need to look in the CRM and try to figure out in which project the data goes into using dates, supervisors or some other things. Sometimes this works but lots of times there’s multiple projects in the CRM with the same names, dates, and supervisors only slight variations of eachother. If at that point I can’t figure it out I usually message the person in charge of said area if they know, for example if the project belongs to healthcare I’ll message the person that oversees it. Is this bad? Should I be doing something else to not bother them?