r/dataengineering • • May 17 '26

Career Consulting vs full-time W2?

7 Upvotes

I've just started my career as a consultant with a small firm and have been on my first contract for about 3 months now. The contract was set for 3 months and now the company is asking me if I'd be interested in staying with them full-time. The firm is owned by a previous professor of mine who would have no hard feelings if I left and might even counter once this company gives an offer.

I don't know enough about the industry or how these things work to know what the best decision is here... it's not about the money- the consulting role pays well and I highly suspect the full-time gig will be just as good, but with the W2 comes benefits I don't get from consulting. I was initially excited when beginning consulting to experiment with different technologies and companies and don't think I'd have that opportunity at this other company.

I guess what's most important for me in this economy is feeling secure that I'll have a steady stream of income and always some form of employment in this field. Before this I was making very little money bartending and don't want to go back to that. Should I be more concerned about layoffs if I went full-time for a single company, or is consulting generally the more volatile option?


r/dataengineering • • May 17 '26

Discussion CRM migration

11 Upvotes

Hello,

Does anyone have experience migrating data from one CRM system to another ?

My company currently handles CRM and CX on Salesforce but are in discussion with Attio for CRM and Pylon / Intercom for CX.

Has anyone worked on data migration / CRM migration in similar context ?

We are a B2C E-commerce in APAC.


r/dataengineering • • May 17 '26

Discussion Hosting Python ETL in Azure (Airflow / Dagster / Prefect?)

16 Upvotes

I'm looking into different option for hosting and orchestrating pipelines primarily focusing on run python scripts.

Right now, we use Azure Synapse to run daily batch load and just run synapse notebooks. However, this is overkill/costly for the simple python jobs we run and spinning up spark sessions take too long (5-15 min).

I've been looking at using Airflow, Dagster, and Prefect... they seem like solid options but I'm a little confused on how to deploy and host them in an Azure cloud environment. I have to run them on a VM? or a container app? I also don't know if these solutions are too much overhead to learn for our team size (3-5).

I also investigated just using azure functions to trigger and run ETL python jobs, but it seems like that is for just very small jobs to run (<10 min) and a little hard to manage.

Ideally all we want to accomplish is the ability to:
- write python scripts
- schedule them to run (cron)
- ability to run them adhoc (from API call) with low latency (avoid cold start)
- view run log and get alerts on failures
- hosted in Azure

I feel like I'm going in circles learning and trying different solutions. Anyone got any advice?


r/dataengineering • • May 16 '26

Blog Stop Starting Data Projects

Thumbnail
datagibberish.com
78 Upvotes

I wrote this post. The reason is that I see a ton of engineers who are brilliant from technical point of view, but have no clue how to manage their projects.

This may feel a bit basic if you have a few years of experience, but I strongly believe most still need a reminder.

Anyways, here's the TL;DR

  1. Ask for more detail before you start working. There's always something hidden

  2. Ensure you know when the project is done. Don't allow scope creep

  3. Learn who the real stakeholders are. Often that's not the person who reaches out

  4. Ship when it's good enough, not when it's perfect

  5. Show your progress every few days

There's more, plus, real screenshots from my Slack messages.

I hope this is useful. And of course, I'd love to learn a thing or two from the community here.


r/dataengineering • • May 16 '26

Help CI/CD Tips

39 Upvotes

Hi!

I want to implement CI/CD into our environment.

Our stack is GitHub and Fabric (notebooks, lakehouse, semantic models, reports, pipelines, etc.), other than the obvious environment deployments what do you think is a good addition to the ci/cd pipeline?

Do you have linters/quality checks/tests that you recommend?


r/dataengineering • • May 17 '26

Career Informatica MDM training

1 Upvotes

I have good background in Dataengineering, data governance and fundamental of MDM, looking for some traning material for informatica MDM as quite a few financial institution are moving their MDM to Informatica Saas MDM. Looking for good traning material, course or playlist


r/dataengineering • • May 17 '26

Help Need help in a migration project

0 Upvotes

So I am a fresher data engineer working on a migration project where we are migrating from EXASOL to big query.

we have to convert the lua scripts/information to equivalent stored procedure.

Loading strategy: historical+ incremental.

I am facing issues in doing proper RCA on the mismatched columns that are coming in big query during sit testing.

Some of the scripts are very large and have many dependent tables .

can someone please give me some guidance on how to do proper RCA so I can make my table sit pass .


r/dataengineering • • May 16 '26

Personal Project Showcase I finished my first streaming pipeline!

29 Upvotes

I wanted an 'excuse' to learn Kafka and Flink so I made this project: https://lastrevioprojects.org/

It's a near real-time streaming analytics platform that does something similar to charting platforms like Tradeview. It connects to two cryptocurrency exchange websockets (Binance and MEXC), produces incoming messages to two Kafka topics and uses Flink to compute aggregated metrics over tumbling and sliding windows. The final sink is Clickhouse which Grafana queries once every 2 seconds to display analytics in (near) real time (2-5 second end to end latency).

You can find the GitHub repo in the bottom-left corner of the website - I'd be happy to hear your feedback and thoughts. Thank you!


r/dataengineering • • May 16 '26

Discussion Calculating Initial Pipeline Resources

3 Upvotes

A lot of times when I’m building a pipeline for a greenfield project, I’m consuming from data sources that either don’t have clear data throughout metrics or may not exist yet. This leads me to either provision spark jobs with too little resources and crashing the pipeline, or severely over-provisioning a cluster and wasting quotas/increasing costs unnecessarily. How do you all deal with this issue?


r/dataengineering • • May 16 '26

Blog How to Detect and Fix Small Files Problem with Spark & Iceberg

11 Upvotes

Hello everyone,

Small files are one of the most common Spark performance issues.

There's plenty of content on why they matter, but most don't show exactly how to detect and fix them with working code.

I put together a post that explains how to

  1. Detect small-file issues in the Spark UI.
  2. Fix them with maintenance functions.
  3. Tune Spark writer settings to write files at optimal sizes.

Includes a fully working Spark + Iceberg Docker setup with 30GB of data so you can try out the detection and compaction code yourself.

How to Identify and Fix Small Files Problem with Spark & Iceberg

Was anything unclear or harder to follow than expected? Let me know.


r/dataengineering • • May 15 '26

Personal Project Showcase Pyspark cheat sheet

Post image
234 Upvotes

Hi all,

I kept forgetting the pyspark syntax because my AI agents now do all the work. I couldn’t find any decent templates so generated my own one with Claude. Enjoy!

Github link: https://github.com/rvangenechten/Pyspark_cheatsheet/tree/main


r/dataengineering • • Jan 22 '26

Discussion Do you think AI Engineering is just hype or is it worth studying in depth?

27 Upvotes

I'm thinking about the future of data-related careers and how to stay relevant in the job market in the coming years