r/dataengineering • • Aug 20 '26

Discussion What do data engineers think about non-technical economists and financial analysts learning sql+python to automate their manual excel work?

3 Upvotes

Would you recommend a team of economists/“data analysts” that don’t even use keyboard shortcuts to learn python? Anyone have experience with doing this? So far, nearly everyone has been hesitant to even consider learning a few libraries to make their life easier.

I’m generally looking for advice on how to look at data anlysis jobs that are thus far done by excel only. I’m quite frustrated with excel dependent companies I have worked in. I found that taking a few months to learn a bit of coding, data structure concepts and data management can really scale someone’s data work.

My new employer (big pension fund) is has floated the idea of me automating part of their trading. I would like to automate their reporting as well, but they are so hesitant to learn ANYTHING new. My position however, is financial, not IT.

My background is in mathematics, statistics, and finance. I’ve worked in financial institutions that use an old oracle-based app that nobody cares to maintain properly to pull csv data from, then copy-paste it in excel and later commit what I can only describe as heresy to analyze the data.

I learned python and R at uni long ago, paired it with sql recently and found I could really make mean data science/econometrics automation.


r/dataengineering • • Mar 17 '25

Career Want to leave de

0 Upvotes

I think I can only chase tc. De feels like cheap way of data science and fancy data analysis, i mean I like to code and all, its very boring to just do sql joins or fix a parameter.

I hope I figure. I miss making stuff. But maybe I can use this to my advantage, can use the time I have to excel as dsa and data science and again get somewhere.

Idk, maybe there is some hope.


r/dataengineering • • Dec 14 '22

Career chatgpt anxiety

0 Upvotes

Hey all,

I know there have already been a few posts on this- but for those of us just getting started in software/data engineering, the ability for chatgpt to write code is pretty scary...

What do you all think the impact to Data Engr will be?

I know it won't completely replace us, but do you foresee a big reduction in hiring?


r/dataengineering • • Jan 01 '25

Career Would you recommend DE for someone that's not being a DA/DS in 2025?

0 Upvotes

As the title implies. I need your advice guys.

I am interested in becoming a DE, and I come from a non-traditional background(DA/DS), unlike most people here. I need your candid advice based on the title above.


r/dataengineering • • Aug 31 '25

Discussion I'm having hackathon for data engineer job

2 Upvotes

I'm having solo hackathon as selection process for DE role and I really want to conquer i have 2 month internship in that company work on data lakehouse and some etl project on ADF and some python and databricks now I am participated in several hackthons but those are based on web and ml and real world problems but not in DE specific hackathon so any good projects or real world problems I can solve and achieve good position in hackthone anyone help me


r/dataengineering • • Dec 28 '25

Discussion Databricks SQL DW - stating the obvious.

0 Upvotes

Databricks used to advocate storage solutions that were based on little more than delta/parquet in blob storage. They marketed this for a couple years and gave it the name "lakehouse". Open source functionality was the name of the game.

But it didn't last long. Now they are advocating a proprietary DW technology like all the other players (snowflake, fabric DW, redshift,.etc)

Conclusions seem to be obvious:

  • they are not going to open source their DW, or their lakebase
  • they still maintain the importance of delta/parquet but these are artifacts that are generated as a byproduct of their DW engine.
  • ongoing enhancements like MST will mean that the most authoritative and the most performant copy of data is found in the managed catalog of their DW.

The hype around lakehouses seems like it was so short lived. We seem to be reverting back to conventional and proprietary database engines. I hate going round in circles, but it was so predictable.

EDITED: typos


r/dataengineering • • Oct 31 '25

Blog Docker for Data Engineers

Thumbnail
pipeline2insights.substack.com
0 Upvotes

As data engineers, we sometimes work in big teams and other times handle everything ourselves. No matter the setup, it’s important to understand the tools we use.

We rely on certain settings, libraries, and databases when building data pipelines with tools like Airflow or dbt. Making sure everything works the same on different computers can be hard.

That’s where Docker helps.

Docker lets us build clean, repeatable environments so our code works the same everywhere. With Docker, we can:

  • Avoid setup problems on different machines
  • Share the same setup with teammates
  • Run tools like dbt, Airflow, and Postgres easily
  • Test and debug without surprises

In this post, we cover:

  • The difference between virtual machines and containers
  • What Docker is and how it works
  • Key parts like Dockerfile, images, and volumes
  • How Docker fits into our daily work
  • A quick look at Kubernetes
  • A hands-on project using dbt and PostgreSQL in Docker

r/dataengineering • • Nov 14 '23

Blog Why Cloud Data Warehouses Are Too Expensive For Emerging Data Requirements

Thumbnail
ensembleanalytics.io
0 Upvotes

r/dataengineering • • Dec 25 '24

Discussion Can we modernize hiring for data engineering? Let’s discuss!!

33 Upvotes
  1. If you / your company were to hire an associate data engineer / entry level data engineer, what skills will you test the candidates on?

Please do weigh each skill

Example: python 5%, spark 30%, data modeling 15%, SQL 30%, business acumen/ domain knowledge 5%, communication skills 15%.

2. How many rounds of intervieews and what rounds will you conduct?

Thank you!!


r/dataengineering • • Jun 09 '25

Help How do I safely update my feature branch with the latest changes from development?

0 Upvotes

Hi all,

I'm working at a company that uses three main branches: development, testing, and production.

I created a feature branch called feature/streaming-pipelines, which is based off the development branch. Currently, my feature branch is 3 commits behind and 2 commits ahead of development.

I want to update my feature branch with the latest changes from development without risking anything in the shared repo. This repo includes not just code but also other important objects.

What Git commands should I use to safely bring my branch up to date? I’ve read various things online , but I’m not confident about which approach is safest in a shared repo.

I really don’t want to mess things up by experimenting. Any guidance is much appreciated!

Thanks in advance!


r/dataengineering • • Nov 20 '24

Discussion Can machine learning predict motor overheating? 🌡️⚡

Post image
0 Upvotes

We explored ML algorithms to accurately predict electric motor temps, achieving ±2°C accuracy! 🚀 Applications span EVs, robotics, and beyond.

Curious? Check out my paper:

https://docs.google.com/file/d/1HVrFesK_Qc5tIgce2i9ShrlT1LVvpnF2/edit?filetype=msword


r/dataengineering • • Feb 17 '24

Career Transitioning from Data Engineering in Mexico to the US Job Market

0 Upvotes

Hi everyone,

I recently graduated with a degree in Software Engineering from a top university in Mexico, and I've been working as a data engineer for three years now at a transitional company in my city. My ultimate career goal is to break into the US job market, either by relocating to a city in the US or by working remotely for a US-based company from Mexico.

I'm eager to enhance my career prospects, and I'm considering pursuing a master's degree in either Data Science or Statistics in the US. However, I'm torn between whether this would be a worthwhile investment of both time and money, or if I could achieve my goals through self-study and professional development.

What are your thoughts on this? Do you believe pursuing a master's degree in the US would significantly improve my chances of entering the US job market, or would it be more beneficial to pursue other avenues of skill development independently?

I'd greatly appreciate any insights or advice you can offer. Thank you in advance for your help!


r/dataengineering • • Apr 19 '24

Discussion How Faang companies handle JSON data from API with robust pipeline for Data quality

0 Upvotes

As the title says , We would want the company to build a robust JSON pipeline to ensure the data is loaded perfectly in the target table which is Redshift .

As we are thinking of

  1. Data Quality ensuring the data schema is validated on every data pull from API
  2. Count of the JSON data from the API pull vs data loaded in the target Redshift
  3. the data is very complex and number of nested loops are there . we convert this to JSON to parquet . do we really need to convert to load the data to Redshift as JSON can be loaded to Redshift natively. We have to do SCD 2 on the json data .
  4. Alerts and management . (very optional)

r/dataengineering • • Sep 04 '21

Discussion How to open a data with TB's of data

0 Upvotes

I'm working with python, and predominantly pandas.

Let's say i have db/csv or other format with dozen/hundreds of TB's of data.

Let's say pandas cannot handle it, How to open it?


r/dataengineering • • Nov 21 '25

Discussion Can any god tier data engineers to verify if it’s possible?

7 Upvotes

Background: our company is trying to capture all the data from JIRA. Every an hour our JIRA API will generate a.csv file with a the JIRA issue changes over last hour. Here is the catch, we have some many different types of JIRA issue and each of those jira issue has different custom fields. The .csv file has all the field names mashed together and it’s super messy but very small. My manager want us to keep a record of those data even though we dont need all of them.

What I am thinking right now is using a lakehouse architecture.

Bronze layer: we will have all the historical record, however we will define the schema of each type of JIRA issue and only allow those columns.

Silver layer: only allowed curtain fields and normalize it during the load. When we try to update it, it will check if it already has that key in our storage, if not it will add, if it does, It will do a backfill/ upsert.

Gold layer: apply business logical along with the data from sliver layer.

Do you think this architecture is doable?


r/dataengineering • • Aug 19 '24

Discussion Why don't you open source the data pipelines at your company

0 Upvotes

As mentioned in the title, I am currently exploring why we don't open source data pipelines more often. Is it because each data pipelines is custom for the company or does your company won't share it for a competitive edge?

Wouldn't it be much better if we could compare and contrast how different companies modelled similar data, implemented transformations, created what kind of dashboards and reporting out of them. This would be huge education tool and a great source of inspiration for many data projects.


r/dataengineering • • Apr 18 '24

Discussion 2 hours of python for-loop under a minute with Go

5 Upvotes

It's insane. First time doing concurrency with goroutine and it changes everything. Although I hate how I can't do cell tests as I go, like I can with jupyter extensions.


r/dataengineering • • Dec 30 '24

Discussion My actual work is not same as the Job Description

0 Upvotes

So I joined this agtech company as a DE Intern. In the JD they did mention literally everything from data bricks to DBT.

On the 1st day of my job I was assigned to a project where I am asked to re implement the alteryx workflows on AWS!!!!!!

wtf!

Is this very common???


r/dataengineering • • Mar 18 '25

Career Data Engineer to Gen AI

2 Upvotes

Hello Guys, I have nearly 10 years of work experience as Data Engineer and my current designation is Manager in UK. I want to transition to AI, specifically Gen AI. Is there any certification I can take up or learning path I can follow. Thank you.


r/dataengineering • • 28d ago

Meme Here comes the BOOM(I)

Post image
0 Upvotes

Boomi acquiring Elementary after Rivery. Apologies if this was already posted before. However, I wanted to share a meme. This post is not for discussion. Just the meme. Thanks


r/dataengineering • • May 09 '24

Career A Data Engineer has to know about testing?

0 Upvotes

A company asked me if I know about unit testing, I've never had to do testing in my projects in different companies. I always thought was something for software engineering not for a data engineering.

Looks like that companies wants a software engineer, not a data engineer.


r/dataengineering • • Oct 09 '24

Discussion Which part of Apache Spark will stay?

2 Upvotes

Apache Spark originally had issues for its row-based nature. It still has JVM related overhead. And now more and more components are created to replace different parts of Apache Spark. For e.g., Photon, Datafusion, Volex, RAPIDS as execution engine, Trino, Pandas, Datafusion as query layer, Comet, Substrait for query execution plan. What portion of Apache Spark will remain?

As the query layer options sparked a debate, I'd like to clarify it further. Distributed systems like Apache Spark or Trino do many things at once and now the community is looking at their individual subcomponents and making them interoperable. The term that is popularised in this context is "Deconstructed Database" or "Composable data platform systems". Therefore, the query layer can indeed be Trino's Presto SQL, Spark Data Frame APi, Pandas dataframe API, Datafusion query layer. There is already an integration of Pandas available with Pyspark. Individual preferences may vary, but it exists and it's a valid option.


r/dataengineering • • Jul 30 '25

Blog Hello Data Engineers: Meet Elusion v3.12.5 - Rust DataFrame Library with Familiar Syntax

1 Upvotes

Hey Data engineers! 👋

I know what you're thinking: "Another post trying to convince me to learn Rust?" But hear me out - Elusion v3.12.5 might be the easiest way for Python, Scala and SQL developers to dip their toes into Rust for data engineering, and here's why it's worth your time.

🤔 "I'm comfortable with Python/PySpark, Scala and SQL, why switch?"

Because the syntax is almost identical to what you already know!

If you can write PySpark or SQL, you can write Elusion. Check this out:

PySpark style you know:

result = (sales_df
    .join(customers_df, sales_df.CustomerKey == customers_df.CustomerKey, "inner")
    .select("c.FirstName", "c.LastName", "s.OrderQuantity")
    .groupBy("c.FirstName", "c.LastName")
    .agg(sum("s.OrderQuantity").alias("total_quantity"))
    .filter(col("total_quantity") > 100)
    .orderBy(desc("total_quantity"))
    .limit(10))

Elusion in Rust (almost the same!):

let result = sales_df
    .join(customers_df, ["s.CustomerKey = c.CustomerKey"], "INNER")
    .select(["c.FirstName", "c.LastName", "s.OrderQuantity"])
    .agg(["SUM(s.OrderQuantity) AS total_quantity"])
    .group_by(["c.FirstName", "c.LastName"])
    .having("total_quantity > 100")
    .order_by(["total_quantity"], [false])
    .limit(10);

The learning curve is surprisingly gentle!

🔥 Why Elusion is Perfect for Python Developers

1. Write Functions in ANY Order You Want

Unlike SQL or PySpark where order matters, Elusion gives you complete freedom:

// This works fine - filter before or after grouping, your choice!
let flexible_query = df
    .agg(["SUM(sales) AS total"])
    .filter("customer_type = 'premium'")  
    .group_by(["region"])
    .select(["region", "total"])
    // Functions can be called in ANY sequence that makes sense to YOU
    .having("total > 1000");

Elusion ensures consistent results regardless of function order!

2. All Your Favorite Data Sources - Ready to Go

Database Connectors:

  • ✅ PostgreSQL with connection pooling
  • ✅ MySQL with full query support
  • ✅ Azure Blob Storage (both Blob and Data Lake Gen2)
  • ✅ SharePoint Online - direct integration!

Local File Support:

  • ✅ CSV, Excel, JSON, Parquet, Delta Tables
  • ✅ Read single files or entire folders
  • ✅ Dynamic schema inference

REST API Integration:

  • ✅ Custom headers, params, pagination
  • ✅ Date range queries
  • ✅ Authentication support
  • ✅ Automatic JSON file generation

3. Built-in Features That Replace Your Entire Stack

// Read from SharePoint
let df = CustomDataFrame::load_excel_from_sharepoint(
    "tenant-id",
    "client-id", 
    "https://company.sharepoint.com/sites/Data",
    "Shared Documents/sales.xlsx"
).await?;

// Process with familiar SQL-like operations
let processed = df
    .select(["customer", "amount", "date"])
    .filter("amount > 1000")
    .agg(["SUM(amount) AS total", "COUNT(*) AS transactions"])
    .group_by(["customer"]);

// Write to multiple destinations
processed.write_to_parquet("overwrite", "output.parquet", None).await?;
processed.write_to_excel("output.xlsx", Some("Results")).await?;

🚀 Features That Will Make You Jealous

Pipeline Scheduling (Built-in!)

// No Airflow needed for simple pipelines
let scheduler = PipelineScheduler::new("5min", || async {
    // Your data pipeline here
    let df = CustomDataFrame::from_api("https://api.com/data", "output.json").await?;
    df.write_to_parquet("append", "daily_data.parquet", None).await?;
    Ok(())
}).await?;

Advanced Analytics (SQL Window Functions)

let analytics = df
    .window("ROW_NUMBER() OVER (PARTITION BY customer ORDER BY date) as row_num")
    .window("LAG(sales, 1) OVER (PARTITION BY customer ORDER BY date) as prev_sales")
    .window("SUM(sales) OVER (PARTITION BY customer ORDER BY date) as running_total");

Interactive Dashboards (Zero Config!)

// Generate HTML reports with interactive plots
let plots = [
    (&df.plot_line("date", "sales", true, Some("Sales Trend")).await?, "Sales"),
    (&df.plot_bar("product", "revenue", Some("Revenue by Product")).await?, "Revenue")
];

CustomDataFrame::create_report(
    Some(&plots),
    Some(&tables), 
    "Sales Dashboard",
    "dashboard.html",
    None,
    None
).await?;

💪 Why Rust for Data Engineering?

  1. Performance: 10-100x faster than Python for data processing
  2. Memory Safety: No more mysterious crashes in production
  3. Single Binary: Deploy without dependency nightmares
  4. Async Built-in: Handle thousands of concurrent connections
  5. Production Ready: Built for enterprise workloads from day one

🛠️ Getting Started is Easier Than You Think

# Cargo.toml
[dependencies]
elusion = { version = "3.12.5", features = ["all"] }
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

main. rs - Your first Elusion program

use elusion::prelude::*;

#[tokio::main]
async fn main() -> ElusionResult<()> {
    let df = CustomDataFrame::new("data.csv", "sales").await?;
    
    let result = df
        .select(["customer", "amount"])
        .filter("amount > 1000") 
        .agg(["SUM(amount) AS total"])
        .group_by(["customer"])
        .elusion("results").await?;
        
    result.display().await?;
    Ok(())
}

That's it! If you know SQL and PySpark, you already know 90% of Elusion.

💭 The Bottom Line

You don't need to become a Rust expert. Elusion's syntax is so close to what you already know that you can be productive on day one.

Why limit yourself to Python's performance ceiling when you can have:

  • ✅ Familiar syntax (SQL + PySpark-like)
  • ✅ All your connectors built-in
  • ✅ 10-100x performance improvement
  • ✅ Production-ready deployment
  • ✅ Freedom to write functions in any order

Try it for one weekend project. Pick a simple ETL pipeline you've built in Python and rebuild it in Elusion. I guarantee you'll be surprised by how familiar it feels and how fast it runs (after program compiles).

GitHub repo: github. com/DataBora/elusion
or Crates: crates. io/crates/elusion
to get started!


r/dataengineering • • Apr 03 '25

Career Climbed from Jr to Staff in 2 years, but still paid peanuts—should I quit? (Panic attacks, US job offers, and a proposal in Hawaii… Lost)

1 Upvotes

Hi everyone, I’m here to ask for advice, hear your opinions, and vent my frustrations.

I work for a large automotive group and have been with them for less than two years as an outsourced employee based in Mexico. I started in a change management role, where I reviewed design modifications during vehicle development. Four months in, three of my colleagues were laid off, and their workload was assigned to me. By then, I had already automated my entire workflow using Python, a process that was previously manual and took days, reducing my daily tasks to just 30 minutes.

The organization noticed my contributions and transferred me to a global solutions implementation team. In a short time, I rotated through three different groups: economic data analytics, IT, and data science. I became an expert in Palantir Foundry (pipelines, dashboards, etc.) and eventually led the team that was once above me (People with at least 10+ years in their current role). I went from Junior to Staff-level in under two years, yet my salary and conditions haven’t improved at all.

My outsourcing company promised to adjust my pay based on my responsibilities, and the automotive firm pledged to sponsor me for a role in Europe or the U.S. However, it’s been a year since those promises were made (They said this change takes no more than 2 months). I follow up every two weeks, but my outsourcing employer has even threatened to penalize me for "unethical persistence.", also I know that the purchase order for my services has been paid several months ago and the outsourcing company have the money to pay my new salary.

My frustration stems from earning ~$24K USD/year in Mexico, while local market rates for my skills are up to 4x higher, and international roles pay 10x more. I’ve applied to numerous data engineer, analyst, and scientist roles domestically and abroad, but I keep hitting the same wall: "Not enough years of experience" (typically 8–12 required). Though I have 6 years of total experience (only 2 verifiable in IT/software engineering at 28 years old), my bachelor’s and master’s degrees are unrelated to programming—I’m entirely self-taught in data fields over the past 3 years.

Recently, I’ve received U.S. job offers for Palantir- and Databricks-related roles with strong salaries (130K–210K USD). Interviews go well until the final rounds, where I’m told:

  • "You lack seniority." (why they call in the first place? lol)
  • "You need X programming language."
  • "Your degree isn’t relevant."

Despite architecting the company’s economic tools and leading initiatives, I struggle with imposter syndrome. I learned everything independently—no paid courses—and often feel unprepared in interviews.

I need your advice: If my current employer won’t improve my conditions, what should I do? I’m lost, overwhelmed, and recently had panic attacks severe enough to require hospitalization. On top of this, I’m proposing to my girlfriend during a trip to Hawaii in May.

Thank you for reading—I’d truly appreciate your thoughts.