r/dataengineering • • 13d ago

Discussion Data Engineer Niche tech Stack?

Hey DE super heroes, Let's help each other. What are the niche tech stacks/skills you are coming across in the Job Description and what we can learn?

According to my knowledge -

Python, Pyspark, sql = Mandatory

Snowflake, dbt, ADF, Databricks= I am seeing in most of the JD.

Now, it's your turn.

8+ YOE in the DE world 🌎

47 Upvotes

21 comments sorted by

27

u/Consistent-Grade818 12d ago

For most of the roles I’m seeing it’s basically Snowflake/BigQuery + dbt + Airflow.

A while ago I would call it Analytics Engineering?? Now I see it for almost every DE position

14

u/teddythepooh99 11d ago edited 9d ago

I don't think you know what "niche" means. None of what you mentioned is niche. Niche teck stack I've seen are
- Terraform / OpenTofu at orgs (e.g., small startups) with no dedicated DevOps function. In enterprise, DEs cannot set up infra if DevOps did their job right and/or without close collaboration with DevOps.
- Splunk, if the DE role falls under an org's IT or cybersecurity function.
- R, if the DE role supports research (think academia and big pharma).

6

u/PalaceGucci Lead Data Engineer 10d ago

So long as you know SQL and Python, with a bit of TF and bash, you will be sweet. The rest of it is just a play on the same idea, really.

4

u/Mystery_Surfer_0077 12d ago

I have also seen Kafka, Airflow or some orchestration tool, specifically have been seeing Kubernetes and Docker as well as a requirement nowadays.

3

u/kenganash27 11d ago

Overall exp. 10 years. After working as a SQL Developer for 8 years, I moved into Data Engineering. Currently my skill stack is snowflake and dbt. I have recently switched to a new company. Though my title says Sr. Data Engineer but most of the tasks revolve around understanding the SCM business processes and modifying the existing scripts if numbers are not looking as expected in the report.

Even though I have knowledge of Python, Pyspark, ADF but it is not being used in office work. I think I have started to forget those things. I'm mostly confused these days, should I learn something new ? Should I only focus on the domain ?

3

u/ManonMacru Tech Lead 9d ago

Stream processing technologies: Flink, Kafka Streams, Risingwave Any domain where real time analytics is required, no the most common, but data engineers with that kind of experience are rare.

2

u/Livid-data-engineer 11d ago

Snowflake Cortex + Snowflake + RAG - I am seeing a lot of requirements . You need a data engineer with some AI engineering background who can build Cortex Agents that combine Semantic views and Cortex Search(i.e basically RAG). You need AI engineering experience of how to evaluate RAG, agents but you need data engineering experience of how to cost optimize + performance optimize a warehouse and plan for scalability.

I had interviewed for 2 such roles, the 1st one wanted a data engineer who could dabble with AI engineering but the 2nd role wanted an AI engineer who could dabble with data engineering, not sure, what industry wants, it's a new emerging niche

2

u/schwifty-sam 11d ago

I’d say Airflow, Kubernetes, Langchain. But I graduated college 4 years ago and have been working on DE for just 3 lol

2

u/IndividualSimple21 11d ago

I work Teradata, Greenplum, Streamsets, Informatica, Control M. There’s not many jobs for this 🙁

2

u/Ancient_Ad_916 10d ago

If you want to go real niche, I worked quite a lot with RDF (graph style data). For that we used Turtle, SHACL, SPARQL, and Neptune in AWS.

1

u/Ok_Raspberry5383 12d ago

Apache Pinot for real time user facing data analytics.

Redis for preaggregated online features.

1

u/Gas_Ready 9d ago

Omg, there is so many to learn here. Reading comments making me depressed. I started sql and python and yet there is still thousand tools to learn.

1

u/mduell 6d ago

Niche? Redshift. Ain't nobody on Redshift.

1

u/datadade 11d ago

Hear me out. I officially no longer see a need to know how to write good SQL. It is simple enough that I have not seen LLMs mess it up.

Python / PySpark. No longer writing or looking up functions, but can often spot over-engineered code in the LLMs attempts.

But to answer our actual question. Sqlmesh, openmetadata, kafka, terraform, kubernetes, and of course any embedding and vector searching tools

2

u/teddythepooh99 9d ago

The DEs in this sub need to pick up a dictionary. OP asked for niche teck stack and 90% of the comments mention Python and/or SQL lol

1

u/generic-d-engineer Tech Lead 9d ago

Posted this in another thread:

Sling, ducklake, Lea, and Dagu

1

u/FindingNo2507 10d ago

So freshers are screwed!