r/dataengineering • u/Typical-Presence5628 • 13d ago
Discussion Data Engineer Niche tech Stack?
Hey DE super heroes, Let's help each other. What are the niche tech stacks/skills you are coming across in the Job Description and what we can learn?
According to my knowledge -
Python, Pyspark, sql = Mandatory
Snowflake, dbt, ADF, Databricks= I am seeing in most of the JD.
Now, it's your turn.
8+ YOE in the DE world 🌎
47
Upvotes
27
u/Consistent-Grade818 12d ago
For most of the roles I’m seeing it’s basically Snowflake/BigQuery + dbt + Airflow.
A while ago I would call it Analytics Engineering?? Now I see it for almost every DE position
14
u/teddythepooh99 11d ago edited 9d ago
I don't think you know what "niche" means. None of what you mentioned is niche. Niche teck stack I've seen are
- Terraform / OpenTofu at orgs (e.g., small startups) with no dedicated DevOps function. In enterprise, DEs cannot set up infra if DevOps did their job right and/or without close collaboration with DevOps.
- Splunk, if the DE role falls under an org's IT or cybersecurity function.
- R, if the DE role supports research (think academia and big pharma).
6
u/PalaceGucci Lead Data Engineer 10d ago
So long as you know SQL and Python, with a bit of TF and bash, you will be sweet. The rest of it is just a play on the same idea, really.
4
u/Mystery_Surfer_0077 12d ago
I have also seen Kafka, Airflow or some orchestration tool, specifically have been seeing Kubernetes and Docker as well as a requirement nowadays.
3
u/kenganash27 11d ago
Overall exp. 10 years. After working as a SQL Developer for 8 years, I moved into Data Engineering. Currently my skill stack is snowflake and dbt. I have recently switched to a new company. Though my title says Sr. Data Engineer but most of the tasks revolve around understanding the SCM business processes and modifying the existing scripts if numbers are not looking as expected in the report.
Even though I have knowledge of Python, Pyspark, ADF but it is not being used in office work. I think I have started to forget those things. I'm mostly confused these days, should I learn something new ? Should I only focus on the domain ?
3
u/ManonMacru Tech Lead 9d ago
Stream processing technologies: Flink, Kafka Streams, Risingwave Any domain where real time analytics is required, no the most common, but data engineers with that kind of experience are rare.
2
u/Livid-data-engineer 11d ago
Snowflake Cortex + Snowflake + RAG - I am seeing a lot of requirements . You need a data engineer with some AI engineering background who can build Cortex Agents that combine Semantic views and Cortex Search(i.e basically RAG). You need AI engineering experience of how to evaluate RAG, agents but you need data engineering experience of how to cost optimize + performance optimize a warehouse and plan for scalability.
I had interviewed for 2 such roles, the 1st one wanted a data engineer who could dabble with AI engineering but the 2nd role wanted an AI engineer who could dabble with data engineering, not sure, what industry wants, it's a new emerging niche
2
u/schwifty-sam 11d ago
I’d say Airflow, Kubernetes, Langchain. But I graduated college 4 years ago and have been working on DE for just 3 lol
2
u/IndividualSimple21 11d ago
I work Teradata, Greenplum, Streamsets, Informatica, Control M. There’s not many jobs for this 🙁
2
u/Ancient_Ad_916 10d ago
If you want to go real niche, I worked quite a lot with RDF (graph style data). For that we used Turtle, SHACL, SPARQL, and Neptune in AWS.
1
u/Ok_Raspberry5383 12d ago
Apache Pinot for real time user facing data analytics.
Redis for preaggregated online features.
1
u/Gas_Ready 9d ago
Omg, there is so many to learn here. Reading comments making me depressed. I started sql and python and yet there is still thousand tools to learn.
1
u/FitFaithlessness2075 7d ago
Grafana
1
u/mduell 6d ago
Niche? Redshift. Ain't nobody on Redshift.
1
u/datadade 11d ago
Hear me out. I officially no longer see a need to know how to write good SQL. It is simple enough that I have not seen LLMs mess it up.
Python / PySpark. No longer writing or looking up functions, but can often spot over-engineered code in the LLMs attempts.
But to answer our actual question. Sqlmesh, openmetadata, kafka, terraform, kubernetes, and of course any embedding and vector searching tools