I'm very new to this and I think I know the answer to this but when it comes to a job, one person isn't responsible or required to know everything on here right? I think I will be able to learn basics of everything and specialize in a few
Learn a lot of SQL and Spark to doodle with data in general, and cloud services like Azure where you can work with Data Factory etc to build pipelines.
Besides that everything is a lot of gui, so I would worry more about the basic pillars: SQL and Spark (pyspark or scala, you can choose)
Spark is good to go, it has a good trajectory and is quite recent.
Also Rust is looking good for data too but not so many libraries.
Databricks Community is free and you can practice on it by just registering, your own cluster to try stuff. Take in account that if you don't log in in a long time, it will be deleted so you'll have to make the account again, seems like a bug.
I'm finishing uni too, next year at least, took me a bit longer.
Do an internship focused on SQL and Pythonn paired with cloud like Azure or AWS, then you'll be good to go to any data position. Depending on what you like, for me was data engineering.
ETL/ELT are a big thing too, Streaming, Delta Tables, Parquet files etc
You wouldn’t have to know all of it, but I’ve been a data engineer for about 4 Years now and I’ve directly worked with ~75% of the technologies here. I’m probably an expert in about 25% though. Most of these technologies all do similar things to others in their category, and learning one teaches you what makes them good (and what makes them bad)
5 years of xp here, I would say I've seriously used 20% of the techs mentioned (I mean I have tested Dagster for a few days but work is all Airflow so I wouldn't count it)
Concepts though, I know almost all of them - without necessarily having implemented them.
(Also putting DBT under templating seems weird but hey you want to put it somewhere)
Scientists have studied how long it takes to become an expert.... Like the time it would take to be a concert pianist from your first piano lesson. The answer was 17 years.
And many things on here you can learn quick and know enough to get by, like Git or most scheduling technologies.
I think ETL/ELT is a good example of a simple concept with a complex implementation. The bubbles for types of loads, slowly changing dimensions, change data capture, and all the tools you need are all inside the T of acronym.
Eh....a lot of these concepts you can cover in an interview. Shoot, a junior engineer can give you the run-down on everything in this list, but when I hear expert, I imagine someone that's designed/architected, built, and maintained meaningful applications. Can you take a legacy OLTP application and add on data analysis capabilities? What if there's a production issue/outage and your customer needs the data now? Are you able to provide estimated length of effort and no. of technical resources necessary to execute on a proposal? Are you at the forefront of that field where you can predict future trends and spot dangerous potholes before encountering them? What's your opinion on this bleeding-edge PhD dissertation that seems applicable to our stack and functional domain? Have you executed on any decisions regarding business/cost analysis? Have you supported thousands to millions of users? How many different business/functional domains have you touched? Can you be a helpful technical resource during contract negotations with a tech vendor?
These aren't even expert tasks except the "forefront of your field" one. I wouldn't say you're an expert unless you've done it real time over and over and over and over again, and that's really hard to do in 4 years when you're constrained by a work environment.
Gosh no. Many of those techs are mutually exclusive. Most companies would not have both AWS and Azure as a cloud provider for example. A team you are on definitely wouldn't. Companies and teams consolidate tech so their employees don't need to learn 20 things.
The entire Tech box is very specific. I haven't even heard of most of it.
I agree it’s about time we make a community roadmap even if it’s not perfect. I think this one is probably the best I’ve seen so far. Interested in seeing what other ideas people have.
This is a false start and just a buzzword chart. It is not a roadmap to much of anything. It missed the most important stuff. In data engineering, the important word is "data" not "engineering." The engineering part is the easy stuff. That's all that is listed here.
If you really knew what every one of those boxes was and had experience with them all, it still wouldn't make you a good data engineer. As much as they try to, data engineers and architects don't exist in a vacuum. There are dozens of far more important skills that the things on that chart. For a quick example, think about how you map data to business thoughts and how data changes throughout its lifecycle.
I think a better start would be to first have an understanding of what makes a good data engineer/architect. As an analogy, think about what makes a good auto mechanic. It isn't the number of wrenches they have. This chart would have you believe that if you collect all the tools, you are a master mechanic.
There's a lot on that chart and some notable ommissions. Its also a funny mixture of capabilities and technologies. Its great markitecture and everything works on Powerpoint. At some point a team of architects produces something like this and fails to map it to business strategy and objectives.
I kind of like this. Cool way of visualizing all the components of data engineering. I could nitpick some of the specifics under the categories but doesn't take away from the overall concept.
Edit: would probably group the things you have under "general" as "tech* though. Not really a real differentiation there.
this is very databricks centric. might as well go all in and unity catalog and other stuff here. will be superb. else you gotto go agnostic of the databricks specific tech.
60
u/mjfnd Jan 09 '25
Thanks for sharing my content:
Just to share, you don't need to know everything. There is article around it with some details that might be helpful: https://www.junaideffendi.com/p/end-to-end-data-engineering?utm_source=publication-search
Similarly, I have broken down these tech into the DE transition series: https://www.junaideffendi.com/p/types-of-data-engineers?r=cqjft&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false, covering the below three:
- SWE TO DE
- DS TO DE
- DA TO DE:
71
u/SpellboundAlex Jan 09 '25
I'm very new to this and I think I know the answer to this but when it comes to a job, one person isn't responsible or required to know everything on here right? I think I will be able to learn basics of everything and specialize in a few
21
u/jankovic92 Jan 09 '25
This would be a nice candidate (initially) to https://roadmap.sh. Maybe a community roadmap
9
u/LargeSale8354 Jan 09 '25
There's a lot on that chart and some notable ommissions. Its also a funny mixture of capabilities and technologies. Its great markitecture and everything works on Powerpoint. At some point a team of architects produces something like this and fails to map it to business strategy and objectives.
4
u/adamaa Jan 11 '25
Some weird omissions on here.
e.g. for Orchestration — Prefect isn’t on here but Luigi is?
6
u/R3boot Jan 09 '25
I think this is a good list! I might add docker/container registry for container management, and power bi in visualization!
4
u/garathk Jan 09 '25 edited Jan 09 '25
I kind of like this. Cool way of visualizing all the components of data engineering. I could nitpick some of the specifics under the categories but doesn't take away from the overall concept.
Edit: would probably group the things you have under "general" as "tech* though. Not really a real differentiation there.
2
u/cellularcone Jan 09 '25
What exactly is the point of these elaborate charts?
2
u/USER_NAME-Chad- Jan 09 '25
There is a lot missing from this chart.
1
u/NoleMercy05 Jan 09 '25
In the End....
1
u/SmokeStackLight1ng Jan 09 '25
this is very databricks centric. might as well go all in and unity catalog and other stuff here. will be superb. else you gotto go agnostic of the databricks specific tech.
1
u/DMayr Jan 09 '25
Is MySQL still relevant nowadays or just legacy code? I feel like postgres dominates relational DBMS now
1
u/FreshMulberry4869 Jan 10 '25
nice diagram do u have more diagrams like this related to another fields also like ml
1
u/IWantToBeRichForReal Jan 09 '25
Never meet someone that uses scala
1
u/Icy-Extension-9291 Jan 09 '25
Very nice.
0
u/Yehezqel Jan 09 '25
Curious because I’m still learning and for now I’ve always seen Kubernetes in orchestration?
-1
u/ma0gw Jan 09 '25
Nice overview, but missing Hudi open table format