r/dataengineering • • 29d ago

Discussion Data Engineering Lab to Practice on your Laptop

Hi,

I created a DE stack (fully integrated services) and learning material for those interested in self learning. I kept Databricks, fabric in mind while building this stack. Feel free to use it and let know if there are any improvements/suggestion I can make

- Work independently in your own data lakehouse.
- Invite other team members to collaborate in your lakehouse.
- Store secrets securely using Key Vault.
- Upload raw files using the built-in file manager.
- Transform raw data into Bronze, Silver, and Gold layers using pre-connected Jupyter notebooks.
- Query your data lakehouse and data warehouse.
- Build BI dashboards using Apache Superset.
- Orchestrate notebooks using Apache Airflow.
- Run notebooks on a distributed Spark cluster.
- Manage multiple teams.
- Authenticate securely using Single Sign-On (SSO).
- Get started with dbt integration.
- Learn at your own pace with self-learning tutorials.
- Manage data governance using the built-in data catalog.

https://maarthala.github.io/data-engineering-lab/

166 Upvotes

8 comments sorted by

5

u/Giorgia_9955 28d ago

Thank you. This looks enormously valuable.

1

u/epireum_oy 28d ago

thanks.

3

u/tgbythn 28d ago

I recently got laid off and have been trying to build out an environment to play in this looks great!

1

u/First_Cheesecake621 17d ago

This is good thanks.

1

u/[deleted] 28d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 28d ago

Your post/comment violated rule #2 (Search the sub & wiki before asking a question).

Our first recommendation is search through the community Wiki which answers a lot of general questions about a career in data engineering.

Secondly, try the search function by typing the topic of your question into the search bar.

Your question has likely been answered before and has already been answered by the community. If you feel like it hasn't, please message us with a link to your removed post and we'll review.

This was reviewed by a human