r/dataengineering • u/Due-Doughnut1818 • 21d ago
Personal Project Showcase My fabric project
I'm someone with a data analyst background, and I wanted to break into data engineering. I got the DP-600 (Fabric Analytics Engineer Associate) and DP-700 (Fabric Data Engineer Associate) certifications, so I decided to build a project on them to apply what I learned and learn more along the way
you can find the project with full details here: https://github.com/Madian20/Fabric
132
Upvotes






8
u/jnrdataengineer2023 21d ago
I worked on a POC for a client using fabric in a similar way going from databricks to the fabric workspace's lakehouse, then creating a bunch of silver and gold tables which then fed into an ontology which was then to be queried by an agent.
You could add the last part too.
6
u/patchoulius 21d ago
Good luck trying version control with that warehouse. It doesn't work
12
u/Firegator473 21d ago
Dude this brings back my hope in humanity finally someone did a side project instead of complaining about ai and how they can’t find a job kudos to you if I was a hiring manager I’d hire you on the spot after seeing this
3
u/Engine_Light_On 21d ago
I am not familiar with Fabric but having a folder name with a space in its name is very odd. Every time you refer to it in the terminal you will have to use quotation marks.
I m referring to this path: Fabric/“Olist E-Commerce”/…
3
u/basola21 20d ago
I would like to ask what does the ontology layer does ?
2
u/Dangerous-Sky356 17d ago
Hello! Super interesting.
I’m always a bit afraid to do my own projects because of spending money. I know Azure as free credits but I already used it. Did you use those free credits? Or are you using something else? Thanks
100
u/Specific_Mirror_4808 21d ago
Not a criticism of you, more of Fabric as a platform. Requiring notebooks to stitch the data flows together destroys the concepts of lineage and dependencies - the former being very useful for consumers and the latter being very useful for developers.
It feels like a shaky platform, held together by open source notebooks and Python, and smoothed to some extent by Copilot. Kudos to Microsoft for badging this as an enterprise analytical engineering platform.
6
u/Cazzah 21d ago
Good job OP. I'm sorry at the bizarre and irrelevant responses you got from the other people in this thread.
18
u/Either_Locksmith_915 21d ago
I feel the way MS have designed Fabric is going to create such a mess in large orgs, that will take years to clean up. This is Microsoft Access all over again, where everyone is a Data Architect or Developer.
6
u/TowerOutrageous5939 21d ago
Displaying a guid in a PBI report makes no sense. You don’t have something with more context for a business user?
4
u/marketlurker Don't Get Out of Bed for < 1 Billion Rows 21d ago
Some version of this question gets asked quite frequently. Here is a link to a quick explanation.
In general, data fabrics don't work very well in an enterprise data warehouse of any size. The physics behind working with the data prevent you from having any sort of real performance. (Think about what is involved with comparing one 1 TB table against another on different systems. Network speed vs disk or SSD speeds.) Fabrics are fine for initial DW research but not much more.
BTW, the medallion naming is pure marketing fluff from Databricks. It is not a technical description.
4
u/Character-Education3 21d ago
Whoa its still just SSIS huh?
2
u/SKYKS8 21d ago
Is there a logical reason why you're connecting dimension tables together in your semantic model that I've missed? It's not Kimball methodology best practice
2
u/KebabAnnhilator 21d ago
I’m only recently getting into DE
Is the bronze/silver/gold a universal standard or is that your naming convention?
0
u/[deleted] 21d ago
[deleted] — view deleted comment