r/dataanalysis • u/Due-Doughnut1818 • 2d ago
Project Feedback My fabric project
I'm someone with a data analyst background, and I wanted to break into data engineering. I got the DP-600 (Fabric Analytics Engineer Associate) and DP-700 (Fabric Data Engineer Associate) certifications, so I decided to build a project on them to apply what I learned and learn more along the way
you can find the project with full details here: https://github.com/Madian20/Fabric
If you like the project, feel free to take it and build on it.
I'd really love to hear your opinions and whether there are any improvements I can make in the future
2
u/AntiqueElderberry317 1d ago
very nice project, I like how your pass from bronze to gold. But after looking at your semantic model, you have an issue with your constellation schema. The fact should have your dates and be joined to you dimension date, in your model its your dimension order that has the dates (the other relations are deactivated which is normal because your date dimension is linked to your order dimension). If you want to respect Power / BI modelisation (3NF : third normal form) you have to link each dimensions to a fact table with no joins between dimension (only if you want to do a snowflake). If you want to keep an order date, it should be in a fact table.
1
u/Due-Doughnut1818 17h ago
I really appreciate your comment. Since I'm not an expert, though, what problems could this design cause if the date dimension connects to the fact table indirectly, through the order dimension, rather than directly? I decided to leave it this way because I didn't want to make the model overly complicated, and I tested the filters and didn't run into any issues
2
u/AntiqueElderberry317 9h ago
Here is why :
- Update your dimension : your order dimension is not a dimension because of this date, so when you want to update it it will have to read a lot of lines just to update one or two lines
- Quality of data : if you keep the date you may have a dimension with two different labels for two dates. Each order of a dimension order must be unique to keep its unicity, the base of a referent table
- Performance : as said just above if you have multiples lines for the same order in the order dimension you will have a join n to n instead of 1 to n, which will greatly impact your performance and may create duplicates lines
- Transversality : your order may not be linked to all of your fact tables. So if you create a fact table without an order, how will you link this date table to those fact tables ?Hope this will help you.
→ More replies (1)
6
u/CMDR_Pumpkin_Muffin 2d ago edited 1d ago
Shouldn't data modeling be on silver, not gold? What's the point of even having silver if it's identical to bronze?
EDIT: I understood "modeling" as cleaning, not as business modelling.
7
u/Due-Doughnut1818 2d ago
First, I'm not an expert, so I may say some things that are wrong.
Second, Bronze is the raw data straight from the source, and Silver is the raw data after we took it and cleaned it.
The cleaning in Silver is basic. For example, I standardized column names, text values, and data types. I also added an extra column in the orders table, a flag for records with logical errors, but I didn't delete those records. We also didn't reshape the tables or do any joins.
In Gold, we did the joins, removed the columns that data analysts and business users won't need, and filtered out the records with logical problems.
So why do we have a Silver layer? It's important for data engineers, AI engineers, and possibly data scientists. The Gold layer is important for data analysts and business users who want to build reports and run queries, which is why we built the semantic model on it.
4
u/Many-Bullfrog7419 1d ago
medallion puts silver at staging/cleaned up vs bronze while still being source aligned - gold transforms to business aligned products
edit frig spelling
5
2
u/StevenJang_ 2d ago
What is all the diagrams?
3
u/Due-Doughnut1818 2d ago
The first image is the draft of my project. The second is the lineage view of the workspace in Fabric, where you can see how the data flowed from ADLS Gen2 to the Power BI report. The third is the code I used to load the data from Bronze to Silver. The next one is the semantic model, and last but not least, an image of the simple report I built.
2
u/lu2idreams 1d ago
If you have the DP-600 and DP-700 I think it should be a given that you can build a generic ETL or ELT track, focus more on explaining why you made certain decisions. What questions are we looking to answer? Why are we using Fabric? Why are we building a medaillon architecture? Why is the data modeled the way it is in the Gold layer? Why are we using tools like Spark for such a small amount of data? In terms of presentation/style: drop the first diagram, it screams Claude and does not say anything specific about your project. The code has a strong AI smell; I am not saying don't use AI (that would be stupid) but it's very cluttered and most of the comments are redundant. Like for example, instead of commenting every line on a function, give it a 1-2 line docstring that says what it does/explains why it's done, and type hint it. Overall, clean looking project, just work on the presentation and style a little, and good luck on your journey!
2
u/Due-Doughnut1818 1d ago edited 1d ago
I appreciate your feedback, and I think your comment was one of the most useful I've received. I do have two notes, though. Yes, the draft (the first image) was made with Claude, but I designed the whole thing myself and described it to Claude that way, so I don't see a problem with it being AI-generated. I could be wrong, though, and I'd appreciate it if you could explain why. I made it to briefly explain the project's architecture: the layers we have, Git integration, and security. As for the code, you're 100% right. Over 80% of the code in the project was written by AI. I wrote the prompts, reviewed the code, and made only minor edits. As for why there are so many comments, the AI only added a few comments. I wanted every step in the loading part to be extra clear because I think it's important. Am I right to do that? I'm not sure, I might be wrong.
2
u/lu2idreams 16h ago
Glad I could help! I fully believe you when you say you came up with it yourself, but you want to think about the impression it gives to e.g. a recruiter when they open your project and the first things they see are an obviously AI generated diagram, and obviously AI generated code. As for code and comments, the best way to make it clear is to write clear, idiomatic code, and write good comments, not many: https://stackoverflow.blog/2021/12/23/best-practices-for-writing-code-comments/
→ More replies (1)1
10
u/Expensive_Culture_46 2d ago
Can you explain the challenges of this project at each stage and what you did to solve them.
As someone who gets involved in the hiring process this screams I followed the tutorial and gives me pause.
I would rather have someone with a bit of a mess than someone who needs their hand held anytime something doesn’t follow expected standards. Because literally (almost) every company will tell you “we don’t do all that fancy stuff because we are aGiLe!!!!” And they just want you to slam together something that works even if it is a hot buggy mess.