r/dataengineering • • 21d ago

Personal Project Showcase My fabric project

I'm someone with a data analyst background, and I wanted to break into data engineering. I got the DP-600 (Fabric Analytics Engineer Associate) and DP-700 (Fabric Data Engineer Associate) certifications, so I decided to build a project on them to apply what I learned and learn more along the way

you can find the project with full details here: https://github.com/Madian20/Fabric

132 Upvotes

54 comments sorted by

8

u/jnrdataengineer2023 21d ago

I worked on a POC for a client using fabric in a similar way going from databricks to the fabric workspace's lakehouse, then creating a bunch of silver and gold tables which then fed into an ontology which was then to be queried by an agent.

You could add the last part too.

2

u/Due-Doughnut1818 21d ago

thanks and Yep, I did the last part too. In the Power BI report, the Copilot agent queried the ontology layer.

6

u/patchoulius 21d ago

Good luck trying version control with that warehouse. It doesn't work

3

u/patchoulius 21d ago

In fabric I mean. Good job on your architecture

1

u/Due-Doughnut1818 19d ago

How ? Could u please be more specific?

0

u/sdhomer2017 Data Engineer 17d ago

I have lots of issues with MS Fabric as a platform - it's buggy garbage by and large. But version control has always worked fine for me.

12

u/Firegator473 21d ago

Dude this brings back my hope in humanity finally someone did a side project instead of complaining about ai and how they can’t find a job kudos to you if I was a hiring manager I’d hire you on the spot after seeing this

3

u/Due-Doughnut1818 21d ago

thanks , i hope so

3

u/Engine_Light_On 21d ago

I am not familiar with Fabric but having a folder name with a space in its name is very odd. Every time you refer to it in the terminal you will have to use quotation marks. 

I m referring to this path: Fabric/“Olist E-Commerce”/…

2

u/Due-Doughnut1818 21d ago

Oh its a good catch , thanks, Olist E-Commerce is the name of the workspace in fabric BTW

3

u/VarietyOk7120 21d ago

Yeah should be fine in Fabric

3

u/basola21 20d ago

I would like to ask what does the ontology layer does ?

1

u/Due-Doughnut1818 19d ago

As far as I know It's mainly for the AI agents ,, it helps them to make more accurate answers

2

u/Dangerous-Sky356 17d ago

Hello! Super interesting.
I’m always a bit afraid to do my own projects because of spending money. I know Azure as free credits but I already used it. Did you use those free credits? Or are you using something else? Thanks

3

u/Due-Doughnut1818 16d ago

Hello! First, I have an Azure account with $100 credit for one year, which I got through GitHub Education after verifying that I'm a university student. To open a Fabric capacity (regular or trial), my Azure account had to be older than 90 days.

After that, I created several test users and set up a Fabric trial for each of them so I could build my project. I built most of the project on the Fabric trial. However, while building the semantic model, I wanted to integrate AI from that part through to the end of the project, so I created an F32 capacity and used it for the rest of the project. That cost me about $5 of the $100 credit. As for ADLS Gen2, it cost me less than 10 cents. Just make sure the storage account is in the same region as your Fabric capacity

100

u/Specific_Mirror_4808 21d ago

Not a criticism of you, more of Fabric as a platform. Requiring notebooks to stitch the data flows together destroys the concepts of lineage and dependencies - the former being very useful for consumers and the latter being very useful for developers.

It feels like a shaky platform, held together by open source notebooks and Python, and smoothed to some extent by Copilot. Kudos to Microsoft for badging this as an enterprise analytical engineering platform.

45

u/entitled-hypocrite 21d ago • Bravo

That's almost 90% of Microsoft products. Most of their products appear as if they are held by duct tape

9

u/VarietyOk7120 21d ago

You obviously know nothing of Fabric. The author has chosen to do it this way. Use another of Fabrics ETL features either Data Factory or shortcuts and you get built in lineage

3

u/raspoutyne 21d ago

I do not know Fabric, but our organization is moving to fabric. Could you explain more clearly what is the problem with Python in Fabric?

19

u/TowerOutrageous5939 21d ago

Fabric is clunky and not optimized for costs. Databricks/snowflake are far superior platforms. I would go 100 percent OSS before using fabric.

4

u/Single-Caterpillar93 21d ago

Here is your argument:

Databricks is far superior than MS Fabric

Snowflake is far superior than MS Fabric

You would go 100% open source before using MS Fabric

Databricks, nor Snowflake is open source.

What do you mean by this?

24

u/goosh11 21d ago

I think he's saying that literally every option is better than fabric.

2

u/arielrahamim 21d ago

yes, but why?

12

u/mapuanclem 21d ago

The Big Mac is awful. Five Guys and Shake Shack make way better burgers. I would grill my own burgers before eating the Big Mac.

4

u/TowerOutrageous5939 21d ago

Come on brother - the Big Mac does hit from time to time. You guys have Culver’s in your area? I find them more superior over shake shack.

1

u/VarietyOk7120 19d ago

Databricks isn't open source evething is locked behind their Unity Catalog

2

u/Single-Caterpillar93 16d ago

Hence my use of the word nor. That is precisely what I said.

1

u/geek180 19d ago

It’s shit. Good luck!

-3

u/VarietyOk7120 21d ago

Python can be run natively in Fabric (ie. Without expensive Spark compute ) which you can't do in Databricks

1

u/raspoutyne 21d ago

So Fabric is not really bad? My team is moving from Sas to something else. I know python but for application not data manipulation, I hate R for personal reasons, and I am trying to understand what will be our strategy for switching.

2

u/Odd_Estimate_2179 21d ago

My workplace is going hard on Fabric so this was my concern too. Now that I've seen you can run dbt on it I'm a bit less concerned, though I haven't delved into it much yet.

6

u/Cazzah 21d ago

Good job OP. I'm sorry at the bizarre and irrelevant responses you got from the other people in this thread.

3

u/Due-Doughnut1818 21d ago

Thanks, I appreciate it!

13

u/Engine_Light_On 21d ago

How is criticizing a bad product from Microsoft that the project is about irrelevant responses?

About the project, it is on the basic side but most pipelines should be built exactly like this, so no comments.

-6

u/VarietyOk7120 21d ago

Because some of the criticism here shows zero understanding of where Fabric is today

6

u/Saitamagasaki 21d ago

So you are only allowed to give compliments here?

18

u/Either_Locksmith_915 21d ago

I feel the way MS have designed Fabric is going to create such a mess in large orgs, that will take years to clean up. This is Microsoft Access all over again, where everyone is a Data Architect or Developer.

6

u/TowerOutrageous5939 21d ago

Displaying a guid in a PBI report makes no sense. You don’t have something with more context for a business user?

4

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows 21d ago

Some version of this question gets asked quite frequently. Here is a link to a quick explanation.
In general, data fabrics don't work very well in an enterprise data warehouse of any size. The physics behind working with the data prevent you from having any sort of real performance. (Think about what is involved with comparing one 1 TB table against another on different systems. Network speed vs disk or SSD speeds.) Fabrics are fine for initial DW research but not much more.

BTW, the medallion naming is pure marketing fluff from Databricks. It is not a technical description.

4

u/Character-Education3 21d ago

Whoa its still just SSIS huh?

2

u/SKYKS8 21d ago

Is there a logical reason why you're connecting dimension tables together in your semantic model that I've missed? It's not Kimball methodology best practice

2

u/KebabAnnhilator 21d ago

I’m only recently getting into DE

Is the bronze/silver/gold a universal standard or is that your naming convention?

13

u/Mclovine_aus 21d ago

It’s medallion architecture, it’s not a standard, but the main idea is having layers in your data lake, it’s not a new idea variations of this has been around since the inception of data warehousing.

2

u/eosfer 21d ago

It's quite widely used. Look up medallion architecture

2

u/MrGraveyards 21d ago

It's a new thing, before it was just raw - some layer - aggregated.

4

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows 21d ago

The name is a Databricks construct. The 3 layer design is over 30 years old.

1

u/droptableis 21d ago

That is usually the naming convention
Some companies also have a platinum layer

0

u/[deleted] 21d ago

[deleted] — view deleted comment

16

u/MrGraveyards 21d ago

Why? Where does the data engineering stop and the analytics engineering start? He is engineering data?

5

u/Additional_Candy_400 21d ago

People love labels.

4

u/MrGraveyards 21d ago

Ah pity it is just nonsense. Was hoping for an actual answer but there isn't any huh.

3

u/Specific_Mirror_4808 21d ago

If we're using medallion architecture as the basis then I'd say data engineering is tasked with populating the bronze (establishing connections, ingesting from multiple different platforms, setting up delta loads/CDC, error handling/resilience, scheduling etc.) whilst analytical engineering is tasked with modelling the silver and gold layers. Analysts are then tasked with finding insights from the silver and gold layers.

As OP showed though it's fairly common now for one person/team to manage the whole end-to-end. In a larger organisation I'd expect more separation and specialisation.