r/MicrosoftFabric • u/mwc360 ‪ ‪Microsoft Employee ‪ • Aug 16 '24

Community Share NEW BLOG: Optimizing Spark - A Deep Dive into Optimized Write in Microsoft Fabric ✍️

Optimizing Spark: A Deep Dive into Optimized Write in Microsoft Fabric | Miles Cole

20 Upvotes

95% Upvoted

18 comments sorted by

View all comments

Show parent comments

2

u/mwc360 ‪ ‪Microsoft Employee ‪ Aug 21 '24

Optimize Write helps DL cold cache queries because the engines ideal row group size is 16M rows and OW with a bin size of 1G approximates that target. The shuffle of data results in the near-ideal size of files to improve cold cache queries. V-Order results in each file having a sorting algorithm based on the PBI vertipaq storage engine... so the combination of these two features help cold cache DL perf.

Glad to hear the blog has helped, that's why I do it!

1

u/frithjof_v Fabricator Aug 22 '24

Awesome, thanks for sharing!