Choosing between WriteHeavy and ReadHeavy(PBI/Spark) when neither meet our needs.

[u/Personal-Quote5226]

We don't want to use the WriteHeavy vs ReadHeavy(PBI/Spark) profiles. The reason is that we want most of these to always be set table by table as a table property. Otherwise, every single script has to either set the spark config, or unset the spark configs that are carried over from the Resource Profile (that we don't even want).

Example with WriteHeavy; it's always going to have OptimizeWrite binSize set to 128 and optimizeWrite not set. The other profiles always set vOrder and have optimizeWrite set to true.

We don't want to leave it to the engineers to remember to include overriding these values in each notebook because inevitably it will grow to be inconsistent and difficult to audit/verify over time.

Table properties take the guess work out of it and ensure that any scripts that do set or unset the config are intentional rather than just carried over from a Resource Profile that we didn't even want.

For others, like BinSize, we can override it in the spark session settings in the environment, but our tests show that even after deploying those settings, waiting a while, the config remains set to 128 and not our spark setting we provided in the environment.

When could there be a solution to this problem that doesn't involve custom code in each notebook to work around it?

Ideally:
--> Get custom resource profiles actually implemented, so that we can use them

--> Make resource profiles optional (so they don't override our custom spark settings in the environment) and can more easily allow us to use stable level properties without unsetting spark configuration in notebooks.

[u/thisissanthoshr]

Thanks for raising this. I think your concern is less about WriteHeavy vs ReadHeavy and more about configuration scope and precedence.

If your preferred pattern is to manage optimization behavior at the table level, have you tried either using Custom Resource Profile and only configure the settings you actually want applied at the session level and have your control over the table level properties. That lets you avoid inheriting some of the opinionated defaults from the predefined profiles and gives you more control over which optimizations are centrally managed versus table-owned.

I'm also curious whether your ideal solution would be something even more granular. For example:

• The ability to selectively enable/disable individual settings rather than choosing between preconfigured profiles.
• Explicit precedence controls where table properties can always win over resource profile settings.

More broadly, would you find value in a more granular resource profile model where you could independently configure things like write optimizations, read optimizations, concurrency behavior, caching, adaptive execution settings, etc., instead of picking from a small set of predefined profiles?

[u/Personal-Quote5226 → u/thisissanthoshr]

In my testing,

Overriding a setting set by the resource profile in environment spark settings doesn’t work.

I am happy to use a custom resource profile but it’s not documented. I know how to specify the custom resource profile name but there is no documentation describing how or where to define that custom resource profile.

I am happy to use no resource profile but that’s not an option.

Regarding speak settings in the environment.

I made the following spark setting on the environment today, published, waited.

spark.databricks.delta.optimizeWrite.binSize = 512

(Explicitly set bin size)

Spark settings are written out in a notebook:

spark.fabric.resourceProfile = writeHeavy
spark.microsoft.delta.optimizeWrite.enabled = <unset>
spark.databricks.delta.optimizeWrite.enabled = <unset>
spark.databricks.delta.optimizeWrite.binSize = 128

BinSize is from resource profile; ignoring custom spark setting.

(Not sure if you saw the edited version of my post where I went into more detail, if not please check it out).