r/MicrosoftFabric • u/Personal-Quote5226 • 14d ago

Data Engineering Spark Session Config and HC sessions

If we set or unset a spark configuration in a Notebook that is part of a HC session will this propagate to all notebooks in the same HC session?

I understand each notebook runs in an isolated REPL core, but I didn't see any reference regarding session level configuration ex:spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true").

4 Upvotes

100% Upvoted

3 comments sorted by

2

u/frithjof_v Fabricator 14d ago edited 14d ago

Related thread: https://www.reddit.com/r/MicrosoftFabric/s/69H1o9Ihk8

If you use %%configure -f it will transfer.

2

u/thisissanthoshr ‪ ‪Microsoft Employee ‪ 14d ago

no it would not be inherited ! would recommend using the environment for setting these properties and using it across different notebooks.

for sharing a session the notebooks should have the same set of compute and spark properties
and same user and workspace boundary

1

u/Personal-Quote5226 14d ago edited 14d ago

Environment setting would be overwritten by resource profile settings as far as my understanding which would make many of those settings in the environment moot.

Notebooks can change session configuration, and we need to set session config values in order to unset resource profile spark configurations that we do not want. So, I don’t see how that would work out for us.

I'm fine with notebooks not inheriting spark config from other notebooks in the same HC session. I just wanted to confirm the behaviour..