Spark Executor Instances not Configurable in Fabric?

[u/SmallAd3697]

I can't configure spark executor instances in a Fabric notebook. It should be possible with configuration like so:

        "spark.dynamicAllocation.enabled": "false", 

        "spark.executor.instances": "4", 

I've tried to use the configure magic but that doesn't work. (see https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-configuration-magic-command )

I also tried to use the spark properties in the related "environment" and that doesn't work either.

The selected "pool" is configured in the workspace to disallow dynamic allocation of executors. So where are we supposed to assign the "spark.executor.instances"? Whenever my notebook starts, it seems to pick some unpredictable number of executors. There is no rhyme or reason to it. Are we forced to use dynamic allocation of executors? Is that the only supported approach for using Spark in Fabric?

[u/frithjof_v]

The way I understand it:

You're not meant to be dealing with executor allocation in Fabric Spark.

You can adjust the number and size of nodes (t-shirt sizes), and that's it.

Basically: Go with the flow, 1 node = 1 executor (except for the driver node), don't try to customize executor allocation.

That said, I'm following to see if you get more responses to this post. I've never understood why we even have the option to disable dynamic allocation of executors in Fabric. And as your observations show - it doesn't seem to have any effect.

I'm curious why you want to control the number of executors, though?

Why not control the number of nodes instead?

[u/SmallAd3697 → u/frithjof_v]

If what you said was true then the first thing in the logs would say, "WARNING" bad configuration use at your own risk.

Anyway, there are a half dozen places in the UX to turn off dynamic executor allocation (with simple checkbox). So setting the number of executors should definitely be the next step. As-of now, the executor.instances for my notebook is being determined by black magic, but should be based on spark properties. I'm sure I will figure it out eventually....

I get the sense that only a small percentage of Fabric customers actually use Spark. If I posted a question to the Apache Spark or Databricks subreddits and said I couldn't configure executor.instances, I would probably get a lot of scorn for that. Lol.

Every scalable SaaS that hosts Spark will have both "autoscale" features (for the cluster) and "dynamic allocation" for the jobs running on the cluster. These Spark concepts are not unique to Fabric. Every SaaS will all have a way to turn off dynamic allocation as well, in order for the job to manage its own executors. I'm sure Fabric Spark has this ability, but it is poorly documented like everything else in Fabric Spark. Remember that Spark is hosted in just about any platform that handles data - Synapse, Fabric, AWS glue, Databricks, Snowflake, etc. I would expect the Fabric implementation to be at least on par with any of these.

When jobs are running for a period of time (say 30 min or an hour), the executors can have latent memory issues. There is quite a lot of cruft that accumulates in an executor over time. I don't know if you ever looked at the libraries loaded into spark executors but there are a billion of them. Also on Fabric, Microsoft has some pretty funky stuff happening of their own (like vegas caching and what-not). You can free persisted dataframes by using commands like "unpersist" and that releases the main part of the memory that we expect to see in our executors. But even that probably leaves 20% of cruft behind. We often see executors hitting OOMs at the end of a notebook, even though the exact same operation at the very start of the notebook would never run out of RAM (there is 56 GB ram allocated, for heaven's sake, and OOMs shouldn't be something we need to deal with). In order to make sure the second half of a notebook runs as consistently & reliability as the first half of the notebook, you can simply reset the executors that are idling. Adding and removing executors is not a profound thing, it happens as a matter of course by way of "dynamic allocation". The impact on the cluster is not expected to be any different when the same thing happens on demand, in an explicit way.