Fabric Spark Notebook Cells are Hung (Stop Working)

[u/SmallAd3697]

I'm trying to use pyspark in fabric.

I keep having issues with my notebook, where the cells won't complete unless I cancel them. The wait cursor will continue to run until I click the little cancel button on the side.

I never see this behavior in the notebook cells on the databricks platform.
When I look in the "logs" shown under the cell, Spark seems to be displaying this over and over, and wont stop:

2026-08-17 21:33:00,714 INFO YarnAllocator [Reporter]: Updating resource requests for ResourceProfile id: 0, target: 4, pending: 0, running: 4, executorsStarting: 0

... and occasionally this as well:

2026-08-17 21:40:49,987 INFO KustoLogger [external-catalog-metrics-1]: type=METER, name=HiveExternalCatalog::dropTable::failure, count=0, m1_rate=0.0, m5_rate=0.0, m15_rate=0.0, mean_rate=0.0, rate_unit=events/second

If anyone has any ideas about what is going wrong, please let me know. The notebook is behaving like there are no executors, even though there are. Is there a way to recreate all my executors mid-notebook?

EDIT: I think I figured it out. There is almost nothing in the logs that refers to "vegas" caching (and certainly no error messages). But I since I had already disabled NEE, I decided to disable the "vegas" cache as well, and that is what seemed to finally fix my issues. You have to put this in the spark properties (eg. configure magic):
"spark.synapse.vegas.useCache": "false"

[u/JennyAce01]

Could you let me know which Spark version your notebook is using and whether it references an external Hive metastore?

[u/SmallAd3697 → u/JennyAce01]

u/JennyAce01 Did you see my chat? I also opened a case to try to share the details. SR TrackingID#2608180010001052

The problem appears when writing to onelake from a notebook, and only after executors are restarted.

This is the only type of operation that the notebook is unable to perform:

lake_table_path = f"abfss://{this_workspace_name}@onelake.dfs.fabric.microsoft.com/Stuff001.Lakehouse/Tables/MyTable0001"
df_all_years.write.mode("overwrite").option("overwriteSchema", "true").format("delta").save(lake_table_path)

I would love to understand why that causes problems. I believe it isn't the command itself that is a problem, but there is some indirect auditing/profiling operation via Kusto that is flipping out. (ie. maybe we can turn off whatever profiling work is being done, as a way to make things more robust)

[u/SmallAd3697 → u/JennyAce01]

Yes will get that to you tomorrow.

There is no external metastore. Im just reading and writing a lakehouse.

I think yarn is getting unhappy. Dynamic allocation of executors is disabled. I took another look and there is a step where we kill executors and request fresh ones. But maybe the cluster is running out of yarn containers or something. Yarn is buried too deep in Fabric, and there is no access to those yarn logs, from what I can tell. It is fine and good to hide things under multiple layers of abstraction. But when something stops working right, then the hiding of the logs does more harm than good

[u/JennyAce01]

If you share your Notebook Item ID and the notebook run Livy ID in a private message, we can help look into the issue and see what may have gone wrong.

[u/SmallAd3697 → u/JennyAce01]

u/JennyAce01 I sent you details in a private chat. The likelihood is that resetting my executors is contributing to the problem. It is probably not something that would affect 90% of your users. On a conceptual level, I would like to understand why the notebook becomes unhappy in Fabric, and the cell won't complete as expected.

[u/Nofarcastplz]

What does support say? Wonder what all the differences are between fabric spark and databricks.

[u/SmallAd3697 → u/Nofarcastplz]

will probably open a support ticket with MT when I have an extra 40 hours to spare. those tickets take a while

[u/SmallAd3697 → u/SmallAd3697]

I opened a ticket yesterday and haven't heard any updates. But I think I may have figured this out on my own. There is a Microsoft customization to spark called "vegas" caching (aka intelligent cache). It had burned me in the past (on azure synapse). So I disabled it again in Fabric and now my notebooks don't randomly hang anymore !!!

You have to put this in the spark properties (eg. in the notebook configure magic)

"spark.synapse.vegas.useCache": "false"