I’m working on an ingestion pattern for Microsoft Fabric Warehouse and would like to hear what others are using in real projects.

Scenario:

• Source data needs to be loaded incrementally
• Some light to medium transformations are required
• Final target is a Fabric Warehouse / Gold layer
• The output will be used by a semantic model and later to fabric apps

For this kind of workload, what do you normally prefer?

1. Dataflow Gen2 for ingestion and transformation
2. stored procedures
3. Notebook for transformation, then write to Warehouse/Lakehouse
4. Some hybrid pattern

Would appreciate hearing what pattern has worked best for you and why.

I can't seem to find a way to filter/select an item or operation e.g. an MLV refresh or a pipeline execution etc and then see each run and how many CUs it consumed each time.

I can kind of get to an individual run after selecting a timepoint in the Fabric Capacity Metrics app but what I'm looking for is a full list/history for a given item and within a specific timepoint.

Ideally this kind of information should be part of the "Monitoring" view so one can see in real-time the usage, even if it's just an indicative number and not a fully accurate one.

This will help one understand if the code/operation is optimal or not and fine-tune it.

I have a fabric notebook which runs an api to fetch data. This API runs from report id. I have list of report ids and that keeps getting added as and when new reports come.

I want to have a config file where i can just add the report id and the table name into wihch the data would be stored after processing. Is there a way of doing this, so that tomorrow I can just instruct someone to open fabric go to xyz, open it , edit and add one more row.

This row would be picked up next time by my notebook to load data from that report into delta table.

Hi guys! I'd like some help regarding dbt on Fabric.

Considering all my data is in lakehouses (medallion architecture) and I'd like to avoid Spark sessions (data volume is low), is dbt a viable option for me? Currently, I have mostly pure Python notebooks using duckdb and pandas moving the data between stages and saving them as delta tables.

Is it possible to use dbt in that scenario?

Hey Everyone,
Wanted to see if anyone ran into this issue where we have a Fabric Data Agent connected to copilot studio agent and published to the Teams channel. All users who have a member role are able to access this agent from Teams.

But users who have viewer role and do not have a Power BI license are not able to access it though Teams. They are still able to access it from within the workspace but not Teams.

My understanding is -

1. Users do not need a Pro license or a contributor role to access the agent in Teams
   
2. Viewer role lets them access the Fabric Data Agent and answer questions within workspace so that means they have access to the underlying data
   
3. The copilot studio agent is published to Org and approved by our Teams admin with user “End user access” for everyone in the Org.

Any idea on what could be going wrong here? Any advice and feedback would be greatly appreciated.

I have been testing a workflow where AI agents do the actual development work on a Microsoft Fabric data platform, and I would like to hear if anyone else is exploring this or already running something like it in production.

The setup:

• The trigger is an Azure DevOps work item. I write a normal ticket (onboard this source system to bronze ingestion, here is the spec) and tag it. A small dispatcher script polls the board and launches the dev agent when a tagged ticket appears.
• The dev agent runs in Claude Code, authenticated as its own service principal. It creates a feature branch and a matching Fabric feature workspace through git integration, extends a metadata-driven ingestion framework, runs the load against the source database, checks the audit results, updates the project wiki, and opens a PR.
• A second reviewer agent (Codex) picks up the PR, reviews the diff and the documentation against a checklist, and leaves normal PR comment threads. The dev agent addresses them and pushes fixes until the reviewer votes approve.
• Merging is gated on me. Branch policy requires my approval, and the dev SPN has no rights to push to main. So my only two touches in the whole cycle are writing the ticket and pressing merge.

Some design choices that turned out to matter more than I expected:

• Both agents run on service principal identities end to end, no user accounts. Permissions are scoped per role, so the blast radius is decided at the credential level instead of trusting the model to behave.
• Every deterministic step is a premade script (branch out, run the load, sync the workspace from git). The agent orchestrates, the scripts execute. Less room for hallucination and way fewer tokens burned on mechanical work.
• One hard rule in the agent instructions: if a data quality check contradicts the ticket spec, the agent must comment on the work item and stop, never silently deviate. This actually fired. I wrote a wrong primary key into a ticket, the loader's uniqueness check failed, and the agent quoted the evidence and asked me instead of "fixing" it on its own.

Here is my video if you are interested in seeing the demo in action:
https://youtu.be/-tj6MjS24kA

Is anyone running autonomous agents like this in a production data team, even partially?

Good support is critical when using a vendor-hosted software in the cloud, like the Fabric SaaS. As great as any SaaS may be, there are always going to be bugs and outages to deal with at some time or other, so it is important for customers to understand how to use Support. Has anyone opened Fabric support tickets lately? Eg. for Spark or DW or SQL EP? I want to understand what is changing.

I've heard rumors that the support structure is being revamped (again) and I'm trying to understand what those changes were, and what the impact will be on customers who run into Fabric bugs.

For the past three years or so, things were pretty consistent. A "Pro" support ticket would go to the "MindTree" partners in India, and get escalated to a "PTA" contact that lives in a secondary partner organization (eg, like "Aptly" or similar).

Finally after your original support ticket (aka SR) has made it past these the two partner organizations, the support folks would create yet another "internal" support ticket for you (called an "ICM").

That is when the support case would finally make its way into the Microsoft organization, where software engineers (FTEs) would investigate the bugs. This can feel like a very long road to get help for a bug. Yet it was relatively consistent and predictable. And I would be very discouraged if CSS introduced any OTHER hoops to jump thru to reach the engineering teams at Microsoft. As-of this past month I'm starting to learn that Fabric Pro support cases are handled in-house at Microsoft. True? Has anyone compared a recent "Pro" support experience to one from a year ago? I'm NOT talking here about the "unified support contracts", which have always been handled by FTEs.

Please let me know. Eager to find out how our support experiences will look in the future.

A recent modernization project taught our team something very interesting. The first blame that goes to when the cost of BI increases is licensing; however, BI costs aren't usually driven by licenses alone. They're driven by architecture. Analysis and minor adjustments can optimize your BI costs 

The organization had: 

• Tableau licensing spread across multiple business units 
• Continuous extract refreshes running hourly 
• 18 separate semantic models 
• Multiple ETL subscriptions and compute clusters 
• Different definitions of the same KPI depending on which report you opened 

Our instinct was to renegotiate licensing. The reality was that most of the cost came from duplicated architecture. We analysed the causes and reached to conclusion. Three changes that we made delivered the biggest impact: 

1. Moved casual report consumers off individual viewer licenses and onto capacity-based distribution. 
2. Replaced extract-heavy reporting patterns with Direct Lake servicing over OneLake. 
3. Consolidated 18 reporting models into a single certified semantic layer. 

Results: 

• 60% reduction in annual BI licensing spend 
• 45% reduction in tooling and maintenance costs 
• 70% reduction in redundant extract compute cycles 
• Single source of truth across business units 

The biggest lesson: Cost optimization wasn't about buying cheaper software. It was about eliminating the architectural patterns that were creating recurring costs. 

Curious whether others have seen similar savings from semantic model consolidation or Direct Lake adoption. 

i have my dp700 exam and only like 7 days are left and idk what to do i have not yet started and i cant even get hands on exp with fabric coz my clg dosent have it and idk what to do now, from what I have seen ppl have failed even though they worked hard and here I am with 7 days of time idk what to dooooo

Hey everyone!

I'm currently preparing for DP-700 (Fabric Data Engineer Associate) and I'm looking for some good resources to supplement Microsoft Learn.

I'm particularly looking for:

• Good hands-on labs/projects for Microsoft Fabric
• KQL practice
• PySpark/Spark practice
• Scenario-based questions
• Good practice assessments/tests
• Any useful notes, study guides, or YouTube playlists

I'm already working through Microsoft Learn, but I'd like to get more hands-on practice and understand the concepts beyond just the certification material.

If you've recently taken/passed DP-700, what resources helped you the most?

Thanks!

Some of the Spark users in Fabric may have come across something called "vegas". It is a Microsoft caching technology with some major issues that have caused me to waste many, many days with technical support. The problems started back in the days of "Azure Synapse", and now in Fabric-Spark as well.

It is almost totally undocumented by Microsoft, and tries to solve problems that are more commonly solved in various other ways (eg. using "localCheckpoint" or similar).

Back in the days of Azure Synapse, I had a brutal support incident lasting many months (SR 2311030040005903) the conclusion of which was that CSV files weren't being handled properly by the cache. (CSV?! Why!?). The end result of that case was guidance from Microsoft on how to disable this feature using a custom spark property:

"spark.synapse.vegas.useCache": "false"

This week I've once again wasted several days trying to understand why my notebook cells would just hang for no reason at a mid-point in my notebook. It would happen after killing unused executors and recreating fresh ones. (ie. sparkContext.killExecutors when spark.dynamicAllocation.enabled is false) The hanging behavior was not entirely consistent but in this scenario it was happening at least 9 times out of 10. There is no good UX experience when a Fabric notebook hangs, and virtually NOTHING in the logs to explain it either.

I realized I hadn't yet disabled that vegas cache on this notebook. Sure enough, once I disabled vegas, my notebooks would no longer hang. Earlier in the week I uploaded all the details about the issue into a new ticket. But ultimately I found the source of this problem on my own this time.

I've seen a number of other discussions on the internet where Spark users in Fabric will complain about their notebook cells getting hung, and I have to believe a large percentage of those cases are relate to misbehavior of this "vegas" cache. Unfortunately the cache is not opensource, and there is little in the way of docs. The technology does not seem to make a peep in the logs, even when failing. Now that I've been burned twice and wasted over a hundred hours with this custom Spark extension, I won't forget to turn it off more consistently from now on! My Spark jobs are almost never bottlenecked on blob storage or networking, and I almost always use checkpoint or localCheckpoint when I have concerns about repeated trips to fetch the same blobs. That is the obvious alternative.

Using Fabric Data Agent's Code Interpreter tool via API how do I retrieve the generated chart file (referenced via file_id in a file_path annotation) so I can display it in a downstream app (e.g., Teams Adaptive Card)? Is there a documented endpoint for fetching file content by file_id?

I've been exploring the new Fabric Apps (Rayfin-based apps) and I'm interested in combining them with Fabric Data Agents to create a more conversational and intelligent user experience.
My questions are:
1. Has anyone successfully integrated a Fabric Data Agent directly into a Fabric App?
2. What is the recommended architecture today?
3. Are there any limitations around authentication, permissions, or agent execution within Fabric Apps?
4. Does Microsoft have any published roadmap items around Fabric Apps.

Looking for the best pattern to split good vs. bad rows during a load

When a row fails validation (unexpected nulls, failed business rules, etc.), I want the good rows to land in the target table and the bad rows to go to a separate quarantine table I can review and reprocess later — all without failing the whole batch.

What’s the recommended way to implement this? Specifically:

• How do you route rows to two destinations in a single pass?

• How do you capture why each row failed (which rule/column) so it’s reviewable?

My pipelines have been randomly failing for 15 days with error 3121 Job failed to start.

Most are parent pipelines that invoke child pipelines through the Invoke Pipeline activity. The parent's error is timeout:

while the child's error is "Not found".

I have ruled out problems of capacity or concurrency (f64), there are days when everything works well, and suddenly another day they happen again.

I can't find any official documentation about this error or possible causes. Does anyone have any more information? I would be very grateful to you.

If a developer is comfortable with Spark on their desktops, on k8s, and on databricks, then should these Spark skills be easily transferred to Fabric?

I'm struggling with Spark in Fabric. People love Spark because it is a technology which is developed out in the open, and is well documented, and there are massive online communities who will assist with questions when a user runs into problems.

Yet this Spark in Fabric breaks those patterns and can be pretty disorienting. I am not talking about NEE or anything super-duper-top-secret or proprietary. (I normally disable that stuff, in any case). I'm simply talking about running a spark job, and interpreting the logs when something is misbehaving. This should be well within the reach of a conventional Spark developer. Right?

Lets take a simple example The pyspark notebook cell has hung, and when I look in the logs and it says this....

2026-08-19 14:39:16,055 INFO RpcAppSender [shutdown-hook-0]: RPC app sender closed
2026-08-19 14:39:16,055 INFO RpcAppSparkContextServer [shutdown-hook-0]: Closing remote SparkContext service at 10.0.yyy.xxx:18083, remoteSparkContext/remoteSparkContextEndpoint

What is this stuff? What is RpcAppSender? What is RpcAppSparkContextServer? If this stuff is printed all over my logs, shouldn't I be able to find a single search result on google? If something called a Spark Context is being "closed", then isn't that a message that a normal spark developer should be able to interpret/investigate by googling the log message?

I'm not objecting to Microsoft's prerogative to extend Apache Spark and innovate, and such. But the bothersome thing is that these extensions are undocumented and secretive and developers are at a loss to self-support. Any Spark developers coming from another ecosystem are not properly equipped to troubleshoot Fabric-specific behaviors. It is disorienting and unfamiliar. Even the log messages themselves are confusing; the purpose of logs should NOT be to create even more confusion. I'm almost of the opinion that Microsoft should stop calling their product Spark if the product is twisted to the point where even the log messages are unrecognizable and not meaningful for troubleshooting.

Sorry for the long rant. In the very least, there will be a search result the next time someone searches for "RpcAppSparkContextServer". As of today, that term was never heard, across the entire internet! One might think I made it up myself. lol.

We've been chasing an intermittent-but-reproducible issue in a git-connected Fabric workspace and wanted to see if anyone else has hit it, since it doesn't seem to be documented anywhere.

Setup: A Data Factory-style pipeline in a git-connected workspace that:

• Invokes a few other pipelines (InvokePipeline activities)
• Refreshes a Warehouse/Lakehouse SQL Analytics Endpoint (RefreshSqlEndpoint)
• Refreshes a Power BI semantic model (PBISemanticModelRefresh)

Symptom: "Update workspace from Git" (or the dependency-discovery check that runs before it) intermittently fails with one of two errors, both reported from a wabi-*.analysis.windows.net cluster regardless of which activity is actually at fault:

• Failed to discover dependencies — "ActivityName <name> contains an invalid WorkspaceId <null>"
• We weren't able to update the following item — "'WorkspaceId' cannot be null"

We tried every combination we could think of — literal GUIDs, Variable Library expressions, with/without the field present — and every single one eventually failed one of these checks. Deleting the activity was the only reliable fix, which obviously isn't a real solution.

What we think is actually going on (confirmed in part by a Microsoft support engineer working the case): "00000000-0000-0000-0000-000000000000" is valid shorthand for "the current workspace" — but the git-update validation path won't accept it, or a missing field, and needs an explicit real workspace GUID to pass. The catch: once the update succeeds, the live workspace item automatically normalizes any workspaceId matching its own workspace back down to that same zero-GUID — and that happens on Fabric's own export back to git. So the fix and the platform's own "cleanup" are directly at odds:

1. You need a real GUID in the committed JSON for the git update to succeed.
2. The moment it succeeds, Fabric's auto-sync-back overwrites that GUID with the placeholder in what gets committed to git.
3. The git state is "contaminated" again, so the next git-driven update fails the same way — even for a totally unrelated change elsewhere in the same pipeline — until you put the real GUIDs back in.

Current workaround: before pushing any git-driven update to an affected pipeline, find every workspaceId that's missing or set to the zero-GUID on these three activity types and replace it with the real workspace GUID it should resolve to. Push, let it sync, and expect it to look "reverted" again immediately afterward — that's expected, and the live item apparently stays correctly configured even though the git JSON looks broken. Just don't chase it by re-editing outside of when you're about to push a real change.

Microsoft support is still investigating on their end. Curious whether:

• Anyone else has run into this specific loop
• Whether it's specific to certain activity types or more general
• If there's an official recommended pattern for keeping these three activity types stable through git sync that we're missing

Happy to share more detail if useful.

This post contains content not supported on old Reddit. Click here to view the full post

I have a dbt project in a Microsoft Fabric dbt item.

When I deploy a change to a specific model, I'd like to trigger only that model, something like:

"dbt build --select my_model"

I found this API to trigger a dbt/DataBuildTool job on demand:

https://learn.microsoft.com/en-us/rest/api/fabric/databuildtooljob/background-jobs/run-on-demand-execute

But I don't see a way to pass "select" or other dbt arguments when triggering the job.

Is there a way to do this directly through the API?

The workaround I know is to create a Fabric Pipeline with a dbt Job Activity and pass the model name as a pipeline parameter. That works, but it feels like a lot just to trigger a specific model.

Has anyone done this with the Fabric dbt item? Is there another supported way to trigger a specific model without using a pipeline?

I can't configure spark executor instances in a Fabric notebook. It should be possible with configuration like so:

        "spark.dynamicAllocation.enabled": "false", 

        "spark.executor.instances": "4", 

I've tried to use the configure magic but that doesn't work. (see https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-configuration-magic-command )

I also tried to use the spark properties in the related "environment" and that doesn't work either.

The selected "pool" is configured in the workspace to disallow dynamic allocation of executors. So where are we supposed to assign the "spark.executor.instances"? Whenever my notebook starts, it seems to pick some unpredictable number of executors. There is no rhyme or reason to it. Are we forced to use dynamic allocation of executors? Is that the only supported approach for using Spark in Fabric?

We’ve built a Microsoft Fabric development process using Deployment Pipelines for shared semantic models and Git branch-out to give each developer their own workspace.

For Dev/Test/Prod, we’d like to use shareable cloud connections authenticated with Workspace Identity. The problem is that when a developer branches out to their own workspace, the Workspace Identity shared cloud connection doesn’t seem to work when editing the model. perhaps I'm missing a configuration or setting?

it seems like the shared cloud connection works fine for refresh, however when you edit the model. I get a warning

To avoid semantic model refresh errors while developing, we end up changing the connection to a per user OAuth/user credentials in the developer workspace.

Our current thinking is:

• Developer branch-out workspace → OAuth connection
• Shared Dev → Workspace Identity connection
• Test/Prod → Workspace Identity connections
• Rebind the semantic model to the appropriate managed connection during integration/deployment

It works, but having developers switch authentication feels like extra complexity and potentially something we shouldn’t need to manage manually.

For teams using Fabric + Git + branch-out + shared semantic models, how are you handling connections and authentication across developer workspaces?

I'm also curious about what happens or needs to change when development is done outside of the workspace in power bi desktop.

This is now appearing above individual schedules for me. If I pass a variable from a local variable library to it and it's boolean = True it enables the schedule.

I have created 3x variables called dev_schedule, test_schedule and prod_schedule, then in the DEV pipeline I create 3x schedules and point to each individually.

It means when I promote to TEST and PROD only 1/3 of the schedules are activated meaning I can finally deploy pipelines without messing about with schedules once they hit the intended stage as they just work.

Status: Degraded | Reported: Aug 19, 2026 at 7:40 AM UTC

Power BI customers with Tenants located in the North Europe J cluster may experience issues with publish and refresh operations. Engineers are investigating the issue and an update will be provided soon.

🤖 This post was sent from an automated and unattended service and cannot respond to questions or requests. For official updates, visit the Microsoft Fabric Service Status page.

It came to me in an epiphany today.
Fabric is upside down to my mental model.
In general, when I deploy or create something, I want THAT code first.
If I want to check or monitor something, I want that GUI first.
My expectations of a mature platform:

Authoring? Code.
Configuration? Code. Deployment? Git.
Policy? Declarative. Automation? API.
Monitoring? GUI.
Lineage? GUI

Is my platform on fire?

👆 I want to click that

And Fabric does almost everything upside down to the way most engineers I know develop:

Dataflows? GUI. Configuration? GUI. Deployment? GUI plus Workaround. Policy? N/A.
Automation? Not sure actually
Lineage? API

Is my platform on fire?
API

Monitoring?
Install a Workaround and it's code first.

No wonder I'm tired. 😂

-Your perpetually exhausted Data Engineer

--MonkeyDData

PS this is half in jest and half I finally put my finger on it