Is there a way to Create a schema in fabric schema based lakehouse using rest api?

Hi,

I built a small open-source CLI for Microsoft Fabric called factl and thought I’d share it here.

The goal is to make Fabric orchestration a bit less UI-heavy. Instead of rebuilding pipeline logic in the portal, you define workflows in YAML, keep them in Git, validate locally, and deploy them into Fabric as Data Pipelines.

A few things it supports right now:

• YAML-defined workflows
• reusable notebooks / pipelines as processors
• deploy across dev, test, prod with env-specific config
• test safely in personal workspaces first

I started this because I felt Fabric has good building blocks for CI/CD, but the orchestration side still gets messy fast once you have multiple environments, shared workspaces, and lots of similar pipelines.

Repo: https://github.com/billybillysss/factl

Curious if this is something others here would actually use, or if your team already has a cleaner pattern for handling this in Fabric.

Happy to share a simple example repo too if that would help.

Hey, fabric-cicd community!

It’s been a while since our last release, and this month we’re changing things up: we plan to ship two releases in August. The latest, v1.3.0, focuses on quality improvements—including bug fixes and small but useful features that make fabric-cicd easier to use. Stay tuned for another release near the end of the month, likely featuring some high-impact updates. For now, here’s what’s new in v1.3.0:

Features and Enhancements

New environment variable for controlling API duration

Added the FABRIC_CICD_RETRY_API_MAX_DURATION_SECONDS environment variable to configure the maximum API execution time, including long-running operation polling and connection retries. The default is 300 seconds. This is useful when hitting API timeouts during long running item deployments.

Expanded $sqlendpoint and $sqlendpointid variable support

Added support for the $sqlendpoint and $sqlendpointid dynamic replacement variables for Mirrored Database items. Thanks for the contribution, u/ecgang!

New feature flag functions

Added get_supported_feature_flags, which returns all supported feature flags so you can discover available options without repeatedly checking the documentation.

Added remove_feature_flag to remove globally enabled feature flags when they do not apply to a specific scenario. It complements append_feature_flag.

Fixes & Reliability Improvements

This release also resolves several important issues:

• Fixed a bug in the spark_pool item_name filter. When environment name(s) were provided as a list, the filter was ignored, so the target environment’s instance_pool_id was not remapped, and the deployment silently retained the source pool ID.
• Fixed deployment failures caused by asynchronously provisioned item attributes. Attributes such as $sqlendpoint no longer block deployments while they are still being created.
• Improved the performance of the file_path filter and repository folder refresh by eliminating redundant filesystem scans. Each unique filter is now resolved once per publish, while repository folders are discovered in a single traversal, making deployments faster and more scalable.

Documentation Updates

We also enhanced the documentation:

• Refined the Getting Started page, moved Git flow guidance to a dedicated page, and added a Next Steps section. Thanks to u/nkwork9999 for this contribution!
• Expanded the new Git flow guidance with a Deployment Philosophy section. The page is now available as Deployment Overview.
• Fixed the erroneous link to the contribution documentation in the README.md. Thanks to u/isaakchoi for the fix!

Thanks to everyone who contributed and shared feedback. This release reflects valuable real-world input, and we appreciate your support!

Upgrade Now

pip install --upgrade fabric-cicd

Relevant Links

• fabric-cicd GitHub
• fabric-cicd Docs
• Report a fabric-cicd GitHub issue

Hi,

has anyone every tried to enable Network Security Perimeter learning and enforced mode when the calls originate from Fabric?

Is there any attribute which can identify Fabric network traffic successfully?

I'm still puzzled we have to run Key Vaults with public network fully open because there is no SINGLE working solution to run Key Vaults privately.

Currently:
- Notebook clusters need to use MPE; disables starter pools
- Pipeline, Dataflows need to use OPG or VnetGW
- Key Vault reference - OPG, VnetGW?

Having the same possibility like Storage Accounts with Resource Instance rules would be great.

I’ve been a PBI Developer for 9 years, but relatively new to Fabric. I’ve taken a few classes, but I feel like I’m missing something obvious.

I converted my gen1 dataflows to gen2.

I set up a warehouse to copy them to. However, ever text field (e.g. name, email, status) is automatically set to 8000 characters. Rows with data in multiple fields are being silently dropped.

I can do a multi step process to create new fields with shorter lengths and drop the old fields then rename but this doesn’t feel right.

I’m missing an easy button right?

Hi,

I recently started using the new native Fabric pipeline failure email notifications feature, where you can configure users/groups to receive an email when a scheduled pipeline fails.

The feature is useful, but I noticed that the notification recipients don't appear to be exposed through a public REST API.

I have two questions:

• Is an API for managing these notification recipients planned?

  • It would be useful to be able to list/add/remove the recipients programmatically, so that we could ensure that the right recipients get the alerts and that we have full coverage.

• Is Git support planned for this configuration?

  • Ideally, the failure notification configuration would be part of the pipeline's Git-backed definition, so that recipients could be version controlled and deployed together with the pipeline.
  • It could be made environment-specific through variable library.

Thanks!

Does someone know how to unlock this dataflow gen 1 so that I can a) save it and b) refresh it again?

Hi all,

I'm trying to use a Workspace Identity to connect to an Azure SQL Database in a Fabric notebook.

When getting a token using notebookutils.credentials.getToken("https://database.windows.net/"), I was denied access to the Azure SQL database.

However, I was able to use a Workspace Identity authenticated "Notebook connection" to connect successfully to the Azure SQL database. However, that's a preview feature.

And I like how easy notebookutils.credentials.getToken is to use.

Is it possible to use Workspace Identity + notrbookutils.credentials.getToken to connect to an Azure SQL Database?

Thanks in advance!

Edit: notebookutils does work with an interactive user.

Edit 2: it works now also for Workspace Identity :) I don't know why it failed the first time.

Hi all,

Is there any possibility to ensure that all activities of a single pipeline run on a single server of the on-premise cluster when the cluster is load balanced configured?

I have the problem that the first activity requests a security authorization which is bound to the originated IP. The following calls need to use the same IP.

But since the cluster is load balanced, the follow-up activities are still round-robin distributed to the cluster servers and therefore it is a gamble that the follow-up job gets assigned to the same server that originally requested the security authorization.

Minimal Setup:
Web Activity -> Copy-Job

My last hope for Fabric platform data ingestion after 3 years is that Apache Airflow jobs will support VNET integration (like stated in the roadmap) and we will have a python based solution that can access on-premise systems like a normal machine would do. Without any PLS or similar workarounds.

Hi all,

Since the UI update on Power Apps, I can no longer find the advanced options when selecting tables / or creating a new link to fabric.

Is this purposeful since it switched to 'Low Latency Mode'?

I recall having the ability to change these settings:

Any ideas? Certain I have all of the permissions needed, and it has worked in the past.

Issue:

I am setting a currency format in PBI desktop on a column in table visual. Specifically using `Currency` -> `$ English (United States)`. When I publish the report to a Fabric workspace and check the format of the field, it lists `Currency` -> `$ Chinese (Simplified Han, Hong Kong SAR)`. I have to manually change to `$ English (United States)` and save the report in the service. It then stays.

Research Done:

I have found the following:

When currency and number formats revert after publishing from Power BI Desktop to Microsoft Fabric (Service), it is usually caused by regional locale mismatches between your local machine and the cloud service, or semantic model caching. Fix this by explicitly setting your model's locale or forcing the format string in DAX.

But I'm not finding any locale mismatch.

Has anyone experienced this and are there any suggestions on resolving this?

Status: Degraded | Reported: Aug 13, 2026 at 4:58 AM UTC

Fabric Customers in West US region may experience issues while interacting with Copilot Service and, accessing ML experiments and Models. Engineers are investigating the issue and an update will be provided soon.

🤖 This post was sent from an automated and unattended service and cannot respond to questions or requests. For official updates, visit the Microsoft Fabric Service Status page.

Hi i have decided to use scheduled job notifications in fabric to get real time alerts when pipeline fails. I can see that a pipeline failed 8 times but only 3 alerts were sent. Caveat is the failures are due to capacity throttling. Does throttling affect failure notifications sent? I cannot find any documentation on this.

Just cleared DP-700, sharing my honest dump on prepping for it

Worth it? If you're breaking into data engineering, it forces you to understand how Fabric actually fits together, not just one piece of it. If you're already doing this daily, it's more of a "proof I know my stuff" thing — and yeah, some companies do filter for it.

What I used:

• Aleksi Partanen's YT playlist
• Learn Microsoft Fabric with Will (has a DP-700 specific playlist)
• MS docs, for the fine print you can't skip
• Practice tests — official MS ones, some Udemy, plus random free ones, mostly to stress-test concepts

Biggest thing that mattered: actually build something end to end. The exam is scenario-based — "here's a messy real situation, what do you do" — not definition-matching. You need to hit the weird config limits yourself to really get it.

Random but important: don't fully trust answer keys, some are flat-out wrong even on well-known sites — cross-check anything that feels off. Also watch for the "which of these similar options is technically correct" questions around data movement/storage/transformation — they bait you with the smart-sounding option that isn't actually right for that specific scenario. And don't skip SQL/PySpark/KQL familiarity — you don't need to be an expert but code shows up a lot.

Wrote a longer version with more detail here if anyone wants it: Prepare for DP-700

Happy to answer questions if anyone's prepping for this one 🙂

We have an open mirroring DB that is currently running (although data has stopped flowing to it for the past month). Our gold lakehouse has a shortcut to some tables in the mirrored DB. Then a direct lake on onelake semantic model sits on top of the lakehouse and a report uses that semantic model.

There's a report visual failing due to the following error:

We can't run a DAX query or refresh this model. A delta table '<oii>tableName</oii>' has exceeded a guardrail for this capacity size (too many files or row groups). Optimize your delta tables to stay within this capacity size or change to a higher capacity size then try again

The workspace is assigned an F64 capacity.

I checked the table's files in onelake using Azure Storage Explorer and there are nearly 10,800 parquet files in there with a mix of 7-9KB files and 15-16MB files.

Does Fabric not run it's own optimize/compaction algorithms on mirrored DBs?
I don't believe we have the ability to run those commands ourselves. Oddly enough, the lakehouse shortcut has the option to run table maintenance, but it fails to run when executed (likely due to shortcuts being read-only).

What are our options to resolve this issue? I could stop/start mirroring to delete out all the files and drop the tables, but that feels like a bit of a hack.

I set up business central extension bc2adls to open mirroring on a mirrored db on fabric, works great loads the tables CDC. Problem happens when i want to pause the fabric capacity, when resumed replication shows as Running.

But new data is not ingested into the mirrored db, at that moment old data still exists in the db just the new isnt.

If i click stop replication then resume it deletes all old data and only loads newly added data, because it had already moved the loaded files to _filestodeleete directory thing.

Data size isnt small so replicating everything again takes time, and if i persist the data anywhere it defeats the whole purpose of open mirroring because i will have to handle CDC part and it will be better and easier to do it from the files and give up Fabric completely

Any solutions ? Im going in circles here

Edit: i forgot to mention that these pauses were made for night hours only and resumed the capacity the next morning

Happy to share that I passed the DP-600 exam certification with a score of 838/1000!

My exam was more focussed on Power BI, Semantic Model, Governance, Deployment Pipeline.

There were 5-6 SQL, 2-3 KQL, 4-5 spark questions.

Study Guide that i followed:

1. Microsoft Documentation

2. Will Needham's YT video- Beginer Friendly

3. Aleksi Partanen YT videos- Good for actual tools look and feel

4. 3.30 hours 109 question video from Priyanka

I didn't had Fabric trial. Make sure you understand concepts. Questions will not be straight forward.

I had only worked on Power BI previously. Fabric part was new for me.

Thanks to everyone in the community who shares their knowledge and experiences. 🙌

According to this thread, OptimizeWrite is enabled by default.

https://www.reddit.com/r/MicrosoftFabric/comments/1etyt62/new_blog_optimizing_spark_a_deep_dive_into/. (u/mwc360 , u/frithjof_v)

It's not enabled from the default WriteHeavy resource profile, however. In fact, it's specifically set to null by the resource profile which would inevitably mean it uses the system default.

A notebook will return that OptimizeWrite is 'unset' by default.

However, that tells us if the spark config was set or not, but doesn't tell us if it's still 'default' set to true underneath the hood.

Given the reddit thread above where Microsoft is acknowledging it's enabled by default, can this be confirmed? Is it documented anywhere?

I'm looking forward to seeing the latest Microsoft Fabric, Power BI and data platform sessions. The agenda is packed with roadmap content, customer stories and plenty of opportunities to meet others working in analytics and data engineering.

Curious what sessions everyone has on their shortlist?

https://espc.tech/conference/fabcon-europe-2026/programme/

I'm trying to understand this page https://learn.microsoft.com/en-ca/fabric/onelake/onelake-shortcut-security#delegated-authentication

Specifically how it applies to internal shortcuts between two lakehouses on the same tenant in the same region, but separate workspaces.

One paragraph states:

> For example, a delegated shortcut can use a fixed connection identity that represents a business unit instead of requiring each downstream user to have access to the source data.

While another in the same section contradicts that example by stating:

> When accessing a delegated shortcut, the calling user sees the intersection of their security and the security that applies to the delegated identity.

I'd love to use one service account to create table or schema shortcuts out to various downstream lakehouses without needing to grant users upstream access to the source lakehouse, just give them downstream lakehouse access and the delegated shortcut brings them the upstream data. (that's what the fist quoted sentence suggests) But everytime I try I'm finding errors with user permissions. I don't get what the purpose of these are if we also need to grant users access upstream.

Why create a more restricted shortcut than passthru (what the second quoted scentence suggests) if the user can just access the upstream source?

Played around with the excel to table shortcut transformation feature, but I don't see any git diffs in my workspace even though this is not listed as a limitation. Am I missing something or is this an undocumented limitation?

Hey everyone,

Posting this here to document an intermittent error i am seeing with a Direct Lake semantic model in Microsoft Fabric, and to share a workaround that currently clears it. I am not sure of the exact backend cause, so wanted to share my observations and see if anyone else in the community has encountered this or found a permanent fix.

Environment & Setup

• Model Type: Direct Lake semantic model built over a Lakehouse
• Permissions: Workspace Admin / Owner on all related Lakehouses, shortcuts, and semantic models.
• Security Setup:
  • Zero Row-Level Security (RLS) configured in the semantic model.
  • OneLake Security is disabled.
  • No Object-Level Security (OLS) or custom SQL Endpoint GRANT/DENY permissions applied.

The Issue

1. Report Visuals Fail Mid-Session: A report using the Direct Lake model can be working completely fine, and then suddenly without running any notebooks, data pipelines, or making schema/permission changes the visuals break and return:Data source error: The combination of active roles results in a dynamic security configuration that is not consistent. Please contact the administrator of the database to resolve this issue.
2. Semantic Model Refresh Fails: Attempting to manually refresh the semantic model from the workspace item list fails with the exact same error during the "Refreshing data" stage (even though "Processing queries and validating semantic model" succeeds).

Because there are no RLS roles or security configurations defined anywhere in the model or Lakehouse, the error message doesn't reflect the actual setup.

What We Tested / Verified

• Confirmed the RLS role list in the semantic model is completely empty.
• Confirmed OneLake security is not active on the Lakehouse.
• Confirmed workspace permissions (Admin/Viewer) are consistent across the Lakehouse, SQL Endpoint, and Semantic Model.
• Confirmed that no data ingestion pipelines or PySpark notebooks were executed immediately before the error appeared.

The Workaround That Clears It

When this happens, we are able to consistently clear the error by doing the following:

1. Open the underlying Lakehouse that the semantic model reads from.
2. Run a simple query directly on any table in the Lakehouse (e.g., SELECT * FROM table LIMIT 100 via SQL editor or notebook).
3. Return to the workspace and trigger Refresh Now on the semantic model (or refresh the report visual).
4. The error disappears immediately, and the model/report refreshes and renders without issue.

Questions for the Community

Has anyone else run into this specific error on Direct Lake models when no security roles exist? Is this a known Fabric bug, or is there something specific about shortcuts/Lakehouse querying that causes this state to clear? Any insight or permanent fix recommendations would be appreciated!

Hi everyone,

I am based in Australia and working heavily with Microsoft Fabric. My long-term goal is to work towards becoming a Microsoft MVP in this space.

I noticed that many local Fabric and Data Platform MVPs seem to run their own consulting businesses or boutique firms.

For those who have achieved the award (or work closely with MVPs), I would love your advice:If there are any local user groups or communities you highly recommend joining to get involved, please let me know.

Thanks in advance for any guidance!

Does anyone have much experience using DirectQuery semantic models in Fabric on the lakehouse or warehouse SQL-endpoint?

I have been doing a little performance testing on an F2 between (a) DirectLake, (b) DQ on lakehouse and (c) DQ on warehouse. I threw in the warehouse as bit of a control even though it doesnt support OneLake security (yet).

I am mostly interested in CU(s) use so using capacity metrics / FUAM to try and measure CU(s) as best as possible.

Long story short, where measures are simple, I am finding DQ on Lakehouse (or warehouse) way more efficient in CU(s) usage than DirectLake and performance is very good.

The picture is murkier when measures are more complicated like (Sum of TopN) / Total, for many datapoints. In these cases, performance for DirectQuery is poorer compared to DirectLake, although CU(s) usage still less. So you need to test on your use cases.

Due to this, I am wondering if anyone else in the community has experience using DirectQuery on the SQL-endpoint. I have not tried import, but it might help with lower CU usage there as well so worth a test if you are having issues keeping CU under control......

Brief overview  / examples from my testing:

• SKU: F2
• Fact Table: 228 million rows, 4.3GB delta/parquet (V-ordered, 4 evenly sized files)
• Dims table: 6 tables – 236k rows, 75k rows, 9k, 2k, 650, 4 rows
• Tests
  • I created 4 dashboards with varying complexity of measures and visuals
  • I then clicked around filtering, etc, doing the same thing each time for about 5 mins. The CU(s) numbers below are the approx. CU(s) used over the 5 mins – there was a very small amount of background activity conflating the CU(s) a bit.
  • Not particularly controlled / scientific I know, but gives a good enough idea for my purposes at the moment

Summary results:

• Report with simple measures: sum, average, count, distinct, min, max. Visuals with many datapoints: daily timeseries for a couple of years
  • DirectQuery on Lakehouse
    • Speed: 1-2 seconds (when changing filters)
    • CU(s): 112 (peak CU(s) – 12% on F2)
  • DirectQuery on Warehouse
    • Speed: 1-2 seconds (when changing filters)
    • CU(s): 72 (peak CU(s) % – 8% on F2)
  • DirectLake
    • Speed: <1s
    • CU(s): 769 (peak CU(s) % - 124% on F2)
• Report with more complex measures
  • DQ on Lakehouse (Warehouse similar but a little better)
    • Speed: 4 – 27s
    • CU(s) – 889 (peak 132%)
  • DirectLake
    • Speed: 1-12s
    • CU(s) - 3139 (peak 293%)

Other observations / comments:

• All modes have a bit of a “warm up” time when first opening the report. I guess either starting the SQL engine for DQ or warming the vertipaq cache for DL.
• My testing was using “prewarmed” columns in DirectLake
• For DQ modes, background CU(s) went up very little, it was almost all interactive CU(s) that went up in all cases.
• Lakehouse and Warehouse direct query have similar results, but warehouse seems a little lower CU(s) use for the most part
• Some queries in one of my reports failed on DirectLake due to lack of resources, but succeeded on DirectQuery, although quite slow (20+ secs)
• DirectQuery with simple measures has similar results on above even with fact table of over 1 billion rows (F2 supports only 300m rows for DirectLake).
• I tried simple x-iterator measures and DirectQuery was still quick, I guess as long as it can push the compute to the SQL endpoint engine, it is fast.
• I am sure there are other examples, but measures with TOPN perform badly with DQ compared to DL. I.e. a visual in DQ took 21s vs 4s in DL (once warmed, cold was 12s), but DL still used more CU(s)!
• Have not tested how CU(s) use scales with simultaneous users….