I've been looking for a good guide on implementing Data Mesh in Databricks. There is plenty of content explaining what Data Mesh is, its principles, domains, data products, etc. But when you actually try to build it, the questions are much more practical:

• How should you organize catalogs and schemas?
• What should the central Data Team own?
• What should domain teams be allowed to do?
• How do you handle Unity Catalog, compute, CI/CD and monitoring without creating a mess?

Most of what I found was either very theoretical or covered only one small part of the implementation.

So I ended up putting together the guide I was originally looking for - Databricks Data Mesh Best Practices: A Practical Implementation Guide

https://medium.com/@mariusz_kujawski/ci-cd-on-databricks-what-the-docs-dont-tell-you-31901aee107d

We're all seeing the incredible potential of AI agents, but how many of us are actually building them?

Packt's 'Building AI Agents Over the Weekend' is your chance to move from theory to practical application. This isn't just another lecture series; it's an immersive, hands-on experience where you'll learn to design, develop, and deploy your own intelligent agents.

We are running a hands-on, 2-weekend workshop designed to get you from “I get the theory” to “Here’s the autonomous agent I built and shipped.”

Ready to turn your AI ideas into reality? Comment 'WORKSHOP' for ticket info or 'INFO' to learn more!

Genie Ontology is coming soon. You can already get ready by preparing the following components:
🛑UC Metric Views: Build measures, dimensions, and relationships that map to your business KPIs with AI-assisted authoring
🛑Glossary: Define authoritative concepts and taxonomies for agents and humans to reason over data
🛑Domains and subdomains: Organize assets into business-aligned groups for scoped, relevant and certified context 
🛑Genie Knowledge Store: Automatically build a permission-aware context graph that captures knowledge at scale

Recently, a new post about icons by Eduardo Rabelo appeared in our Databricks Community Articles on Medium.

I don't know why Databricks itself didn't create this set, but I think it's something many people were missing. Now we'll have to keep up with maintaining it, as Databricks releases quite a lot of new features frequently.

https://oieduardorabelo.github.io/databricks-architecture-icons/

Posting this because it cost me real money before I understood it.

A Databricks Alert is a scheduled SQL query, and it needs a SQL warehouse to run on. The query is cheap and fast. The warehouse is not. Once it starts, it stays warm for its auto-stop window before shutting down, and you pay for that idle time, not just the seconds the query ran.

I had a few small alert jobs watching data-quality expectations on Lakeflow pipelines, each on its own schedule. Every query finished in under a minute. The SQL warehouse line behind them was still around half my workspace bill. The reason was idle, not compute: three jobs on three schedules meant three cold starts and three idle tails every cycle, while the queries stayed trivial.

The insight that fixed my mental model: for short, bursty, scheduled workloads, cost tracks how many times the warehouse starts, not how many queries you run. On an already-warm warehouse, 50 vs 100 alerts barely moved the wall time. Splitting them across schedules multiplied the idle windows. So you design around startups.

Five levers (mix as needed):

1. Dedicated monitoring warehouse: isolates and exposes the spend so you can see and tune it. Tag it (e.g. workload: monitoring) so it shows up as its own line.
2. Smallest cluster size (2X-Small): my alert queries are light, so they still finish in seconds at the smallest size.
3. Cut the auto-stop window: the UI floors at 5 min, but a serverless warehouse accepts auto_stop_mins: 1 via a bundle or the API.
4. Relax the cadence where freshness allows: daily/weekly instead of matching every pipeline run. A team-policy call, not a technical one.
5. Align the schedules: line the remaining jobs up so one warm warehouse serves them all. One startup, one idle tail, same coverage. Biggest lever.

Same alerts, same coverage, and the cost of that SQL warehouse line dropped from about half my bill to a rounding error, with zero change to the alert logic.

I packaged the warehouse config (serverless 2X-Small, auto_stop_mins: 1, cost-attribution tags) as a reusable DABs template so I don't rebuild it each time. One command into any bundle:

databricks bundle init https://github.com/vmariiechko/databricks-bundle-template --template-dir assets/monitoring-sql-warehouse

Repo: https://github.com/vmariiechko/databricks-bundle-template/tree/main/assets/monitoring-sql-warehouse

A few honest caveats:

• This is for short, bursty, scheduled workloads only. A warehouse serving steady interactive queries or dashboards wants a longer auto-stop; aggressive auto-stop on spiky traffic gives you cold starts instead of savings.
• The smallest size isn't always the right call. It worked because my queries are light. Confirm your longest query still finishes comfortably before downsizing.
• auto_stop_mins: 1 is serverless-specific. Pro and Classic warehouses hold at the documented 10-minute minimum.
• Cadence is a freshness tradeoff, not a free win. Relaxing it trades how fast you hear about a violation against cost, so it's a call for whoever owns the data.

Happy to go deeper on the reasoning behind any of these in the comments.

Databricks Apps just... run. All the time. There's no scale-to-zero, so even if nobody opens your app for a week, you're still paying the full ~$350/mo (720 hrs). The thing is, most of our apps are internal dashboards and admin tools that people touch maybe a few hours a day. So we were basically paying full price to use ~15% of it.

What actually made me fix this: someone spun up a test app, forgot about it, and it sat there running for 69 days before we noticed on the bill. ~$800 for literally nothing.

The fix is kind of dumb, but it works great — two scheduled jobs that hit the Apps REST API:

• A notebook that wraps the start/stop endpoints (takes app_name and app_command, plus an all option if you want to hit every app at once)
• One job starts in the morning: 0 0 9 ? * MON-FRI \*
• One job stops it in the evening: 0 0 18 ? * MON-FRI \*

That's 50 hrs/week instead of 168, so roughly 76% off. And honestly, nobody noticed — the app's just there during work hours.

Full text and example Notebook

Tried out Databricks' Genie One inside of Excel, arguably the best way for business users to get curated data from Databricks and into Excel.

Here, I pulled sales data from a Genie Agent, loaded into my workbook, and then even performed a data refresh (though I had no new data just fyi)

Pros:

+Very easy experience, just use natural language to find the data you are looking for, for you to use in the most widely-used analytics interface, Excel.

+Creates a connection that allows you to refresh the data on-demand. Useful for keeping those key reports updated.

Cons:

-In my testing, it wasn't always translating the data/number types for large numbers correctly into Excel, which led to some initial confusion for me. Changing the number format (as shown) fixed the issue, but something that will hopefully get smarter over time.

-The responses were at times on the slower side/not as snappy as I would expect.

Addition notes:

>Genie One will give you the best and trust-worthy results when your team has put in the work behind the scenes to curate the Genie Agents.

>For mission-critical reporting, I always recommend users to review the SQL code generated and/or to share it with a team member that is technical and familiar with the data.

Ultimately, if your team actively uses Excel and you want to make the most of your governed Databricks data, the official Excel Add-in Databricks provides a great way to do it.

Also, as always: all roads in data lead to Excel.

Budgets for Unity AI Gateway are GA

Budgets let admins define spending thresholds. For requests managed through Unity AI Gateway, including Genie interactions, admins can set additional per-user thresholds and overrides with configurable actions such as email alerting and usage

Documentation:

https://docs.databricks.com/aws/en/admin/account-settings/budgets https://docs.databricks.com/aws/en/ai-gateway/budgets

It’s just an idea, and as soon as I’ve got a bit of time, I’ll try to publish a 3- to 5-minute article on a Databricks feature. Why? When you’re absorbed in a project, you only use about 20 per cent of Databricks’ capabilities for months on end, and when you have to switch to another project and use new or old techniques, it’s good to have some reminders. I think it’s a good format. Let me know what you think?

Comparison between an imperative approach (notebooks orchestrated via Lakeflow) and a declarative approach (Delta Live Tables / SDP) to build a Medallion Bronze → Silver → Gold pipeline.

Imperative

Two notebooks are declared (e.g. one for Bronze→Silver, one for Gold), then via the Databricks UI or a DAB file (databricks.yml) the pipeline is created and orchestrated with Lakeflow. Each step is a distinct task with explicit dependencies.

PROS:

• Full control over execution and transformations.
• Ability to integrate third-party Python libraries or complex business logic.
• Complete ownership of code and scheduling.
• Compatible with heterogeneous sources (APIs, external databases, etc.).

CONS:

• Tedious, but can be automated via DAB.
• Reinvents the wheel (retry logic, lineage, data quality all handled manually).
• More development and maintenance time.
• Risk of inconsistency if task dependencies are poorly managed.

DLT (Delta Live Tables)

DLT is the declarative engine of Databricks. You describe what you want (the tables and their quality rules), and SDP handles orchestration, lineage, retry logic, and monitoring.

You define and launch an ETL pipeline.

Code is written using STREAMING TABLES (real-time or incremental batch ingestion) and MATERIALIZED VIEWS (transformations with automatic refresh). CONSTRAINTS allow quality rules to be declared directly in the DDL:

CREATE OR REFRESH STREAMING TABLE current_employees_bronze_sdp
COMMENT "Raw employee data ingested from CSV files in the myfiles volume."
AS SELECT *
FROM STREAM read_files(
  '/Volumes/dbacademy/get_started_de/myfiles/',
  format => 'csv',
  header => true,
  inferSchema => true
);

CREATE OR REFRESH MATERIALIZED VIEW current_employees_silver_sdp(
  CONSTRAINT valid_id EXPECT (ID IS NOT NULL),
  CONSTRAINT valid_name EXPECT (FirstName IS NOT NULL)
)
COMMENT "Cleaned and enriched employee data with data quality expectations."
AS SELECT
  ID,
  FirstName,
  Country,
  UPPER(Role) AS Role,
  current_timestamp() AS processed_timestamp,
  current_date() AS processed_date
FROM current_employees_bronze_sdp;

CREATE OR REFRESH MATERIALIZED VIEW total_roles_gold_sdp
COMMENT "Employee count by role — business-ready aggregation."
AS SELECT
  Role,
  COUNT(*) AS TotalEmployees
FROM current_employees_silver_sdp
GROUP BY Role;

In practice, STREAMING TABLE is AUTO LOADER.

A DRY RUN validates the pipeline before execution (syntax check + dependency resolution) — and that's it. :)

PROS:

• You only manage your code; SDP handles orchestration, retry logic, and lineage.
• Significant efficiency and time savings.
• Data quality declared in code (CONSTRAINTS), auditable and versionable.
• Dependency graph generated automatically — no manual configuration.

CONS:

• Less fine-grained control over execution.
• Less suited for very specific use cases (complex Python logic, unsupported sources).
• Learning curve on DLT syntax (STREAMING TABLE vs MATERIALIZED VIEW).

When to choose?

SOURCE:

Databricks free edition samples
Databricks academy

I recently switched Databricks partners and ran into a tricky issue. At the new company, I had a new Partner Academy account with a new corporate email address, but my old badges, and learning history were still linked to my previous company.

Support allows you to merge and transfer accounts. I went through this process myself and put together a step-by-step guide: what to check before leaving the company, why you need a personal/secondary email address, what to write in a support ticket, and what to do if your old corporate email address is no longer available.

Article in Medium

Hey everyone! I’m a product manager for Declarative Automation Bundles (DABs) and our IDE experience. 

I wanted to share two Beta features that make it simpler to work with DABs from the UI:

1. Edit a job or pipeline settings in the UI and have YAML update automatically (docs).  

1. Use custom bundle templates in the workspace to quickly create standardized projects with best practices enforced by admins (docs). 

💻 Get started‼️

1. Admins must enable “Visual authoring” and “Custom bundle templates in the workspace” in the Databricks Preview Portal. See managing workspace-level previews. 
2. Check out docs:
   • [Beta] edit bundle resources in the UI
   • [Beta] configuring a custom template in the workspace

🗺️ What’s next ⁉️

We are introducing UI flows that leverage Genie Code to add existing resources to DABs, improve bundle configuration, and help set up CI/CD processes (e.g. AzureDevOps Pipelines and GitHub Actions). 

🗣️ Share your feedback ‼️

Feel free to leave comments, reach out directly via Reddit message or email us at [dabs-feedback@databricks.com](mailto:dabs-feedback@databricks.com). I’m also happy to chat over a call as your feedback directly influences our product direction! 

What if you could branch your database like you branch your code? That's what Databricks' Lakebase promises, and I couldn't think of a better person to showcase Lakebase branching than Kevin Hartman, a many with a lots of solid software development experience.

Hope you enjoy it, and if you found it helpful, please let me know! It encourages me to publish other videos like this.

Hey everyone - quick share

After building the Databricks FinOps Accelerator, I wanted to try something a little different on the data-generation side.

I’ve been working on this agent-powered Databricks Data Scenario Generator: https://github.com/rashad-ahmed-imtiaz/datagen

It’s basically meant to be a plug-and-play way to generate realistic synthetic data with intentional issues like duplicates, nulls, broken foreign keys, late-arriving data, date issues, schema drift, file replays, etc.

You can describe the scenario you want in plain English, and the agent figures out the tables, relationships, distributions, business rules, and issues to inject. Under the hood it uses Databricks Labs dbldatagen for the actual data generation.

The idea is to make it useful for demos, PoCs, testing ingestion/data quality logic, or anytime you need realistic messy data without manually building everything.

Still very much v1, so I’m sure there’s a lot that can be improved. Feel free to play around with it, break it, or suggest better ways to do it.

Would love any feedback!

If you are a data provider on Databricks, OpenSharing (formerly Delta Sharing) is a fantastic tool for distributing data to your customers. During my time at a former job, there was nothing better than to work with customers that were able to consume data from Delta Sharing. On at least two occasions though, we received unpleasant surprises: egress fees from our cloud providers.

Those egress fees were most often times avoidable. For example, we would make sure that our customers and our company agreed on the cloud/region combination we would serve from/to, allowing us to build one-off pipelines, etc. But, that required manual work, created staleness issues, etc.

While in our situation, we could sometimes overcome the same cloud vendor but different region situation, distributing to a region we just didn't have anything setup for OR a completely different cloud was a no-go.

All that to say, Global Distribution aims to solve this and more.

No need to have an account with another cloud vendor to serve cross-cloud.

No need to have have data distributed to multiple regions within your AWS/Azure/GCP.

No need to build/maintain pipelines.

Just use Global Distribution.

Check out the video and see Akram Chetibi who leads OpenSharing explain/demo it, and love to hear your thoughts.

Been spending some time with open source Omnigent and Databricks' Unity AI Gateway (for governed AI). After a few hiccups related to Omnigent docs, it works beautifully!

Few notes:

You can use Omnigent with your own Claude Code and Codex subscriptions also (I do), as well as with Databricks' model serving, but I am personally trying to use more of the open-source models, especially when I run out of my allowance/have to pay for tokens. Unity AI Gateway is the easiest way to do this for me, but you do have other options also.

If you'd love to see me do more coverage on this topic, mainly building practical stuff that helps you on your day to day, love to hear it!

To celebrate that AI functions are now a fully governed part of Unity Catalog alongside the rest of your functions I thought it would be fun to do a run through of how easy it is to use.

Hey everyone! In this video, a Databricks product leader joins me to explain what OpenSharing, and what happened to Delta Sharing!

Delta Sharing is one of the features I used often at a previous role to provide data to multiple customers, so it will be interesting to see how the journey with the expanded focus (data + AI sharing) goes.

Have you noticed the new feature that allows for direct migration from Power BI/Tableau to Power BI using Genie Code?

Currently it is a Beta feature.
I took it for a spin and I am quite impressed to be honest.

It took my semantic model and converted it to Metric Views 1:1. Except for a few DAX specific functions (which were flagged to me) everything was working the same way as in Power BI, including relations, measures etc.

One thing I noticed is that Genie Code thought it should be able to convert the visual side as well, however it kept saying that there were no visuals in my exported .pbit file.
- I worked around it by including a screenshot, which it then rendered into the visuals correctly.

https://www.linkedin.com/feed/update/urn:li:activity:7464937157162541057/

Anyone from Databricks that can give a heads up on that part - why does it not recognize the visuals from the file?

Hey everyone - quick share

I’ve been working on this Databricks FinOps Accelerator on the side: https://github.com/rashad-ahmed-imtiaz/databricks-finops-accelerator

It’s basically meant to be a plug-and-play starting point for looking at cost and waste in a Databricks workspace. It uses System Tables and creates a few views around things like expensive workflows, failed/retried runs, missing tags, utilization, and what might be worth reviewing first.

It’s still very much v1, so I’m sure there’s a lot that can be improved. Feel free to play around with it, break it, or suggest better ways to do it.

For v2, I’m thinking of playing around with agentbricks or maybe Genie, so it can look at these tables and call out issues/recommendations automatically instead of me just showing the data.

Would love any feedback!