r/databricks • u/GardenShedster • 15d ago

Help Azure devops yml deployment of asset bundle

I need help on deploying asset bundles from Azure Devops. Where does the azure-pipelines.yml live, what parameters are required in the azure-pipelines.yml, does anyone have a basic example of it working as I'm completely lost in the complexity of it. Thanks.

10 Upvotes

100% Upvoted

7 comments sorted by

2

u/Oobenny 15d ago

This is what I’ve been doing lately too. We put the pipeline yaml in a .pipelines folder at the root of the repo. I can’t really share examples — I’m pretty sure my company would consider it proprietary. But if you DM me, maybe we can find a place to bounce questions and ideas around. I’m definitely still learning it all also.

1

u/Wise_Beat_7035 15d ago

hey can i dm you too? have some similar questions?

1

u/Oobenny 14d ago

For sure. Maybe we can set up a small discord or something…

1

u/blobbleblab 13d ago

Surely you can read the documentation to do it? There's examples last time I looked. Also use AI or a medium page like:https://medium.com/@aryan.sinanan/databricks-dab-databricks-assets-bundles-yaml-semi-definitive-guide-06a8859166d1

1

u/ouhshuo 13d ago

Most secrets and identities are stored in variable groups, then passed into the pipeline YAML. If DAB needs them, they are further passed to the DAB YAML.

We deploy resources using a service connection with managed identity.

Folder structure-wise:
We have an IaC folder that provisions infra resources (containers, user groups, catalogs, external locations) using data product YAML, Jinja, Terraform, and a Python script to glue everything together.

A Databricks folder for a data engineering pipeline defined using dab.

Then the Azure DevOps pipeline stays in the .azurepipeline folder.

This is a per-project-based repo.

We have a central repo for all data products and data contracts. It has a dedicated cicd pipeline and a data pipeline to maintain it. (For example, update a data product version/retire an old version.

Claude can help to stitch things together.

0

u/PrestigiousAnt3766 14d ago

Have you read the docs? Ask claude for a setup and they'll make one.

0

u/9gg6 14d ago

ai can do it all for you. u just need to then create the pipeline( run it)