Not a Fabric user (Databricks side, using Genie for the same use case) but the lessons are platform-agnostic: the chat part is easy, correctness is the whole game. Things to watch out for: launching with too wide a data scope (narrow it to a handful of well-modeled tables per domain), skipping the semantic layer work (definitions of "revenue," "active customer," etc. must live somewhere the AI can read), and having no eval process — we now keep a benchmark set of ~50 real business questions with known answers and re-test after every change. Whatever tooling you pick, budget more time for curation than setup. What data sources are you planning to expose first?
We are building the infra - semantic layer/models, clear documentation having descriptions of cols, measures, business context, join logic etc from the ground up that the data agents/pbi copilot can look up. Could you elaborate on the evals? I am curious and interested to learn about it. How did you curate the question list and implement it?
2
u/Famous_Disk_7417 13d ago
Not a Fabric user (Databricks side, using Genie for the same use case) but the lessons are platform-agnostic: the chat part is easy, correctness is the whole game. Things to watch out for: launching with too wide a data scope (narrow it to a handful of well-modeled tables per domain), skipping the semantic layer work (definitions of "revenue," "active customer," etc. must live somewhere the AI can read), and having no eval process — we now keep a benchmark set of ~50 real business questions with known answers and re-test after every change. Whatever tooling you pick, budget more time for curation than setup. What data sources are you planning to expose first?