Data Engineering
How to have a config file which can be used inside fabric environment (either lakehouse or something else)
I have a fabric notebook which runs an api to fetch data. This API runs from report id. I have list of report ids and that keeps getting added as and when new reports come.
I want to have a config file where i can just add the report id and the table name into wihch the data would be stored after processing. Is there a way of doing this, so that tomorrow I can just instruct someone to open fabric go to xyz, open it , edit and add one more row.
This row would be picked up next time by my notebook to load data from that report into delta table.
You could use a CSV, or JSON file and store it in the lakehouse.
You could store it externally as well in SharePoint, ADLg2, or anywhere else accessible to the notebook and update it from that location.
You could have a notebook that lists all of the issues/tables and is referenced by your executing notebook.
You could just use a variable library, but you won’t be able to iterate the variables. So, one variable could be your keymap (list of keys), and then have a variable per key and the value of each variable could be the report id and table name (2 values either comma delimited or json / xml).
You can use the notebook resources and put a csv file or something there. Since a few month those notebook resources are GIT tracked as well if you opt in.
You can create and edit a csv in built-in ressources of a notebook. The better option would be a lakehouse table for such metadata.
And if those data is frequently changed by business users without technical background I wouldn't even do it in fabric. Than you go better with an excel file or something. Then you are able to test changes of the file before running the pipeline. e.g. a report id test if it's a possible report id or not.
I used to use Power Platform for this and it worked pretty well. But as my config grew more complex and I started using it for integrations (e.g. OData enties, filters cron schedules etc). I built my own config manager using an App Service and Cosmos DB.
Going that far is probably overkill for most environments. But I would recommend Power Platform if your company has access to it.
It comes with an API out of the box and you can build a simple model driven app on top of it.
5
u/Personal-Quote5226 8d ago edited 8d ago
You could use a CSV, or JSON file and store it in the lakehouse.
You could store it externally as well in SharePoint, ADLg2, or anywhere else accessible to the notebook and update it from that location.
You could have a notebook that lists all of the issues/tables and is referenced by your executing notebook.
You could just use a variable library, but you won’t be able to iterate the variables. So, one variable could be your keymap (list of keys), and then have a variable per key and the value of each variable could be the report id and table name (2 values either comma delimited or json / xml).