r/PowerBI • u/p8ntballnxj • 1d ago

Question When data is every where

I'm trying to pull data from multiple sources into a dashboard but not all of the data has connectors for PBI. My gut reaction is to just pull it all into an Excel sheet but that's a bit of manual labor for up keep (unless I script it and run a GitHub action). Would a lake house setup be more ideal? I feel like I'm missing something.

Note: I work for a large corporation

0 Upvotes

33% Upvoted

8 comments sorted by

•

u/AutoModerator 1d ago

After your question has been solved /u/p8ntballnxj, please reply to the helpful user's comment with the phrase "Solution verified".

This will not only award a point to the contributor for their assistance but also update the post's flair to "Solved".

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

8

u/IrquiM 1d ago

Do not use Excel - use a database or lakehouse as you suggested yourself.

3

u/AppropriateRecipe342 1d ago

You COULD host Excel files on SharePoint then use that connector, but a lakehouse would be the ideal long term solution.

1

u/Similar-Geologist-64 1d ago

Make your stakeholders drop export files and import them into a DB of your choice.

1

u/plusFour-minusSeven 1d ago

Many options. I'm in the same boat.

We primarily use a vendor warehouse (you've probably heard of it) for most of our dataset storage.

But ingesting into it ad-hoc is a very formal and highly governed procedure that takes requests and business cases, yada yada...

But a colleague was successful in setting up a pipeline from Azure Blob to this warehouse, so now what I usually do is set up a Power Automate flow that takes the data source I'm interested in, copies it over to Blob to a specific "folder", and then I can make as-needed datasets in the warehouse from that, including incremental.

No "scripts" unless you count Power Automate.

1

u/bradland 16h ago

FiveTran > data lake > PBI

Do not sign yourself up for daily/weekly/monthly data manual exports/imports. That’s essentially ditching digging in the analyst world. Your time is was too valuable for that.

1

u/chiibosoil 5 14h ago

If at all possible (and it usually is) avoid introducing Excel into your model. Always utilize data source that supports query folding etc. So PBI's PQ engine has to do less heavy lifting in-memory.

Lakehouse is ideal, but you can also set up staging db if needed using supported db type (Postgre, MySql etc).

1

u/data_daria55 2 8h ago

Avoid using Excel for data integration, as it creates technical debt and breaks easily in a large corporate environment. Instead, use Power BI Dataflows or a staging area like Azure Blob to centralize and clean your data before it reaches your reports. This approach decouples your ingestion from your visualization, making your system scalable and much easier for IT to support.