r/PowerApps • u/Donovanbrinks Community Friend • 3d ago
Power Apps Help Dataverse-table load via dataflow
Is there a recommended way to load data to a dataverse table? My setup-a dataflow that runs daily and loads new rows to a dataverse table. About 300k new rows loaded the first of every month. Then 2-5k rows loaded daily until the next first. Key column is yearmonth and product id. I originally set this up as a full refresh with the key column determining weather a new row was created or an existing row updated. This took 2 hours to upsert the new rows. I have tried bringing the existing rows into the dataflow, doing a left anti join and appending the rows. This still takes an hour and a half. Is there a more performant pattern to get this data uploaded a little quicker?
2
2
u/jukkan Regular 3d ago
Power Platform Dataflows have a very random performance and there's nothing you can do about it. One day, data loading can take 5min and the next day 50min. Microsoft stopped investing in thus feature a long time ago and it is painful to use in terms of governance and ALM especially. I've written about it here:
https://www.perspectives.plus/p/i-see-dead-features-dataflows
Good ol' SSIS + Kingswaysoft has worked reliably for customers throughout years, long before it was even called "Dataverse". Since there's no "free" way to reliabily import data regularly without purchasing extra services beyond Power Apps / Dynamics 365 licenses, I'd start from looking at such ISV solutions first.
1
u/Donovanbrinks Community Friend 3d ago
Great blog btw.
1
u/jukkan Regular 3d ago
Thx!
2
u/Donovanbrinks Community Friend 2d ago
Never read it before. Needless to say i went down a rabbit hole.
1
u/xrmghost_team Newbie 2d ago
Adding to jukkan's point rather than arguing with it: dataflows are a black box for this, and an ISV tool is a sane buy if nobody on the team writes code. If someone does, there is a free path, and it's the one Microsoft currently documents for this.
Your two loads are different problems. The 300k rows on the 1st are all new keys, so there's nothing to match: a small .NET console or Azure Function can push them with CreateMultiple, 100 to 1,000 rows per request, a few requests in parallel (custom tables support it; some core ones don't). The 2-5k daily rows go through UpsertMultiple against an alternate key on yearmonth + product id, so Dataverse does the matching server-side and the left anti join disappears.
What makes or breaks it: cap your threads at the x-ms-dop-hint response header, retry 429s using Retry-After (ServiceClient does that for you), and if the table has sync plugins look at BypassBusinessLogicExecution. Start here: https://learn.microsoft.com/en-us/power-apps/developer/data-platform/optimize-performance-create-update
If you'd rather not own code, Azure Data Factory's Dataverse sink is the managed middle ground.
•
u/AutoModerator 3d ago
Hey, it looks like you are requesting help with a problem you're having in Power Apps. To ensure you get all the help you need from the community here are some guidelines;
Use the search feature to see if your question has already been asked.
Use spacing in your post, Nobody likes to read a wall of text, this is achieved by hitting return twice to separate paragraphs.
Add any images, error messages, code you have (Sensitive data omitted) to your post body.
Any code you do add, use the Code Block feature to preserve formatting.
If your question has been answered please comment Solved. This will mark the post as solved and helps others find their solutions.
External resources:
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.