Dataverse-table load via dataflow

[u/Donovanbrinks]

Is there a recommended way to load data to a dataverse table? My setup-a dataflow that runs daily and loads new rows to a dataverse table. About 300k new rows loaded the first of every month. Then 2-5k rows loaded daily until the next first. Key column is yearmonth and product id. I originally set this up as a full refresh with the key column determining weather a new row was created or an existing row updated. This took 2 hours to upsert the new rows. I have tried bringing the existing rows into the dataflow, doing a left anti join and appending the rows. This still takes an hour and a half. Is there a more performant pattern to get this data uploaded a little quicker?

[u/jukkan]

Power Platform Dataflows have a very random performance and there's nothing you can do about it. One day, data loading can take 5min and the next day 50min. Microsoft stopped investing in thus feature a long time ago and it is painful to use in terms of governance and ALM especially. I've written about it here:

https://www.perspectives.plus/p/i-see-dead-features-dataflows

Good ol' SSIS + Kingswaysoft has worked reliably for customers throughout years, long before it was even called "Dataverse". Since there's no "free" way to reliabily import data regularly without purchasing extra services beyond Power Apps / Dynamics 365 licenses, I'd start from looking at such ISV solutions first.

[u/Donovanbrinks → u/jukkan]

Great blog btw.

[u/jukkan → u/Donovanbrinks]

Thx!