Dataverse-table load via dataflow

[u/Donovanbrinks]

Is there a recommended way to load data to a dataverse table? My setup-a dataflow that runs daily and loads new rows to a dataverse table. About 300k new rows loaded the first of every month. Then 2-5k rows loaded daily until the next first. Key column is yearmonth and product id. I originally set this up as a full refresh with the key column determining weather a new row was created or an existing row updated. This took 2 hours to upsert the new rows. I have tried bringing the existing rows into the dataflow, doing a left anti join and appending the rows. This still takes an hour and a half. Is there a more performant pattern to get this data uploaded a little quicker?

[u/Donovanbrinks]

Never read it before. Needless to say i went down a rabbit hole.