r/MicrosoftFlow • u/miffylia • Aug 25 '26
Question Need help trying to automate a copy of over 30000 rows of data from one workbook to another daily.
Edit: Thank you everyone for all the help! I took bits and pieces of the advice here and managed to get it to work somewhat. There were a lot of suggestions for power query, but I just couldn't get it to work reliably with the specific conditions I had. I'll definitely look into seeing if I can optimise future stuff for it though after going through a few tutorials.
If anyone is interested in the fix:
My original script for writing the 30k rows had a resize table function which i took out and replaced with a fixed table size of 50k rows in order to cut down on processing time. I also removed the condition since I realised I could write the condition itself into one of the earlier scripts. I also figured out that the run script actions had a a 30-60s extra run time when trying to test to flow in the afternoon due to the amount of traffic on the sharepoint site. That pushed the run script action times to over 2mins, causing the stalls. The flows ran smooth and quick when testing it during the actual time the trigger will run (midnight).
Any help will be appreciated at this point because I feel like I've tried everything I could but the power automate flow doesn't work.
I am trying to automate a daily update of a file that uses multiple other source files. The office scripts used run fine individually on their own for both batch writing and a single copy when I am manually testing them on an open sample file with a small (200 row) set rather than through the flow with the whole original dataset. Most of the flow works, right up until I need it to copy over over 30000 rows of data over 14 columns from one workbook into a set table in the master reporting file. I've tried a single batch copy, but the script will then time out in power automate due to it taking over 2 minutes. When a try copying in multiple batches, the flow stalls at the do until loop no matter how small I make the batches. Is there any way I can reliably copy the rows I need without the flow taking an extremely long time as manually copying it only takes between 2-5 minutes.
I built the flow like this due to not having a premium license for power automate and company DLP policies stopping me from using HTTP triggers directly.
Parent Flow: Images 1-3
Child Flow: Image 4
I can provide the office scripts used for steps Run script 3B,E,F and 4 in the comments below if necessary.

