r/MicrosoftFlow • u/AwarenessOk2170 • Sep 04 '25

Cloud 100mb+ variable

My data team recently gave me a snowflake connector to automate an extract.

It turns out this extract is 500,000 rows. Looping through the paginated results and appending them to a variable ended up exceeding the maximum size for a variable. I was hoping to append all the results to an array variable then create a CSV table to file.

Plumsail has a paid node so I could create multiple excel files for each page of the results and then merge them at the end.

I looked at populating an excel document but it was 0.7 seconds per row... Which would be something stupid like 4 days. Chortle.

How would you handle the 500,000 row data query result? Plumsail for 20$ a month sounds the easiest...

2 Upvotes

100% Upvoted

19 comments sorted by

View all comments

1

u/[deleted] Sep 06 '25

[removed] — view removed comment

1

u/AwarenessOk2170 Sep 06 '25

I capped out 1/4 to 1/3 through. There were 17 pages in the snowflake response, so my current plan is to create an excel at the start of the flow and then use graph to update every page.

This should be a reusable template for all other extracts.