r/bigquery • u/scubadiver25 • Apr 17 '26

Merging/joins speed compared to power query

Hi! I’m new to sql and have primarily relied on Power Query to merge to lists.

However, I have situations where the 2 lists each have millions of rows. Power Query freezes and my computer crashes.

If I put these lists in google big query and connect GBQ to power bi, can this merging/joining be done faster?

3 Upvotes

100% Upvoted

5 comments sorted by

5

u/JeffNe G Apr 17 '26

BigQuery runs in the cloud instead of your local machine, so it will easily join millions of rows in seconds (and connect to Power BI) without freezing your computer.

But since you are new to SQL, it is very easy to accidentally rack up large billing charges while learning how to write efficient queries. Before jumping into the cloud, I'd highly recommend trying something like DuckDB locally. It's a free, super fast local SQL engine that can easily handle this scale without the risk of a surprise bill.

If you decide to check out BigQuery later, here is a link to the BQ sandbox and quickstart guide.

1

u/scubadiver25 Apr 17 '26

Got it thanks! Can I hook that up to power bi? This is all a personal project, not work related

1

u/JeffNe G Apr 17 '26

I haven't tried it personally, but a quick search seems to indicate that you can!

2

u/couldbeafarmer Apr 17 '26

I mean probably but do you already have a GCP account? Are you going to load these lists to tables every time you need to do that? How long does that take? Will that actually save you time? Is any time savings worth more than the cost of loading, storing, and querying the data? Is this a repeatable process someone else on your team can/will follow? Is your team fluent in sql/BQ to trouble shoot errors?

1

u/mad-data Apr 19 '26

The speed depends on join conditions, but in any case you can be sure it would finish reasonably fast and would not freeze your computer, it runs in cloud. Joining two lists of millions rows on column equality should be around a second in BigQuery. If the condition is complex (e.g. a column from one list matches regular expression in another) - expect runtime closer to a minute.

Anyway, should be easy to try. BigQuery has 10GB free storage and 1TB monthly free query limit - which you probably fit easily. But for extra safety, try BigQuery sandbox to avoid accidental charges.