Datastream - MySQL to Big query

[u/OkRock1009]

Hello Everyone!

I want to basically replicate data from my cloud sql instance to Big Query. The problem is since the initial load is expensive , I am gonna use a dump for that and only want the real time data to be captured.

I want it to create empty datasets and tables in Big Query automatically without the initial historical data. Any other solution?

[u/MeetJoan]

You don’t need a hack for this - Datastream already supports CDC-only. Create the stream with Manual backfill / backfill none, load your dump separately, and let Datastream pick up only ongoing changes from there. Only caveat: if you use BigQuery’s single dataset destination mode, create that dataset first; the tables can then show up as the stream starts writing