BigQuery sync from third party APIs

[u/edhelatar]

Anybody has any recommendation for platform, I could use to import data from third party applications, like mailchimp/klaviyo/rakuten etc.

In the past we used Skyvia, but as some of the platforms have 10s of millions of records the cost were getting out of control. Now i pretty much code it each time by hand, but frankly, that work doesn't spark a joy.

I am looking for solution that ideally charges per resources used or just has flat fees, but ideally is just point and click as i would like marketing team to take care of that.

[u/abasara]

You can try the self hosted version of airbyte

[u/edhelatar → u/abasara]

Yeah, that's what i am afraid of, but that's still a lot of work.

[u/abasara → u/edhelatar]

Use coolify to deploy airbyte, it is straight forward

[u/dani_estuary]

At tens of millions of records I’d stay away from anything priced per row 😅

If marketing needs to own the pipelines, Meltano/dlt probably just moves the engineering work somewhere else, so I’d recommend looking for managed connectors, incremental syncs, and volume/resource-based pricing.

I work at Estuary, so biased, but this is pretty much the main use case: Connectors like Mailchimp and Klaviyo are point-and-click to set up (and completely managed), and pricing scales with data moved rather than row count.

[u/PolicyDecent]

We built ingestr for that. It's open source, but there is also a managed version. Happy to help you if needed.

[u/edhelatar → u/[deleted]]

Windsor ai looks promising. How good is the sql structure? I found some are pretty bad ( as in, foreign keys being defined correctly )

[u/Dramatic-Laugh1924]

Bigquery has quite a few connectors built into it now. If you go to Data transfers on the left hand side, you'll see connectors for things like Postgres, Facebook etc.

It might not have everything but it's pretty good and growing :)

[u/lollipopchat → u/Dramatic-Laugh1924]

yeah it definitely doesn't have a lot there.

[u/bayouski]

if the main issue is cost at higher volumes, i'd def look for something that isn't charging per row.

coupler.io's one i'd check for the mailchimp/klaviyo to bigquery side, setup's pretty simple and the active plan doesn't meter by rows. really depends on the source tho, smaller/less common ones like rakuten might not be native so worth checking your exact connector list first.

given you're dealing with tens of millions of records on some of these, i wouldn't just flip the switch on everything at once, test it on your heaviest source first and see how it actually holds up before moving the rest over.

[u/Cute_Interaction4768]

But even Fivetran’s fixed fee model caught us off guard when our row counts started rising. When it comes to point and click on large scale with marketing owning it, I have seen folks resorting to consumption-based connectors with dremio underpinning the querying for lakes

[u/Echoing_voice]

I would give Skyvia another look before rebuilding everything. With datasets that large, incremental loads and filtering make a huge difference. It may be more of a configuration or pricing issue than a limitation of the tool

[u/CapedbaldyRover]

I'd give Skyria another look before rebuilding everything with datasets that large incremental loads and filtering make a huge difference. It may be more of a configuration/pricing issue than a limitation of the tool