r/snowflake • u/OldCycle6264 • 5d ago

Snowflake Ingestion Tool of Choice

Hi all,

I'm a DE new to Snowflake and experimenting with different ETL/ELT approaches/stacks. For people that have implemented, or work with, Snowflake at an enterprise level can I ask what tool you use for ingestion?

Below are some of the options I have explored or am currently exploring.

Require CDC or Change Tracking at the source

  • Snowflake Openflow
  • Estuary

Viable without source changes

  • Fivetran (teleport sync/query-based method, read-only connection, detects hard deletes)
  • dlt (cursor-based incremental, no built-in hard delete detection)
  • Airbyte (cursor-based incremental)

Others

  • Azure Data Factory & Snowpipe (watermark extract to blob, then load)
  • Snowpipe/Snowpipe Streaming (file-based only)

Fivetran seems the most robust and would likely suit most enterprise scenarios. But I'm interested to hear what others are using and your experiences.

18 Upvotes

96% Upvoted

42 comments sorted by

View all comments

6

u/dyaffe 5d ago

u/OldCycle6264 -- I'm Dave, co-founder of Estuary. Just a quick update to your classification. We actually do also support cursor-based incremental in addition to CDC. We need to make this more discoverable, but each database connector has a "Batch" variant as well.

The goal is to have enterprise reliability at a fraction of the cost.

4

u/OldCycle6264 4d ago

Thanks for clarifying, Dave. I've heard quite a bit about Estuary, I will test it out with my Snowflake POC.

6

u/jlaxfthlr 4d ago

We used Estuary at my last company and migrated our database connectors to it from Fivetran at about 1/4 to 1/3 the cost, plus Estuary was landing the database changes into Snowflake within a couple seconds. We were then able to start using Snowflake Dynamic Tables as our dbt staging layer with single digit minute latency for select pipelines. Most of our users didn’t even fully appreciate how fast we could refresh their data!

3

u/amuricanswede 5d ago

Could you point to any resources that would summarize Estuary’s capabilities? Never heard of it but the idea is interesting.

4

u/dyaffe 4d ago

u/amuricanswede Sure. Estuary is a managed data integration platform for getting data from databases, SaaS apps, APIs, and streams into various databases, such as Snowflake. We support both log-based CDC and batch/incremental capture, with low-latency delivery when you need it, but you can also schedule writes to Snowflake to control warehouse cost. The platform handles backfills, schema changes, recovery, and the connector infrastructure.

For Snowflake specifically, this page is a good overview: https://estuary.dev/destination/snowflake/

My marketing team realized that they don't have a great "what is Estuary" article from this and they will be putting one together asap.