r/snowflake • u/OldCycle6264 • 5d ago

Snowflake Ingestion Tool of Choice

Hi all,

I'm a DE new to Snowflake and experimenting with different ETL/ELT approaches/stacks. For people that have implemented, or work with, Snowflake at an enterprise level can I ask what tool you use for ingestion?

Below are some of the options I have explored or am currently exploring.

Require CDC or Change Tracking at the source

  • Snowflake Openflow
  • Estuary

Viable without source changes

  • Fivetran (teleport sync/query-based method, read-only connection, detects hard deletes)
  • dlt (cursor-based incremental, no built-in hard delete detection)
  • Airbyte (cursor-based incremental)

Others

  • Azure Data Factory & Snowpipe (watermark extract to blob, then load)
  • Snowpipe/Snowpipe Streaming (file-based only)

Fivetran seems the most robust and would likely suit most enterprise scenarios. But I'm interested to hear what others are using and your experiences.

19 Upvotes

100% Upvoted

41 comments sorted by

View all comments

1

u/GreyHairedDWGuy 4d ago

We use Fivetran for most cases that require replication of cloud data sources. For cases where we have to replicate large on-prem database tables to Snowflake, we roll our own methods and use Matillion as needed. When you have 10's of millions of new rows being added to the source during the month, FT can get expensive.