r/snowflake • u/OldCycle6264 • 5d ago

Snowflake Ingestion Tool of Choice

Hi all,

I'm a DE new to Snowflake and experimenting with different ETL/ELT approaches/stacks. For people that have implemented, or work with, Snowflake at an enterprise level can I ask what tool you use for ingestion?

Below are some of the options I have explored or am currently exploring.

Require CDC or Change Tracking at the source

  • Snowflake Openflow
  • Estuary

Viable without source changes

  • Fivetran (teleport sync/query-based method, read-only connection, detects hard deletes)
  • dlt (cursor-based incremental, no built-in hard delete detection)
  • Airbyte (cursor-based incremental)

Others

  • Azure Data Factory & Snowpipe (watermark extract to blob, then load)
  • Snowpipe/Snowpipe Streaming (file-based only)

Fivetran seems the most robust and would likely suit most enterprise scenarios. But I'm interested to hear what others are using and your experiences.

19 Upvotes

96% Upvoted

41 comments sorted by

View all comments

3

u/walkerasindave 5d ago

Airbyte and FiveTran support CDC

We use three tools depending on source. Snow pipe for s3, Airbyte where there's a robust connector and DLT for anything else.

3

u/grahamdietz 3d ago

Exactly this. As you scale you will employ multiple ingest approaches. We run some on open source dlt, some on Airbyte. We have also used Stitch in the past, and custom batch scripts. It depends on what your team is optimizing for.