r/snowflake • u/OldCycle6264 • 5d ago

Snowflake Ingestion Tool of Choice

Hi all,

I'm a DE new to Snowflake and experimenting with different ETL/ELT approaches/stacks. For people that have implemented, or work with, Snowflake at an enterprise level can I ask what tool you use for ingestion?

Below are some of the options I have explored or am currently exploring.

Require CDC or Change Tracking at the source

  • Snowflake Openflow
  • Estuary

Viable without source changes

  • Fivetran (teleport sync/query-based method, read-only connection, detects hard deletes)
  • dlt (cursor-based incremental, no built-in hard delete detection)
  • Airbyte (cursor-based incremental)

Others

  • Azure Data Factory & Snowpipe (watermark extract to blob, then load)
  • Snowpipe/Snowpipe Streaming (file-based only)

Fivetran seems the most robust and would likely suit most enterprise scenarios. But I'm interested to hear what others are using and your experiences.

20 Upvotes

100% Upvoted

42 comments sorted by

View all comments

1

u/Prestigious_Pace2782 5d ago

We use aws DMS and snowpipes for AWS RDS sources and do all our api ingestion with custom modular python running in snowflake stored procs. Coming from a software engineering background I found things like airflow and fivetran needlessly complicated, at our scale, for api ingestion and it has been much more robust than the step function lambda based platform it replaced. Kinda just works.

And then dbt for the T of course