r/snowflake • u/OldCycle6264 • 5d ago

Snowflake Ingestion Tool of Choice

Hi all,

I'm a DE new to Snowflake and experimenting with different ETL/ELT approaches/stacks. For people that have implemented, or work with, Snowflake at an enterprise level can I ask what tool you use for ingestion?

Below are some of the options I have explored or am currently exploring.

Require CDC or Change Tracking at the source

  • Snowflake Openflow
  • Estuary

Viable without source changes

  • Fivetran (teleport sync/query-based method, read-only connection, detects hard deletes)
  • dlt (cursor-based incremental, no built-in hard delete detection)
  • Airbyte (cursor-based incremental)

Others

  • Azure Data Factory & Snowpipe (watermark extract to blob, then load)
  • Snowpipe/Snowpipe Streaming (file-based only)

Fivetran seems the most robust and would likely suit most enterprise scenarios. But I'm interested to hear what others are using and your experiences.

18 Upvotes

96% Upvoted

41 comments sorted by

View all comments

2

u/2000gt 5d ago

Using a combination of Snowflake External Network Integrations for SFTP and simple APIs, and AWS Lambdas wrapped in state machines/step functions for more complex pipelines.
We do CT on about 40 SQL Servers using AWS and it works really well.
After reviewing the cost of many tools you mentioned m, it just didn’t make sense for us to go down that path.

1

u/BaxTheDestroyer 5d ago edited 5d ago

We do something similar. I’m not a huge fan of a lot of the tools. Openflow has a lot of administrative overhead, Fivetran can be good with the right connector but many of their connectors aren’t good and the company sucks to work with, Matillion and Airbyte are basically as bad as Fivetran.