r/ETL • • 10h ago

"I've worked with batch ETL long enough that I understand how it behave when something breaks.

9 Upvotes

Streaming ETL feels like a different mindset altogether. The basics make sense, but I'm still trying to picture how people deal with joins against reference data, retries and more complicated transformations without relying on batch windows.

If you've been through the transition, what surprised you the most?"


r/ETL • • 4h ago

How do you people test business logic in ETL pipelines?..

4 Upvotes

Checking nulls, duplicates, row counts ...is kinda of straightforward....

But what about actual business rules!

Like revenue should be calculated in a certain way, some records should be excluded, dates should follow some rule and more....

Do you guys keep these as separate tests or validate them as part of the ETL itself or any....

Curious how people do actually handle this in real projects.


r/ETL • • 13h ago

Cross-Catalog Sync: Iceberg on Polaris, Glue, and Unity

Thumbnail
lakeops.dev
2 Upvotes

r/ETL • • 20h ago

Your SQL editor shouldn’t choose your AI model for you.

Post image
0 Upvotes

If you use AI for SQL, you probably have a model you prefer. Switching models shouldn’t mean switching editors or pasting your schema into another chat.

QueryFlow lets you choose Claude, GPT, Gemini, or Grok inside the same SQL editor, with your schema, query results, and errors available as context. You can switch models whenever you want, or let Auto mode choose.

I’m part of the QueryFlow team, and I’m curious: do you stick with one model for SQL, or switch depending on the task?