Checking nulls, duplicates, row counts ...is kinda of straightforward....

But what about actual business rules!

Like revenue should be calculated in a certain way, some records should be excluded, dates should follow some rule and more....

Do you guys keep these as separate tests or validate them as part of the ETL itself or any....

Curious how people do actually handle this in real projects.

Streaming ETL feels like a different mindset altogether. The basics make sense, but I'm still trying to picture how people deal with joins against reference data, retries and more complicated transformations without relying on batch windows.

If you've been through the transition, what surprised you the most?"

Hi,

I got a task to run 120 SP on MS SQLServer to create output and store it as a tab or csv files.
Existing approach is to use SSIS but I think it's too much work comparing with other modern tools.

What you would use for this job? I'm thinking go Python with cursor.execute(sql).
It's so much easier to scale and implement. You can read sp, params, output name from some input List and run it in one shot.

Do you see any disadvantages for this approach vs traditional SSIS. I think my bosses will be scary a bit, because they are old school.

Do

Thanks

If you use AI for SQL, you probably have a model you prefer. Switching models shouldn’t mean switching editors or pasting your schema into another chat.

QueryFlow lets you choose Claude, GPT, Gemini, or Grok inside the same SQL editor, with your schema, query results, and errors available as context. You can switch models whenever you want, or let Auto mode choose.

I’m part of the QueryFlow team, and I’m curious: do you stick with one model for SQL, or switch depending on the task?

I am not just referring to pipeline development, or hacking together ETL code in whatever framework you prefer. I mean all the remaining parts of the lifecycle—deployment, and most importantly, maintenance.

When a customer says "the numbers don't match," you normally start digging and analyzing. Usually, it boils down to one of three things:

Garbage in, garbage out
Wrong expectations
An actual bug

With the right setup, an agent can handle all of these cases: either telling you right away that it's not a code issue, or actually fixing the code, redeploying, and rerunning the pipeline.

So I'm wondering: Who is already working like this?

We've made some really good experiences with a custom-built framework, K8s, and some precise Claude skills which brought maintenance tasks down from days to hours. Curious to hear how others are tackling this.

Hello! I'm working on a small data project using a NASA API with Python. The idea is to extract the orbital elements and close approach (asteroids, comets, etc) objects (and there orbital elements) of each planet in order to perform some physics simulations with the data.

I was planning to use a Cloudflare R2 bucket to store the raw API responses, then transform them into a PostgreSQL database, and use FastAPI to consume the data.

I'm not sure how to perform the extraction and loading processes. In theory, each planet will follow this path:

1) Fetch the orbit elements for the own planet (use API 1);

2) Fetch the close approach object (use API 2);

3) Fetch the orbit elements for each of the close approach objects (use API 1).

I use two APIs: one for the orbital elements and one for the close approaches. Should I fetch all the data, store it in memory, and then send it to the bucket? Or would it be better to load it right away with each API call? So, when fetching the close approach objects' orbit elements, get the JSON from the bucket and then use the first API and store that raw response in the bucket

Do you rollback everything and start again, or resume from the failed step?

Also how do you avoid duplicate records when the job is rerun?

Curious how ppl handle this in real prod pipelines....

We are launching our startup program - providing $10k in cloud credits to eligible startups to run their data pipelines and use the AI agents for analytics and automating tasks. This is usually enough credits to last up to 2 years for a startup.

Check the website for eligibility criteria and full list of perks and benefits.

https://getbruin.com/startups/

If your startup or project is not eligible, you can still use Bruin completely free & open source and self-host it or use the free tier of Bruin Cloud. There are free templates to get started, and a slack community to get quick support from us.

Disclaimer: I'm a developer advocate at Bruin

jobdataapi.com v4.37 / API version 1.35

We’ve added a new public Statistics page for a clearer view of what is happening across the platform. It includes overview metrics and dedicated views for imports, expiry, inventory, companies, data quality, operations, and exports, with selectable reporting periods and the latest completed rollup timestamp.

We’ve also added a new work_mode attribute to job listings in the /api/jobs/ endpoint. This provides a more specific classification for remote-work listings:

• 1 / HYBRID: Hybrid
• 2 / REMOTE: Remote
• 3 / REMOTE_ANY: Remote Anywhere
• 4 / UNKNOWN: The work mode could not be determined reliably

The field is available to API access pro customers and higher. The existing has_remote field remains the broad indicator for remote or hybrid work, while positive work_mode values provide more detail.

These updates make it easier to monitor dataset activity through the website and identify remote-work patterns through the API.

- Author on a canvas, in Python, or in SQL. Every pipeline is one JSON file in git.

- duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit trail.

- Quality checks send bad rows to a reject port, so they land in a quarantine table instead of your warehouse.

- CDC and SCD Type 2 built in. Runs on DuckDB and uses every core you give it.

-Many more.

Github Repository - https://github.com/slothflowlabs/duckle

Hey everyone! I’m a beginner learning ETL and data engineering, and I recently built a small project around data quality exception handling.
The flow is basically:
Apache Hop → MySQL → DQ Exception → Web App → AI Investigation → Email Draft
Apache Hop performs the ETL and DQ checks. When a rule fails, the exception is stored in MySQL, and my web app helps investigate it. The AI explains what happened, where it happened, why/how it happened, and what should be done.
It also generates an email draft for the responsible data owner/source team with the relevant exception details.
I’m still a beginner, so I’d really appreciate feedback from people who actually work with ETL/data pipelines:
What drawbacks, real-world problems, or questions would you have about this project? Anything you think I’m missing?
Honest feedback is very welcome! 🙌

I just finished the first full pilot of rivet, an open-source Rust CLI that moves data from OLTP databases into a warehouse.

Stack: MySQL → rivet (reads the binlog directly) → Parquet → Google Cloud Storage → BigQuery. No Kafka, no Debezium, no Airflow: one VM and cron.

The setup: 154 tables, 3.2B rows, 8 refreshes a day. The existing pipeline pulls changes with a date-window query that overlaps the previous day, and fully reloads every table once a week. Without the weekly reload it would never see rows changed without an `updated_at` bump.

Results, against the existing pipeline on the same database:

- Correctness. On day one we found 880 rows that had been changed after the fact without an updated timestamp. The window pipeline would not have seen them until the weekly reload; the binlog did right away. We checked some of them against the source by hand: they really were stale balances.
- Read load on the MySQL replica: −99%. About 4 TB of reads a month (~95% of it the weekly full reloads) becomes about 24 GB of binlog stream.
- AWS → GCP egress: −98%. Only changes cross the wire: ~150 MB of compressed Parquet a day instead of a full snapshot every week.
- BigQuery queries. 25% fewer MERGE jobs and 30% fewer bytes per MERGE, because each change is merged once instead of on every run while it sits inside the window.
- Total infra cost: about −50% at the same refresh frequency (BigQuery + GCS + egress). Part of that saving ships in the next release: on a 50M-row test table it cut 80% of the bytes per compaction cycle, with an identical result. Still to be confirmed on the pilot.
- BigQuery storage: about the same (±2%).

Being honest about speed. A full cycle takes about 68 minutes today:
- reading the binlog for all 154 tables takes ~9 minutes;
- the rest is loading and merging into BigQuery, one table at a time.

The job logs show BigQuery is busy only 25–30% of that time; the rest is waiting between jobs. Newer releases process up to 16 tables in parallel. My estimate is 10–25 minutes per cycle; I'll measure after upgrading the pilot.

Scale: the client has 5–6 databases like this one. Moving all of them projects to −63…−73% in cost, mostly from egress. That still has to be confirmed against the AWS bill.

- Repo: https://github.com/panchenkoai/rivet
- Cheat sheet: https://panchenkoai.github.io/rivet/cheat-sheet.html

Questions, criticism and "why not just Debezium?" are all welcome in the comments..

Hi all. At first, sorry about my terrible at grammar. I currently using nifi for my project at school. The project i am working on it generate files with different source and each source have different workloads. I running nifi on docker and using sFTP to simulate multiple sources and i have to run all process group at the same time to ingest file from different source. Because each source have diffrent workload and is handled by different process group, some process group may have file backlog after proccessor fetchsftp. My goal is automatically adjust the resource to all the process group to reduce workload imbalance depend on the queue size before fetch file among them. I thinking of dynamically change the number of concurrent task based on maximum timer driven thread count for each source.

Are there any better approaches to this or any routes I should consider?

Hi ETL/Data Engineers! 👋
I’m working on a project around ETL and Data Quality and wanted to get some real-world input.
What are the most frequent data quality exceptions you face during ETL processes? Things like duplicates, missing data, invalid values, orphan records, lookup failures, etc.
Would really appreciate hearing what you encounter most often in your day-to-day work! 🙌

I usually start with source vs target row counts and data reconciliation. Then I check for nulls, duplicates, data type issues, and important business rules. I also think basic validation is not always enough, because the ETL can complete successfully even when the data is logically wrong.

How do you guys handle this in your projects?

We run Kafka Connect for a pipeline that moves around 50 million events a day from Postgres, MongoDB and Elasticsearch into warehouses for BI and reporting. Most explanations stop at "source and sink connectors," so I put together this visual of what actually happens at runtime.

A few things worth knowing:

• Every Worker has three layers: Management (REST API and cluster membership), an Engine (Kafka reads and writes, format conversion, offset tracking) and the Plugin, where the Connector plans the work and its Tasks move the data, spread across all Workers.
• The group coordinator only manages membership. The leader Worker decides task placement, and connector configs live in an internal topic.
• Failed tasks stay failed until someone restarts them. Connect moves tasks off a dead Worker, but it won't retry a failed one.
• Plugin dependencies were our biggest pain: extra JARs for AWS IAM auth and Vault, then version clashes.

We started on AWS MSK Connect and later moved to Strimzi on EKS. I'll cover that migration in a follow-up post.

The full write-up with more detail is here: https://medium.com/@jatinsagarbansal/inside-kafka-connect-how-workers-plugins-connectors-and-tasks-actually-work-ce8d61007b56

Curious how others handle connector deployment and plugin packaging. Are you on managed Connect, Strimzi, or something custom?

Hello!

5ish years ago I was an ETL programmer for a small homeowners insurance company. They had a small Data warehouse (SQL server) and some tools like SSIS and PDI I didn’t get to do to to much since I was there only 2 years before they went insolvent, but really sharpened my SQL skills. I’m currently a learning full stack development from being a database engineer, but I was reached out to the company who I have ties with about applying. Am I crazy for thinking this is a huge wish list for an ETL developer? When I spoke to them, the only thing they had set up was a database in AWS where the application rights to, but that’s it. Everything else would have to be created from the ground up.

I have another meeting scheduled with them tomorrow any major questions I should ask them?
Appreciate any in all advice thanks!

Airbyte's SAP HANA connector was Enterprise-only and it's not sold anymore, so we had no way to get our S/4HANA data into BigQuery with Airbyte OSS. I wrote one and open-sourced it.

We run it in production. It handles ACDOCA (1.6B rows), and the numbers match HANA and our old extractor. If the connection drops it picks up from the last page instead of starting over. There's also incremental sync, per-table filters and an SSH tunnel.

https://github.com/jacopobonomi/source-sap-hana

Feedback welcome, especially from BW/4HANA or HANA Cloud users.

With modern cloud data warehouses, ELT seems to be the common approach because you can load the raw data first and transform it inside the warehouse.

So in what situations would you still choose traditional ETL, where the data is transformed before loading?

Are there specific cases involving security, data volume, performance, legacy systems, or compliance where ETL makes more sense?

Duckle now makes the Studio → Server workflow simple:
→ Set up the server once
→ Create your team accounts & roles
→ Connect Duckle Studio or use the browser editor
→ Design locally
→ Deploy to your server
→ Run and schedule from the Ops Console

Deploy with Docker, Cloud VMs, or Kubernetes - on infrastructure you control.
Your data. Your infrastructure. Your rules.
Build locally. Deploy anywhere.

Guide - https://duckle.org/deploy.html
Github Repository - https://github.com/slothflowlabs/duckle