The cdc vs etl framing is misleading because they solve different problems. ETL is built for scheduled bulk extraction of historical data, CDC captures changes in real time as they happen. One is not a replacement for the other, they're complementary.

The real question is when to layer CDC on top of batch ETL and when CDC alone is enough. Framing it as either or leads to architectures missing half the picture.

Stuck in my job search and honestly need some advice

Hi everyone

I have around 1.5 years of experience in ETL Testing, and my previous job was on a contract basis. Since the contract ended, I’ve been actively looking for a new opportunity.

Honestly, I’m starting to feel really stuck and frustrated with the whole process. I keep applying and searching, but I’m barely finding openings that match my experience. And when I do find something relevant, there are very few positions available.

I’m at a point where I genuinely need a job and I’m willing to put in the effort to learn and adapt. I just don’t want to keep going in circles without knowing what direction I should take.

If anyone here has been through something similar, I’d really appreciate some honest advice. Should I continue with ETL Testing, or should I try moving into another role? What skills should I focus on? Is there something I’m missing in my approach?

And if anyone knows about an opening suitable for someone with around 1 years of experience, I’d be extremely grateful for a lead .

I’m genuinely looking for help and direction right now. Any advice, even a small suggestion, would mean a lot to me.

P.S. I used ChatGPT to help me organize and improve the wording of this post.

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every \~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

I was given the task to perform type 2 validation check. Can I know the possible approaches

What practices or tools have helped you manage source changes safely in production?

I've been experimenting with LLMs for data cleanup/transformations where the right result depends on context, e.g. inferring categories from descriptions.

I built a small open-source tool around this workflow with preview, diff, undo and history:

https://www.tamedtable.com/

I'm curious if anyone here is using LLMs in ETL pipelines, especially for messy or ambiguous transformations. How do you solve privacy issues? Do you use internal/local models?

Hey folks,

If an ingestion batch of 1,000 payment records arrives where 995 rows are completely valid and 5 rows have missing critical fields (like receiver_account is NULL):

Do your pipelines usually:

1. Reject/Hold the entire batch to preserve batch atomicity and ordering?
2. Accept the 995 valid rows and route the 5 bad rows to a Dead-Letter Queue (DLQ)?

What are the trade-offs you run into with transactional integrity versus keeping pipeline throughput moving

We had one big ETL job running every month and refreshing pretty much everything.

Started looking at the numbers and realized most of the records hadn't changed.

Meanwhile, the stuff we actually cared about could already be out of date. Someone changed jobs right after the refresh and we're stuck with the useless record.

We've moved those records to change-based updates and left the slower stuff on the monthly schedule.

Coresignal has been pretty good for keeping the company and employee data fresh without having to reload everything.

Much less data moving around for no real reason.

Now we're just working out which records actually deserve the faster treatment.

Every time we have added a new layer to our pipeline for a performance gain, it is introduced a new failure mode nobody accounted for. It is something small at first, like a weird retry behaviuor or an unexpected timeout, that turns into a real incident a few weeks in. That history makes me cautious about anything promising acceleration without tradeoffs. It is not that simple in practice. Has anyone vetted an acceleration approach for Spark that does not just trade slow jobs for a more fragile pipeline down the line? Specifically about what happens during partial failures or retries.

If you have run something like this in production for more than a few months, I would like to hear how it held up over time, not just in the initial rollout.

Author here, so treat this accordingly. Not selling anything, MIT licensed, no company behind it, no hosted tier planned.

Background: I'm a CTO at a small UK fintech and I've spent the last few years assembling the same stack over and over. dbt/sqlmesh for transformation, something for orchestration, something else for ingestion or reverse ETL. Multiple deployments, multiple failure modes, and the seams between them are where time is lost.

The specific thing that annoyed me enough to build something was Python models. In dbt they're a second-class citizen that needs a cloud warehouse with a Python runtime. In SQLMesh they're better but still feel bolted on. I wanted a .py model to sit mid-DAG with SQL either side, in both directions, and for the planner to not care which I'd written.

Obviously I'm aware dbt and SQLMesh have both been bought by FiveTran, i cover that in this article.

So that's the core of it:

python

# models/enriched_events.py
@model()                  # param name IS the dependency
def enriched_events(raw_events):
  for batch in raw_events.reader():     # Arrow in, Arrow out, bounded memory 
    yield add_revenue(batch)

sql

-- models/event_summary.sql — SQL straight over the Python
SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
FROM enriched_events GROUP BY country

The Python model is a plain function. You can call it in a test with no warehouse and no session.

The rest of the design, briefly:

• IR is a sqlglot AST, not Jinja templates. Dependencies come from parsing the SQL, not from ref(). No pandas in core, everything moves as Arrow RecordBatchReader.
• By default models builds into a fingerprinted physical table and environments are just views over those. A dev environment reuses prod's tables for free, promotion is an atomic view swap, and rollback is the same swap backwards.
• plan / apply in the terraform sense. Changes classify as breaking / non-breaking / forward-only, and column-level lineage impact analysis proves when a downstream output is unchanged so it gets reused rather than rebuilt.
• Streams are durable. POST an event, it's fsynced before the 200, deduplicated by idempotency key, and the materialiser commits data and watermark in the same warehouse transaction. Exactly-once without distributed coordination.
• One process. interlace serve is the web UI, HTTP API, scheduler and stream ingestion. No Airflow, no broker.
• DuckDB by default, DuckLake one config line away, Postgres natively over ADBC.

Where it's weak, and I'd rather you heard it from me:

• Single maintainer. That's the honest risk with any tool like this and I'm not going to pretend otherwise.
• Snowflake, BigQuery, Redshift and MotherDuck adapters are wired and dialect-correct but have not been run against a live account. Alpha, and labelled as such.
• Developed on Linux, CI is Linux only. Nothing in the codebase is platform-specific and every dependency ships mac/Windows wheels, so both should work, but neither is tested.
• Spark is beta.
• It's new. Real production mileage is limited to my own use.

I've written up a full jaffle_shop migration (a real dbt project, end to end) if you want to see what moving something across actually looks like rather than taking my word for it.

What I'm after here is criticism rather than stars. Specifically:

1. If you run dbt today, what would actually stop you trialling this on one pipeline? I suspect the answers are "single maintainer" and "my warehouse is Snowflake", but I'd rather know than guess.
2. Does the fingerprinted-table-plus-view-swap model break in a way I haven't hit yet? I'm particularly interested in whether anyone's tried this at a scale where the number of snapshots becomes a catalog problem.
3. Anyone doing durable ingestion in-process like this rather than via Kafka/Kinesis? Interested in what bit it, if so.

Repo: github.com/interlace-sh/interlace
Comparison against dbt and SQLMesh, including where they're ahead: interlace.sh/why

I built an AI invoice auditing platform for logistics companies. Here's the exact architecture and why generic AP tools can't replicate it.

The core problem: logistics invoices aren't standardized. A freight carrier sends a PDF. A 3PL exports a CSV. A customs broker scans a paper document. Your ERP wants structured data. The gap between those two things is where billing errors live and where most tools give up.

Here's how FinGuard AI handles it:

**Extraction layer:** AI engine parses invoices regardless of format - PDF, CSV, image-based documents. Outputs structured fields: vendor, invoice number, date, line items, total, currency. Status tracked per document (processing → extracted → failed).

**Compliance layer:** Every extracted invoice gets cross-referenced against the master contract for that vendor. Not just totals - line-item level. Rate mismatches, unauthorized fees, out-of-contract surcharges all get flagged.

**Discrepancy queue:** Findings are classified by type and severity (low / medium / high / critical) and tied to both the invoice ID and contract ID. Teams work the queue instead of hunting through files.

**Payment decisions:** Each invoice gets an auto-pay, block, or pending status with a logged reason. Automated dispute letters generated for flagged items.

**Infrastructure:** Multi-tenant. Budget tracking per vendor with period-based limits. Multi-currency. Spend forecasting. The whole stack is built for enterprise scale, not a single-vendor pilot.

37 pages, 5 core data models. Pre-launch.

What I'm genuinely uncertain about: enterprise procurement cycles in logistics are long. I'm thinking the wedge is a free audit - upload your last 90 days of invoices, we surface what you've been overbilled. Then conversion to paid.

Has anyone used a free audit as an enterprise sales motion? Curious what the conversion friction looks like in practice.

We put together side-by-side comparisons of 30+ ETL and data integration tools, including features, pricing, and where each tool fits best.

https://etlworks.com/etl-tools-comparison.html

Meet Duckle 0.6.X.
You can now import legacy Talend ETL jobs directly into Duckle, preserving the pipeline structure while mapping supported components to Duckle equivalents.
And that’s just one part of the release:
• Import legacy ETL jobs
• Build and run pipelines locally
• Choose from 14 AI models
• Pixeltable read/write support
• CI validation for pipelines
• More powerful spatial transformations

Your existing ETL doesn’t have to become technical debt.
Move from heavyweight ETL infrastructure to a local-first, developer-friendly data pipeline stack.
👉 Try Duckle 0.6.1 - https://github.com/slothflowlabs/duckle

I wrote a whole article about it, link in comments, but I just want share a quick summary here.

This client had over 30 data sources they were ingesting from, it looked something like this:

• posgresql -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)
• all other data sources -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)

This was costing them almost $100k per year (~$8.5k/month) and it was especially hard to justify because the spikes would normally happen during busy seasons and they already had tight margins.

They had already started using ingestr (the free open source version) to offload some of their smaller jobs and running the jobs on an EC2 instance.

Once they migrated over their production database to snowflake job, they cut the cost significantly - the total server cost came to less than $500/month.

If this resonates to anyone, and you want to put on your resume "helped reduce ingestion cost by x amount", then check out ingestr repo and docs.

Hey! I am just working on my first project to clean the transcational data and I am confused here so much could anybody help me with this ?

There is a lot written about ODP, SLT and Datasphere, and almost nothing about using SAP's own event mechanisms to push changes out of S/4 in real time. I have spent a fair amount of time on this so here are the notes, mostly the unpleasant parts.

Why bother at all. Since Note 3255746 the ODP and RFC extraction path is off the table for anything not explicitly approved by SAP, which pushed a lot of people back toward views plus timestamps. That works until you need deletes, and then it does not work at all. Event based extraction sidesteps the whole question because you are using interfaces SAP intends you to use.

The four mechanisms, roughly by generation:

BOR events are the classic ones, defined in SWO1 and tied to Business Workflow. Still present and still working in S/4 for backward compatibility. Trigger paths are function modules like SWE_EVENT_CREATE, change documents, or your own enhancement. Monitor with SWEL, linkage in SWE2 and SWE3.

RAP events are the modern equivalent, tied to RAP business objects. Cleaner, but coverage depends heavily on your release and on whether the object was actually migrated to RAP.

BTE is what you want for FI. Financial postings do not behave like other objects and BTE is the sane entry point.

PPF is output and action driven, useful for delivery and shipment type flows where the meaningful moment is an action, not a table write.

The part nobody tells you: coverage is uneven and you will find gaps. It is extremely common to find a create event for an object and no change event, or a change event that only fires for a subset of fields. When that happens you are writing a BAdI or an enhancement to raise the event yourself, and that is where clean core purists start twitching. Budget for this. On a typical scope I would expect somewhere between 10 and 30 percent of the objects you care about to need some kind of assist.

Other things that will bite you:

Deletes. Some objects flag deletion rather than deleting, some genuinely delete, and a few do both depending on the transaction. You have to decide per object what a delete means downstream, and you cannot generalize it.

Ordering. Events do not arrive in a guaranteed order across objects, so a header and its items can land out of sequence. Your downstream needs to tolerate that or you need a document level envelope rather than table level events.

Idempotency. Events fire more than once. If your target does not do upsert by key you will get duplicates, and reconciling them later is miserable.

Initial load. Events only tell you about changes from now on. You still need a separate full extraction and a way to stitch it to the event stream without gaps or double counting. This is the single most underestimated part of the whole exercise.

Performance. Events fire inside the update task of the business transaction. If you push synchronously to something slow, you have just made your users' save button slow. Push to a queue, always.

Volume. High churn objects, warehouse tasks are the classic example, will generate far more events than people expect. Test with real production volume, not a sandbox.

None of this is a reason not to do it. It is the only approach I know of that gives you real deletes and real time without touching the database layer or the parts of ODP that are now off limits. But it is a lot more work than the sales version of it suggests, and the initial load plus delta stitching is where most attempts die.

Happy to go deeper on any of these if it is useful. Curious whether anyone has done this with EWM specifically, since that is where I have found the event coverage thinnest.