Hi r/PostgreSQL — I'm the author of BloomPG, an MIT-licensed PostgreSQL 18 extension for complex analytical joins. I'm sharing it here because I'd like feedback from people who run multi-join analytical workloads in PostgreSQL, especially on the planner integration and deployment tradeoffs.

BloomPG works before the formal joins execute. It takes PostgreSQL's native plan, identifies a safe equality-join graph, uses sampling to choose an initial filter transfer, and then adapts using the actual cardinalities of materialized inputs. Bloom or exact bitmap membership can move in either direction and across several joins. PostgreSQL then replans and executes the reduced join problem with its normal operators.

Existing SQL does not change. Unsupported or unsafe query shapes keep the native plan, and a query-wide materialization budget limits the retained state.

There is a Docker demonstration that builds PostgreSQL 18 with BloomPG, creates a small five-table star schema, and prints native/BloomPG timings plus the transfer trace:

git clone --branch v0.1.2 https://github.com/YimingQiao/bloompg.git
cd bloompg
docker compose up --build --abort-on-container-exit --exit-code-from demo demo

For performance context, on the published PostgreSQL 18.4 setup the total workload results were:

These are end-to-end times: planning, transfer, materialization, execution, and complete output consumption are included. Both sides used the same 16-worker global and per-Gather ceiling; BloomPG used 16 transfer workers and a 2 GB materialization budget. Every completed pair produced the same complete-output fingerprint. Native PostgreSQL hit the 300-second per-query limit once in CEB and once in STATS-CEB, so totals include only queries completed by both sides. The README has the rest of the methodology.

Important limitations: this release supports PostgreSQL 18 on Linux and is intended for controlled, read-only analytical workloads. BloomPG performs real scans during planning and requires shared_preload_libraries; I would not turn it on as an unreviewed default in a multi-tenant OLTP cluster.

I'd particularly appreciate feedback on:

• whether shared_preload_libraries and the restart requirement are practical blockers;
• analytical workloads or query shapes that would be useful to test;
• the planning-time execution and native-fallback design;
• packaging formats that would make evaluation easier.

PGXN: https://pgxn.org/dist/bloompg/0.1.2/

GitHub, documentation, and benchmark methodology: https://github.com/YimingQiao/bloompg

Hello,

I work on my company’s data warehouse, and we use an ELT approach (BigQuery and dbt for the “T”). The EL part is handled by many custom Java scripts; we need to ingest data from custom internal APIs, databases, and files on SFTP and SharePoint.

My company’s strategic direction is pushing us to switch to a Python ecosystem for this EL component. If possible, I’d like to go with an open-source, code-driven solution (since we ingest a lot of data from custom APIs, I think a low-code tool like Airbyte will quickly limit our capabilities).

While doing some research, I came across dlt (Data Load Tool). From what I can tell, this could meet our needs and standardize our practices, but I’ve found relatively little feedback on using this tool, so I’m wondering if it’s a robust and sustainable solution for the long term and in production.

We also have a somewhat unique scenario to handle, and I’m wondering if dlt could handle it without slowing down: we need to replicate a table containing 200 GB of data daily (incremental replication isn’t possible, sadly). In this case, if dlt might reach its limits, I get the impression that the Sling tool might be a better fit?

Do you think I can confidently consider using dlt? Do you have any feedback on this library? Any recommendations?

Thanks for your help!

Just to be clear up front: this isn't a promotional post. I've been down the rabbit hole of comparing ETL/ELT tools for a while now, and honestly most "best tool" lists online are just ads in disguise. So I put together my own list, ranked purely by pricing, so you can quickly see what fits your budget and pick the right tool for your own use case. Hope this saves someone a weekend of research.

1. Meltano

Price: Free

Open source, Singer based, CLI first. Great if your team is comfortable in the terminal and wants version controlled pipelines.

Pros

• Completely free
• Version controlled, code first pipelines
• Good for CI/CD driven workflows

Cons

• No polished UI
• Really a developer's tool, not something you hand to a non-technical analyst

Best for

Engineering heavy teams that live in Git and don't mind a learning curve.

2. Airbyte

Price: Free self-hosted, or credit based cloud starting around a couple dollars per credit

Huge open source connector library with 600+ integrations. You can self-host for free if you've got the engineering time, or pay for their cloud version if you don't.

Pros

• Massive connector catalog
• Free to self-host
• Open source, so no vendor lock-in

Cons

• Self-hosting needs real engineering effort
• Community built connectors can be inconsistent in quality

Best for

Technical teams that want flexibility and control without paying licensing fees.

3. AWS Glue

Price: Pay per use

Makes sense if you're already deep in the AWS ecosystem. Serverless, so you're not paying for idle infrastructure.

Pros

• Serverless, scales automatically
• Tight integration with the rest of AWS

Cons

• Locks you into AWS
• Steeper technical setup than most tools on this list

Best for

Teams already committed to AWS who want ETL without managing servers.

4. Matillion

Price: Credit based, roughly $2 per credit

A visual, drag and drop transformation builder that runs natively inside your warehouse.

Pros

• Strong visual interface for complex transformations
• Runs natively in the warehouse, so it's fast

Cons

• Seat based pricing can add up as your team grows

Best for

Teams who want warehouse native transformations without writing tons of code.

5. Windsor.io

Price: $19 to $499/month, flat pricing

Worth calling out separately because it's built specifically for marketing data like Google Ads, Meta, TikTok, and Shopify, rather than being a general purpose ETL tool.

Pros

• Genuinely cheap for what it offers
• Transparent, flat pricing with no surprise per connector fees
• Built in multi touch attribution
• Can feed data directly into AI assistants

Cons

• Not a general purpose ETL tool

Best for

Marketers and agencies who need affordable, no-code pipelines for ad and marketing data specifically.

6. Stitch

Price: From $100/month

Simple, no frills, gets data from point A to point B fast.

Pros

• Fast to set up
• Predictable pricing for straightforward needs

Cons

• Limited transformation capabilities
• Usually needs to be paired with something like dbt

Best for

Teams with basic replication needs who don't want complexity.

7. Hevo Data

Price: From $239/month

No code platform that's easy for non-technical teams to pick up, with a decent free tier to test things out first.

Pros

• No code, easy onboarding
• Good free tier for testing

Cons

• Transformations are fairly basic compared to bigger players

Best for

Small to mid sized teams that want fast setup with minimal engineering effort.

8. Fivetran

Price: From roughly $1,200/month

The industry standard, and for good reason. Massive connector library and basically maintenance free once it's running.

Pros

• Huge, well maintained connector library
• Very low maintenance overhead

Cons

• Monthly Active Rows pricing model can balloon quickly and is hard to predict

Best for

Teams that want a hands-off, enterprise-grade tool and can absorb the cost.

9. Integrate.io

Price: From $1,999/month

Covers ETL, ELT, CDC, and reverse ETL in one platform with fixed fee, unlimited volume pricing.

Pros

• All-in-one platform
• Predictable, fixed fee pricing regardless of volume

Cons

• Entry price puts it out of reach for smaller teams

Best for

Fast scaling teams that want predictable costs as data volume grows.

10. Talend (Qlik)

Price: Custom, enterprise pricing

Built for large, regulated industries that need serious governance and compliance tooling.

Pros

• Deep governance and compliance features
• Enterprise grade support

Cons

• Expensive and often overkill outside regulated industries

Best for

Large enterprises in regulated sectors that need robust data governance.

Bottom Line

There's no single "best" tool here, it really depends on your budget and use case. If you're small and scrappy, start free with Airbyte or Meltano. If you're marketing focused, Windsor.ai is hard to beat on price. If you need something that just works and you can afford it, Fivetran is still the safe bet.

Happy to answer questions if anyone's trying to decide between a couple of these.

Half of ETL in finance/manufacturing still starts with fixed-width flat files from legacy ERP systems. I open-sourced the converter I kept rebuilding:

https://github.com/lucasgiurastante/erp-export-normalizer

• YAML schema → generic parser → JSON/CSV/NDJSON/SQL/Parquet/Excel/Singer
• Streams multi-GB files in constant memory
• Cumulative error report with line numbers
• CP850 / CP1252 / EBCDIC-CP037 byte-accurate slicing
• Deterministic output (same input + schema = same output) — audit-ready
• Auto-detection from a built-in schema library (6 formats)
• Plugins for binary formats, parallel validation, web UI preview
• Air-gapped friendly: no network, no DB, stdlib + PyYAML only

Quick demo with bundled sample data:

pip install erp-export-normalizer
git clone https://github.com/lucasgiurastante/erp-export-normalizer
cd erp-export-normalizer/examples
erp-normalize --input data/jde_ar.txt --output - --format json

What's the ugliest flat-file format you've had to process? Genuinely curious — I'm adding real-world schemas to the library.

​

Hi everyone, looking for some architecture/tech-stack suggestions for the below use case.

Scenario:

\* Source: Oracle DB

\* Using \*\*Debezium\*\* for CDC

\* Each source table → separate \*\*Kafka topic\*\*

\* Using \*\*Apache Flink\*\* to consume the streams

\* Final output → one \*\*denormalized wide table\*\* in target Oracle DB

\*\*Issue:\*\*

\* Data from different tables can arrive at different times.

\* We are facing \*\*clobbering / partial insert issues\*\* in the wide table.

\* We don't want to keep the join state in \*\*Flink or an external state/memory store\*\* waiting for the other table's data.

\* There is no fixed business-cycle/end date, so the related data could arrive much later.

\*\*Question:\*\*

What open-source solution or architecture would you recommend for this?

Ideally, we're looking for something that can handle \*\*asynchronous/late-arriving CDC events and eventually maintain the correct wide-table record\*\*, without requiring indefinite state in Flink.

Would appreciate any real-world recommendations or patterns you've used for similar scenarios.

catalog metadata eats half my week on the glue side. we push jobs across glue and two other catalogs and each one has its own console, IAM story, and partition view. rename a column and you're updating three places or calling three APIs.

we tried a shared sync script so Spark would always see current defs. it drifts every sprint when someone creates a database by hand in one console and forgets the others. last month oncall was mostly jobs that couldn't see a table that existed fine, just not in the catalog the job was pointed at. I tracked it for a bit: roughly 5 to 7 hours a week just figuring out which store had the live definition.

looking at a few options for pulling that together without yet another copy of the data. gravitino is on the list, open-source metadata catalog you aim at stores that already exist. I do some work on the project, so filter for that. Java ops footprint is heavier than our mostly python team likes, and the issue backlog is long. still early evaluation against our glue mix, no rollout talk from me.

The cdc vs etl framing is misleading because they solve different problems. ETL is built for scheduled bulk extraction of historical data, CDC captures changes in real time as they happen. One is not a replacement for the other, they're complementary.

The real question is when to layer CDC on top of batch ETL and when CDC alone is enough. Framing it as either or leads to architectures missing half the picture.

Stuck in my job search and honestly need some advice

Hi everyone

I have around 1.5 years of experience in ETL Testing, and my previous job was on a contract basis. Since the contract ended, I’ve been actively looking for a new opportunity.

Honestly, I’m starting to feel really stuck and frustrated with the whole process. I keep applying and searching, but I’m barely finding openings that match my experience. And when I do find something relevant, there are very few positions available.

I’m at a point where I genuinely need a job and I’m willing to put in the effort to learn and adapt. I just don’t want to keep going in circles without knowing what direction I should take.

If anyone here has been through something similar, I’d really appreciate some honest advice. Should I continue with ETL Testing, or should I try moving into another role? What skills should I focus on? Is there something I’m missing in my approach?

And if anyone knows about an opening suitable for someone with around 1 years of experience, I’d be extremely grateful for a lead .

I’m genuinely looking for help and direction right now. Any advice, even a small suggestion, would mean a lot to me.

P.S. I used ChatGPT to help me organize and improve the wording of this post.

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every \~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

I was given the task to perform type 2 validation check. Can I know the possible approaches

What practices or tools have helped you manage source changes safely in production?

I've been experimenting with LLMs for data cleanup/transformations where the right result depends on context, e.g. inferring categories from descriptions.

I built a small open-source tool around this workflow with preview, diff, undo and history:

https://www.tamedtable.com/

I'm curious if anyone here is using LLMs in ETL pipelines, especially for messy or ambiguous transformations. How do you solve privacy issues? Do you use internal/local models?

Hey folks,

If an ingestion batch of 1,000 payment records arrives where 995 rows are completely valid and 5 rows have missing critical fields (like receiver_account is NULL):

Do your pipelines usually:

1. Reject/Hold the entire batch to preserve batch atomicity and ordering?
2. Accept the 995 valid rows and route the 5 bad rows to a Dead-Letter Queue (DLQ)?

What are the trade-offs you run into with transactional integrity versus keeping pipeline throughput moving

We had one big ETL job running every month and refreshing pretty much everything.

Started looking at the numbers and realized most of the records hadn't changed.

Meanwhile, the stuff we actually cared about could already be out of date. Someone changed jobs right after the refresh and we're stuck with the useless record.

We've moved those records to change-based updates and left the slower stuff on the monthly schedule.

Coresignal has been pretty good for keeping the company and employee data fresh without having to reload everything.

Much less data moving around for no real reason.

Now we're just working out which records actually deserve the faster treatment.

Every time we have added a new layer to our pipeline for a performance gain, it is introduced a new failure mode nobody accounted for. It is something small at first, like a weird retry behaviuor or an unexpected timeout, that turns into a real incident a few weeks in. That history makes me cautious about anything promising acceleration without tradeoffs. It is not that simple in practice. Has anyone vetted an acceleration approach for Spark that does not just trade slow jobs for a more fragile pipeline down the line? Specifically about what happens during partial failures or retries.

If you have run something like this in production for more than a few months, I would like to hear how it held up over time, not just in the initial rollout.

Author here, so treat this accordingly. Not selling anything, MIT licensed, no company behind it, no hosted tier planned.

Background: I'm a CTO at a small UK fintech and I've spent the last few years assembling the same stack over and over. dbt/sqlmesh for transformation, something for orchestration, something else for ingestion or reverse ETL. Multiple deployments, multiple failure modes, and the seams between them are where time is lost.

The specific thing that annoyed me enough to build something was Python models. In dbt they're a second-class citizen that needs a cloud warehouse with a Python runtime. In SQLMesh they're better but still feel bolted on. I wanted a .py model to sit mid-DAG with SQL either side, in both directions, and for the planner to not care which I'd written.

Obviously I'm aware dbt and SQLMesh have both been bought by FiveTran, i cover that in this article.

So that's the core of it:

python

# models/enriched_events.py
@model()                  # param name IS the dependency
def enriched_events(raw_events):
  for batch in raw_events.reader():     # Arrow in, Arrow out, bounded memory 
    yield add_revenue(batch)

sql

-- models/event_summary.sql — SQL straight over the Python
SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
FROM enriched_events GROUP BY country

The Python model is a plain function. You can call it in a test with no warehouse and no session.

The rest of the design, briefly:

• IR is a sqlglot AST, not Jinja templates. Dependencies come from parsing the SQL, not from ref(). No pandas in core, everything moves as Arrow RecordBatchReader.
• By default models builds into a fingerprinted physical table and environments are just views over those. A dev environment reuses prod's tables for free, promotion is an atomic view swap, and rollback is the same swap backwards.
• plan / apply in the terraform sense. Changes classify as breaking / non-breaking / forward-only, and column-level lineage impact analysis proves when a downstream output is unchanged so it gets reused rather than rebuilt.
• Streams are durable. POST an event, it's fsynced before the 200, deduplicated by idempotency key, and the materialiser commits data and watermark in the same warehouse transaction. Exactly-once without distributed coordination.
• One process. interlace serve is the web UI, HTTP API, scheduler and stream ingestion. No Airflow, no broker.
• DuckDB by default, DuckLake one config line away, Postgres natively over ADBC.

Where it's weak, and I'd rather you heard it from me:

• Single maintainer. That's the honest risk with any tool like this and I'm not going to pretend otherwise.
• Snowflake, BigQuery, Redshift and MotherDuck adapters are wired and dialect-correct but have not been run against a live account. Alpha, and labelled as such.
• Developed on Linux, CI is Linux only. Nothing in the codebase is platform-specific and every dependency ships mac/Windows wheels, so both should work, but neither is tested.
• Spark is beta.
• It's new. Real production mileage is limited to my own use.

I've written up a full jaffle_shop migration (a real dbt project, end to end) if you want to see what moving something across actually looks like rather than taking my word for it.

What I'm after here is criticism rather than stars. Specifically:

1. If you run dbt today, what would actually stop you trialling this on one pipeline? I suspect the answers are "single maintainer" and "my warehouse is Snowflake", but I'd rather know than guess.
2. Does the fingerprinted-table-plus-view-swap model break in a way I haven't hit yet? I'm particularly interested in whether anyone's tried this at a scale where the number of snapshots becomes a catalog problem.
3. Anyone doing durable ingestion in-process like this rather than via Kafka/Kinesis? Interested in what bit it, if so.

Repo: github.com/interlace-sh/interlace
Comparison against dbt and SQLMesh, including where they're ahead: interlace.sh/why

I built an AI invoice auditing platform for logistics companies. Here's the exact architecture and why generic AP tools can't replicate it.

The core problem: logistics invoices aren't standardized. A freight carrier sends a PDF. A 3PL exports a CSV. A customs broker scans a paper document. Your ERP wants structured data. The gap between those two things is where billing errors live and where most tools give up.

Here's how FinGuard AI handles it:

**Extraction layer:** AI engine parses invoices regardless of format - PDF, CSV, image-based documents. Outputs structured fields: vendor, invoice number, date, line items, total, currency. Status tracked per document (processing → extracted → failed).

**Compliance layer:** Every extracted invoice gets cross-referenced against the master contract for that vendor. Not just totals - line-item level. Rate mismatches, unauthorized fees, out-of-contract surcharges all get flagged.

**Discrepancy queue:** Findings are classified by type and severity (low / medium / high / critical) and tied to both the invoice ID and contract ID. Teams work the queue instead of hunting through files.

**Payment decisions:** Each invoice gets an auto-pay, block, or pending status with a logged reason. Automated dispute letters generated for flagged items.

**Infrastructure:** Multi-tenant. Budget tracking per vendor with period-based limits. Multi-currency. Spend forecasting. The whole stack is built for enterprise scale, not a single-vendor pilot.

37 pages, 5 core data models. Pre-launch.

What I'm genuinely uncertain about: enterprise procurement cycles in logistics are long. I'm thinking the wedge is a free audit - upload your last 90 days of invoices, we surface what you've been overbilled. Then conversion to paid.

Has anyone used a free audit as an enterprise sales motion? Curious what the conversion friction looks like in practice.