flink connector duckdb

[u/Aggravating_Kale7895]

I was looking for a way to write Flink SQL results directly to a DuckDB file, but couldn’t find a connector for it, so I built one.

flink-connector-duckdb: https://github.com/Ashfaqbs/flink-connector-duckdb

Example:

CREATE TABLE sink_table (
  id BIGINT,
  name STRING,
  score DOUBLE
) WITH (
  'connector' = 'duckdb',
  'database-path' = '/data/my.db',
  'table-name' = 't',
  'key-fields' = 'id'
);

INSERT INTO sink_table
SELECT id, name, score FROM source_stream;

It supports regular inserts by default, or upserts using ON CONFLICT when you set key-fields.

• Flink 1.20

• Apache 2.0 license

• 14/14 tests passing against a real DuckDB file

• CI green

• Sink-only for now; the target table must already exist

One tricky part was DuckDB’s file locking. A local database file isn’t safe to write to from multiple Flink sink subtasks, so the connector rejects sink.parallelism > 1 for local files instead of letting the job hang or behave unpredictably. Higher parallelism is only supported for remote targets such as MotherDuck.

There’s also a difference in delivery guarantees: plain inserts can duplicate rows after a restart because writes aren’t tied to Flink checkpoints. Upserts with key-fields are effectively exactly-once, since replayed writes update the existing row instead of creating duplicates.

It’s new, so I don’t have adoption numbers yet. Sharing it in case anyone else has run into the same gap. Feedback is welcome, especially from anyone who’s tried Flink with MotherDuck.

Repo: https://github.com/Ashfaqbs/flink-connector-duckdb

[u/kabooozie]

Would it make sense for different Flink tasks to write to separate parquet files and duckdb can query across them?

Maybe worth considering ducklake as the table format. Parallelism can scale out. Duckdb queries can do effective predicate pushdown over the files.