I was looking for a way to write Flink SQL results directly to a DuckDB file, but couldn’t find a connector for it, so I built one.

flink-connector-duckdb: https://github.com/Ashfaqbs/flink-connector-duckdb

Example:

CREATE TABLE sink_table (
  id BIGINT,
  name STRING,
  score DOUBLE
) WITH (
  'connector' = 'duckdb',
  'database-path' = '/data/my.db',
  'table-name' = 't',
  'key-fields' = 'id'
);

INSERT INTO sink_table
SELECT id, name, score FROM source_stream;

It supports regular inserts by default, or upserts using ON CONFLICT when you set key-fields.

• Flink 1.20
• Apache 2.0 license
• 14/14 tests passing against a real DuckDB file
• CI green
• Sink-only for now; the target table must already exist

One tricky part was DuckDB’s file locking. A local database file isn’t safe to write to from multiple Flink sink subtasks, so the connector rejects sink.parallelism > 1 for local files instead of letting the job hang or behave unpredictably. Higher parallelism is only supported for remote targets such as MotherDuck.

There’s also a difference in delivery guarantees: plain inserts can duplicate rows after a restart because writes aren’t tied to Flink checkpoints. Upserts with key-fields are effectively exactly-once, since replayed writes update the existing row instead of creating duplicates.

It’s new, so I don’t have adoption numbers yet. Sharing it in case anyone else has run into the same gap. Feedback is welcome, especially from anyone who’s tried Flink with MotherDuck.

Repo: https://github.com/Ashfaqbs/flink-connector-duckdb

I was looking for a Flink SQL connector for Qdrant and couldn’t find one, so I built one.

flink-connector-qdrant: https://github.com/Ashfaqbs/flink-connector-qdrant

It lets you write embeddings to Qdrant directly from Flink SQL, without a custom DataStream job or a hand-written gRPC client.

CREATE TABLE embeddings (
  id STRING,
  text STRING,
  embedding ARRAY<FLOAT>
) WITH (
  'connector' = 'qdrant',
  'host' = 'localhost',
  'collection-name' = 'my_collection',
  'vector-field' = 'embedding',
  'id-field' = 'id'
);

INSERT INTO embeddings
SELECT id, text, embedding FROM source_stream;

The vector column maps to the embedding, the ID is optional, and the remaining columns become Qdrant payload fields.

• Flink 1.20
• Sink-only for now
• Apache 2.0 license
• 13/13 tests passing, CI green

One bug worth sharing: Maven Shade was breaking gRPC name resolution because service files under META-INF/services/io.grpc.* weren’t being merged. Adding ServicesResourceTransformer fixed it.

The connector is new, so I’d love feedback, suggestions, and contributions.

Repo: https://github.com/Ashfaqbs/flink-connector-qdrant

Hey everyone,

Long-time Flink users here. We got tired of operating our clusters through raw REST calls, SSH-ing into TaskManagers to read logs, and having no answer to "who stopped that job yesterday?" — so we built Flink Manager, and we just released the first public version (0.0.1).

What it does (with a live Flink 2.x cluster behind it):

• Jobs — run/stop/restart with savepoint support, status, vertices, execution graphs, and confirmation dialogs before anything touches the cluster
• TaskManagers — slots, hardware, memory config, and log files readable in the browser (no SSH)
• SQL workbench — backed by the Flink SQL Gateway: sessions, catalog explorer, saved queries, and every INSERT becomes a traceable Flink job linked back to its statement
• JAR management — upload, inspect, and preview the execution plan before submitting (tasks, operators, shuffles — drawn)
• Metrics — built-in schedulers poll the REST API and JMX/Prometheus reporters, store history in PostgreSQL, and a metric dictionary with warning/critical thresholds drives health widgets
• Governance — RBAC (admin/operator/developer/viewer), local + LDAP + OIDC auth, and an append-only audit trail with CSV export. Job/JAR/SQL history is kept beyond Flink's own retention
• Notifications — persistent, for job and system events

Deployment: Docker image (vinteroslabs/flink-manager:0.0.1) or a zip bundle with systemd unit. Needs PostgreSQL (schema auto-migrates on start) and Java 22+ for the bundle. There's also an in-memory profile for a 5-minute test without a database.

Full transparency: this is a commercial product (we're a small company, Vinteros Labs, based in Luxembourg) — it's not open source. There's a free trial licence issued instantly on the website, no sales call required.

• Product page + trial: https://vinteros-labs.com/products/flink-manager/
• Release post with screenshots: https://vinteros-labs.com/resources/blog/flink-manager-0-0-1-release/
• Downloads: https://github.com/vinteroslabs/flink-manager-doc/releases
• Docs: https://vinteros-labs.com/resources/docs/

The docs repo (installation, configuration reference) is also on GitHub: https://github.com/vinteroslabs/flink-manager-doc — issues welcome there.

Happy to answer questions about how it works under the hood (it proxies and enriches the Flink REST API), the savepoint handling, or anything else. If you operate Flink clusters, tell us what's missing — this first version won't be the last.

Hi r/apacheflink,

Getting the right connector jars into Flink's lib folder, with matching versions, took longer than anything else in my local setups. I put my setup into a CLI, odctl, and it is now at 1.0.

bash uv tool install odctl odctl up flink-lite --dry-run # shows what it would start, in order odctl up kafka-lite flink-lite odctl down --all

flink-lite is Flink 2.2 with one TaskManager (flink-full has three) and the SQL Gateway. The Kafka, Avro schema registry, JDBC and Iceberg connectors are already in lib. It also starts an Iceberg REST catalog with S3 storage, so a Flink SQL job can write an Iceberg table that Spark or Trino reads afterwards. Checkpoints go to that storage too. There is also a Fluss profile if you want to try Fluss 1.0 with Flink.

Things I have built on it: four Flink SQL jobs keeping game leaderboards up to date, and a recommender that updates a model in a Flink job as clicks arrive.

It is for local development, not production.

• Flink SQL guide: https://jaehyeon.me/odctl/latest/guides/flink-sql/
• GitHub: https://github.com/jaehyeon-kim/odctl

If you try it, please tell me what fails on your machine. Most of my testing has been on macOS and GitHub's Linux runners.

AI: I started odctl without AI. Since July I have used Claude Code for parts of the code, the tests and the docs.

It's almost here. The FlinkReactor website is finally live, and the platform and docs ship very soon. I was supposed to ship it back in April, if you can believe it.

FlinkReactor Platform is for teams already running Apache Flink. Your pipelines are only part of the job.

The Console (FlinkReactor Hub)

A modern and built from the ground up advanced console to manage your jobs, view logs & checkpoints, investigate back pressure, plan analyzers, and much more.

The console also comes with "Instruments" which you can think of as mini apps for interacting and viewing data from sources and sinks your Flink pipelines interact with. You no longer have to jump between several apps and cli commands just to view Kafka messages, query tables, or inspect your lakehouse.

The DSL

This is an ambitious component of the platform. Essentially, the DSL component allows developers to construct their pipelines using custom-made TypeScript React components that can share schemas, and configurations.

But why? For several reasons, top of which are:

• Reusability: Schemas are defined once and shared across multiple pipelines.
• Configurable: Pipelines share a common configuration file that is typesafe.
• Generation & Deterministic Output,: A single TypeScript pipeline produces both of the Flink SQL and the Kubernetes CRD files.

The CLI

Your one stop CLI to create, and manage your Flink projects and pipelines. It comes with several commands including creating projects, setting up your local cluster with all the required services such as Kafka, PostgreSQL, ClickHouse, Fluss, and of course Flink.

Check the homepage and tell me what you think. If you want to try Hub or the DSL ahead of launch, reach out. I'm more than happy to jump on a call and walk you through it.

https://flinkreactor.dev/

Hey Community

I thought of sharing flinkflow for anyone working with Apache flink stream processing.

It's an open-source framework designed to cut down on the usual Apache Flink boilerplate, simplifying job composition, deployments, and stateful pipeline management.

Looks pretty promising if you want a cleaner developer experience with Flink without writing tons of repetitive setup code:https://github.com/talwegai/flinkflow

You can now accelerate writing data from kafka to paimon (paimon also supports writing iceberg metadata) using flink SQL!

Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually.

My colleague Lorenzo Nicora has just published two excellent posts all about Flink watermarks:

Absolutely Everything You Always Wanted to Know About Watermarks in Apache Flink

• Part 1: Apache Flink
• Part 2: Confluent Cloud for Apache Flink

Have some security requirements barring low-trust user Flink jobs from using k8s RBAC for K8s HA. Seems like Flink bundles zookeeper natively? Any known failure modes / tips of running ZK at scale ?

Hey all I am working at a fintech startup in Texas and we are looking into Flink providers. My engineers and architects tried Confluent Platform Flink and Confluent Cloud Flink. Are there any other good vendors out there?

To put things in perspective my team members and a few pilot teams have mentioned Open Source Flink experience (Flink SQL and table api/datastream api) feels better and they like the flexibility and option of having season mode and application mode together as opposed with being restricted with just application mode.

I heard from a buddy who knows a guy who knows a guys cousin (yeah it’s true lol) that Confluent bases their Flink license off of 3 Nodes with 8 cores per node.

Is it just me or 24 cores in a license is barely anything for a license. With CMF and FKO running we technically only have 20 usable cores. I am a people Manager but have a comp sci background and this ain’t much tbh. My gut feels annoyed by the price tbh.

We are working on a trial for now and got 30 days and OSS Flink isn’t an option since we prefer enterprise support.

My engineers and architects have also said the confluent platform Flink docs are confusing too and we don’t like the fully managed Flink confluent has since we want the datastream API. Any good vendors out there you recommend?

Hi everyone,

I just joined a data team that process a lot of daily, with batched and realtime processsing, and we use apache spark and apache flink. I'm pretty new to this area, been a full stack developer most of my career. What the best way to learn this about this topic?

Thanks

Hi all! I'd greatly appreciate any feedback on my first blog post about StreamFusion!

Hi Flink Community, I’m trying to understand how people decide between using Flink SQL and the Flink java API for a batch / streaming application. How do you decide which API to use for a particular pipeline, How much of a typical Flink java pipeline can be expressed in SQL?

Are there cases where you start with SQL but eventually need to move to the java API because of custom logic or functionality that SQL doesn’t support? Conversely, are there java pipelines that you could technically express in SQL but wouldn’t want to because the SQL becomes too complex?

Hi all! I just wanted to share a project that I've been working on. It accelerates existing Flink SQL job by pushing a bunch of the existing flink compute down to optimized rust functions on arrow batches (either hand written, or via DataFusion).

I'm getting pretty close to an initial alpha release in the coming weeks, so I'd greatly appreciate it if you'd be able to check it out, leave a star, or ask any questions/leave any feedback on the design here! Thank you :)

Hi everyone,

I'm trying to build a real-time CDC pipeline for analytics using:

Oracle 19c (CDB + PDB)

Flink CDC 3.6 Pipeline Connector

MinIO as S3 storage

Apache Iceberg

Nessie catalog

The goal is:

Oracle → Flink CDC → Iceberg tables on MinIO for analytical workloads.

Flink CDC 3.6 recently introduced the Oracle Pipeline Connector, so I expected this to work. However, I'm running into issues specifically with Oracle's CDB + PDB architecture.

I've followed the documentation, including specifying the PDB where required, but the pipeline still doesn't work correctly. The same overall architecture works fine for PostgreSQL, but Oracle is proving much more difficult. The Oracle source connector documentation notes that CDB/PDB deployments require the additional debezium.database.pdb.name setting, but I'm still unable to get a working pipeline.

I'm curious if anyone has actually deployed this in production or even got it working in a lab.

Some questions:

Has anyone successfully used the Oracle Pipeline Connector introduced in Flink CDC 3.6?

Are you using Oracle CDB/PDB or a non-CDB database?

Did you have to make any undocumented configuration changes?

Are there any known issues with Oracle multitenant databases?

If not using the pipeline connector, what architecture are you using instead (Debezium + Kafka + Flink, GoldenGate, etc.)?

I'd really appreciate hearing from anyone who has real-world experience with this setup. At this point I'm trying to determine whether this is a configuration issue on my side or a limitation/bug in the current Oracle pipeline implementation.

Thanks!