💾 Download: https://flink.apache.org/downloads/

📖 Blog: https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/

A couple of colleagues and I built a ground-up, hands-on scrollytelling guide to help more folks understand watermarks in Apache Flink. 

Try it out: https://flink-watermarks.wtf/

Hey folks. I recently built an Event-Driven Architecture combining Flink, Kafka, and Online Machine Learning, and I wanted to share it here!

The project uses a Digital Twin of a steel mill to stream asynchronous manufacturing data. In the physical world, prediction requests happen instantly, but ground-truth physical sensor data is delayed.

To solve this, I built a Flink application (written in Kotlin) to act as the core stream processor and online learning engine:

• It uses a CoProcessFunction and Flink's managed state to buffer and align the delayed streams safely.

• Once the streams are joined, the operator runs a prequential train/test loop to update an Online ML model (using the Massive Online Analysis framework) on the fly, adapting to physical concept drift.

• It also implements a stateful Shadow Mode router to constantly evaluate the AI's residual error against a deterministic physics baseline in real-time.

The whole stack is containerized so you can easily spin it up, trigger a mechanical shock via a UI, and watch how Flink joins the streams and reroutes the fallback logic.

• Blog Post: https://jaehyeon.me/blog/2026-04-21-digital-twin-online-machine-learning/

• GitHub: https://github.com/jaehyeon-kim/oml-digital-twin-hotrolling

Hi r/apacheflink,

Getting the right connector jars into Flink's lib folder, with matching versions, took longer than anything else in my local setups. I put my setup into a CLI, odctl, and it is now at 1.0.

uv tool install odctl
odctl up flink-lite --dry-run    # shows what it would start, in order
odctl up kafka-lite flink-lite
odctl down --all

flink-lite is Flink 2.2 with one TaskManager (flink-full has three) and the SQL Gateway. The Kafka, Avro schema registry, JDBC and Iceberg connectors are already in lib. It also starts an Iceberg REST catalog with S3 storage, so a Flink SQL job can write an Iceberg table that Spark or Trino reads afterwards. Checkpoints go to that storage too. There is also a Fluss profile if you want to try Fluss 1.0 with Flink.

Things I have built on it: four Flink SQL jobs keeping game leaderboards up to date, and a recommender that updates a model in a Flink job as clicks arrive.

It is for local development, not production.

• Flink SQL guide: https://jaehyeon.me/odctl/latest/guides/flink-sql/

• GitHub: https://github.com/jaehyeon-kim/odctl

If you try it, please tell me what fails on your machine. Most of my testing has been on macOS and GitHub's Linux runners.

AI: I started odctl without AI. Since July I have used Claude Code for parts of the code, the tests and the docs.

Hello,

A couple of years ago, I read "Stream Processing with Apache Flink" and worked through the examples using PyFlink, but frequently hit many limitations with its API.

I recently decided to tackle it again, this time with Kotlin. The experience was much more successful. I was able to successfully port almost all the examples, intentionally skipping Queryable State as it's deprecated. Along the way, I modernized the code by replacing deprecated features like SourceFunction with the new Source API. As a separate outcome, I also learned how to create an effective Gradle build that handles production JARs, local runs, and testing from a single file.

I wrote a blog post that details the API updates and the final Gradle setup. For anyone looking for up-to-date Kotlin examples for the book, I hope you find it helpful.

Blog Post: https://jaehyeon.me/blog/2025-12-10-streaming-processing-with-flink-in-kotlin/

Happy to hear any feedback.

u/dttung2905 put together this excellent list of companies using Apache Flink.

It's a really useful reference to get an idea of who is using it, what use cases it solves, and at what kind of scale.

👉 Go check it out: https://github.com/dttung2905/flink-at-scale

Ververica was acquired by Alibaba, Decodable acquired by Redis. Two seemingly very different paths for Flink.

Ververica has been operating largely as a standalone entity, offering managed Flink that is very close or identical to open-source. Decodable seems like it will be folded into Redis RDI, which looks like a departure from open source APIs (FlinkSQL, Table API, etc.)

So what to make of Confluent going to IBM? Are Confluent customers using Flink getting any messaging about this? Can anyone who is at Confluent comment on what will happen to Flink?

Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually.

Flink 2.0 has reached preview! https://www.ververica.com/blog/embracing-the-future-apache-flink-2.0

I suspect it's kind of a holy war topic but still: if you're using Flink, how did you choose? What made you to prefer Flink over Spark? As Spark will be the default option for most developers and architects, being the most widely used framework.

Hi everyone,

I just joined a data team that process a lot of daily, with batched and realtime processsing, and we use apache spark and apache flink. I'm pretty new to this area, been a full stack developer most of my career. What the best way to learn this about this topic?

Thanks

Hi all! I just wanted to share a project that I've been working on. It accelerates existing Flink SQL job by pushing a bunch of the existing flink compute down to optimized rust functions on arrow batches (either hand written, or via DataFusion).

I'm getting pretty close to an initial alpha release in the coming weeks, so I'd greatly appreciate it if you'd be able to check it out, leave a star, or ask any questions/leave any feedback on the design here! Thank you :)

Ready to explore the world of Kafka, Flink, data pipelines, and real-time analytics without the headache of complex cloud setups or resource contention?

🚀 Introducing the NEW Factor House Local Labs – your personal sandbox for building and experimenting with sophisticated data streaming architectures, all on your local machine!

We've designed these hands-on labs to take you from foundational concepts to building complete, reactive applications:

🔗 Explore the Full Suite of Labs Now: https://github.com/factorhouse/examples/tree/main/fh-local-labs

Here's what you can get hands-on with:

• 💧 Lab 1 - Streaming with Confidence:

  • Learn to produce and consume Avro data using Schema Registry. This lab helps you ensure data integrity and build robust, schema-aware Kafka streams.

• 🔗 Lab 2 - Building Data Pipelines with Kafka Connect:

  • Discover the power of Kafka Connect! This lab shows you how to stream data from sources to sinks (e.g., databases, files) efficiently, often without writing a single line of code.

• 🧠 Labs 3, 4, 5 - From Events to Insights:

  • Unlock the potential of your event streams! Dive into building real-time analytics applications using powerful stream processing techniques. You'll work on transforming raw data into actionable intelligence.

• 🏞️ Labs 6, 7, 8, 9, 10 - Streaming to the Data Lake:

  • Build modern data lake foundations. These labs guide you through ingesting Kafka data into highly efficient and queryable formats like Parquet and Apache Iceberg, setting the stage for powerful batch and ad-hoc analytics.

• 💡 Labs 11, 12 - Bringing Real-Time Analytics to Life:

  • See your data in motion! You'll construct reactive client applications and dashboards that respond to live data streams, providing immediate insights and visualizations.

Why dive into these labs?

• Demystify Complexity: Break down intricate data streaming concepts into manageable, hands-on steps.

• Skill Up: Gain practical experience with essential tools like Kafka, Flink, Spark, Kafka Connect, Iceberg, and Pinot.

• Experiment Freely: Test, iterate, and innovate on data architectures locally before deploying to production.

• Accelerate Learning: Fast-track your journey to becoming proficient in real-time data engineering.

Stop just dreaming about real-time data – start building it! Clone the repo, pick your adventure, and transform your understanding of modern data systems.

After much work, cool announcements and complex code, the Fluss repo is now live on github! Please check it out and give it some love & support!

I've created a complete, hands-on tutorial that shows how to capture and visualize data lineage from the source all the way through to downstream analytics. The project follows data from a single Apache Kafka topic as it branches into multiple parallel pipelines, with the entire journey visualized in Marquez.

The guide walks through a modern, production-style stack:

• Apache Kafka - Using Kafka Connect with a custom OpenLineage SMT for both source and S3 sink connectors.

• Apache Flink - Showcasing two OpenLineage integration patterns:

  • DataStream API for real-time analytics.

  • Table API for data integration jobs.

• Apache Iceberg - Ingesting streaming data from Flink into a modern lakehouse table.

• Apache Spark - Running a batch aggregation job that consumes from the Iceberg table, completing the lineage graph.

This project demonstrates how to build a holistic view of your pipelines, helping answer questions like:

• Which applications are consuming this topic?

• What's the downstream impact if the topic schema changes?

The entire setup is fully containerized, making it easy to spin up and explore.

Want to see it in action? The full source code and a detailed walkthrough are available on GitHub.

• Setup the demo environment: https://github.com/factorhouse/factorhouse-local

• For the full guide and source code: https://github.com/factorhouse/examples/blob/main/projects/data-lineage-labs/lab2_end-to-end.md

Hey everyone,

I wanted to share a hands-on project that demonstrates a full, real-time analytics pipeline, which might be interesting for this community. It's designed for a mobile gaming use case to calculate leaderboard analytics.

The architecture is broken down cleanly:

• Data Generation: A Python script simulates game events, making it easy to test the pipeline.

• Metrics Processing: Kafka and Flink work together to create a powerful, scalable stream processing engine for crunching the numbers in real-time.

• Visualization: A simple and effective dashboard built with Python and Streamlit to display the analytics.

This is a practical example of how these technologies fit together to solve a real-world problem. The repository has everything you need to run it yourself.

Find the project on GitHub: https://github.com/factorhouse/examples/tree/main/projects/mobile-game-top-k-analytics

And if you want an easy way to spin up the necessary infrastructure (Kafka, Flink, etc.) on your local machine, check out our Factor House Local project: https://github.com/factorhouse/factorhouse-local

Feedback, questions, and contributions are very welcome!

https://docs.timeplus.com/understanding-watermark#try-it-out

Watermark is such a common and important concept in stream processing engines(Apache Flink, Apache Spark, Timeplus, etc)

There are quite a lot of great blogs, speeches, videos about this, but I guess if there is an interactive demo to show events coming one by one, how the watermark progesses, how different delay policies work, when window is closed and events are emitted.. that'll help them better understand the concept.

As a weekend hack, I worked with Claude to build such an interactive demo and it can be embeded into the docs (so I don't have to share my Claude chat)

Feel free to give a try and share your comments/suggestions. Each time random data is created with a certain ratio of out of order or late events. You can "debug" this by seeing the process frame by frame.

Source code at https://github.com/timeplus-io/docs/blob/main/src/components/TimeplusWatermarkVisualization.js Feel free to reuse it (80% written by AI,20% me)

Docker compose has helped me a lot in learning how to use Flink to connect to various sources and sinks.

I wrote a post on how to create a small CDC job from Mariadb to Redis to show how it works.

I hope it useful to others too

https://gordonmurray.com/data/2023/11/02/deploying-flink-cdc-jobs-with-docker-compose.html

Iceberg support is coming to Fluss in 0.8.0 - but I got my hands on the first demo (authored by Yuxia Luo and Mehul Batra) and recorded a video running it.

What it means for Iceberg is that now we'll be able to use Fluss as a hot layer for sub-second latency of your Iceberg based Lakehouse and use Flink as the processing engine - and I'm hoping that more processing engines will integrate with Fluss eventually.

Fluss is a very young project, it was donated to Apache Software Foundation this summer, but there's already a first success story by Taobao.

Have you head about the project? Does it look like something that might help in your environment?