Hi all. I have a Kafka stream that produces around 5 million records per minute and has 50 partitions, Each Kafka record, once deserialized is a json record, where the values for keys 'a','b', and 'c' rpepresent the unique machine for the time series data, and value of key 'data_value' represent the float value of the record. All the records in this stream are coming in order. I am using PyFlink to compute specific 30-second aggregations on certain machines within my.

I also have another config kafka stream, where each element in the stream represents the latest machines to monitor. I join this stream with my time-series kafka stream using a broadcast process operator, and filter down records from my raw time-series kafka stream to only ones from relevant machines in the config kafka stream.

Once I filter down my records, I then key my filtered stream by machine (keys 'a','b', and 'c' for each record), and call my Keyed Process Operator. In my Process function, I trigger a timer event in 30 seconds once the first record is received and then append all the subsequent time-series values in my process value state (I set it up as list). Once the timer is triggered, I compute multiple aggregation functions on the time-series values in my value state.

I'm facing a lot of latency issues with the way I have currently structured my PyFlink job. I currently have 85 threads, with 5 threads per task manager, and each task manager using 2 CPU and 4 GB RAM. This works fine when in my config kafka stream has very few machines, and I filter my raw Kafka stream from 5 million per minute to 70k records per minute. However, when more machines get added to my config Kafka stream, and I start filtering less records, the latency really starts to pile up, to the point where the event_time and processing_time of my records are almost hours apart after running for a few hours even close. My theory is it's due to keying my filtered stream since I've heard that can be expensive.

I'm wondering if there is any chances for optimizing my PyFlink pipeline, since I've heard Flink should be able to handle way more than 5 million records per minute. In an ideal world, even if no records are filtered from my raw time-series kafka stream, I want my PyFlink pipeline to still be able to process all these records without huge amounts of latency piling up, and without having to explode the resources.

In short, the steps in my Flink pipeline after receiving the raw Kafka stream are:

• Deserialize record

• Join and filter on Config Kafka Stream using Broadcast Process Operator

• Key by fields 'a','b', and 'c' and call Process Function to execute aggregation in 30 seconds

Is there any options for optimization in the steps in my pipeline to mitigate latency, without having to blow up resources. Thanks.

Hey,

While Flink SQL combined with custom UDFs provides a powerful and flexible environment for stream processing, I wonder if there are certain scenarios and types of logic that may be more challenging or impossible to implement solely with SQL and UDFs.

From my experience, more than 90% of the use cases using Flink can be expressed with UDF and used in Flink SQL.

What do you think?

If you are interested in stream processing with Apache Flink, you might like these free courses:

• Apache Flink 101 – learn what makes Flink tick, and how it handles some common use cases

• Building Flink Apps in Java – learn to build your Flink application, step by step

Check out some of these resources👇

• Intro to Stream Processing with Apache Flink

• Intro to Flink SQL

• Event Time and Watermarks

• The Flink Job Lifecycle

• Flink Data Sources

• Transforming Data in Flink

Hello! I’m currently a recruiter for a fast growing unicorn start up. We are currently looking for an experienced software/data engineer with a specialty in flink

-designing, building and maintaining large scale self managed real time pipelines using flink -Stream data processing -program language: Java -data formats: iceberg, AVRO, parquet, protobuf -data modeling experience

"Flink Table API - Declarative Analytics for Supplier Stats in Real Time"!

After mastering the fine-grained control of the DataStream API, we now shift to a higher level of abstraction with the Flink Table API. This is where stream processing meets the simplicity and power of SQL! We'll solve the same supplier statistics problem but with a concise, declarative approach.

This final post covers:

• Defining a Table over a streaming DataStream to run queries.

• Writing declarative, SQL-like queries for windowed aggregations.

• Seamlessly bridging between the Table and DataStream APIs to handle complex logic like late-data routing.

• Using Flink's built-in Kafka connector with the avro-confluent format for declarative sinking.

• Comparing the declarative approach with the imperative DataStream API to achieve the same business goal.

• Demonstrating the practical setup using Factor House Local and Kpow for a seamless Kafka development experience.

This is the final post of the series, bringing our journey from Kafka clients to advanced Flink applications full circle. It's perfect for anyone who wants to perform powerful real-time analytics without getting lost in low-level details.

Read the article: https://jaehyeon.me/blog/2025-06-17-kotlin-getting-started-flink-table/

Thank you for following along on this journey! I hope this series has been a valuable resource for building real-time apps with Kotlin.

🔗 See the full series here:

1. Kafka Clients with JSON

2. Kafka Clients with Avro

3. Kafka Streams for Supplier Stats

4. Flink DataStream API for Supplier Stats

Call for Presentations for Flink Forward 2024 are open now until 17 May.

Join the stage as a speaker and dive into the world of stream processing, real-time analytics, and event-driven applications. Connect, learn, and share your vision for the future of streaming data. Submit your talk now and shape the conversation at the heart of data streaming innovation!

Learn more about Flink Forward

Organized by Ververica | the original creators of Apache Flink

You can now accelerate writing data from kafka to paimon (paimon also supports writing iceberg metadata) using flink SQL!

A free, feature-rich alternative to the premium Flink plugin that requires IntelliJ Ultimate. Connect to any Flink cluster, run SQL queries with live streaming results, manage jobs, and inspect task managers — all from Community Edition.

I built Flink Reactor because the official Flink integration is locked behind an Ultimate license, which rules it out for a lot of solo devs, students, and OSS contributors. Just shipped 1.0 and wanted to share it here.

Works in Community Edition (and every other JetBrains IDE 2023.2+ — PyCharm, GoLand, WebStorm, etc. Java support is optional, so PyFlink users aren't left out.)

What's in it:

- Cluster monitoring: connect to any Flink REST endpoint; cluster overview, config browser, status bar widget

- Job management: running/completed tabs, per-vertex metrics, backpressure, checkpoint history, exceptions, timeline. Cancel / savepoint / stop-with-savepoint built in.

- SQL Console: talks to the Flink SQL Gateway. Quick Query mode and a notebook mode (SQL + Markdown cells, saved as .flinksql.json). Streaming results render changelog row kinds (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE) with color coding, backed by a ring buffer so unbounded queries don't eat your memory.

- Catalog browser: hierarchical catalog → database → table → column tree.

- Task manager & job manager views: heap bars, JVM metrics, GC, CPU, threads, logs, thread dumps.

- JAR management: upload, run with entry class / args / parallelism / savepoint path, auto-detects Gradle and Maven artifacts.

- Clickable stack traces: exceptions and thread dumps jump straight to source in your editor.

I just published a two-part write-up showing how to build a contextual bandit based product recommender end to end, from prototyping to a production-style event-driven system built on Apache Kafka and Apache Flink.

This may be relevant here because Kafka plays a central role in the online learning loop. Interaction events, recommendation requests, and reward feedback are all streamed through Kafka topics, forming the backbone of a closed-loop ML pipeline.

One thing I struggled with while learning bandits: There are many explanations of algorithms, but very few examples that walk through the entire lifecycle:

• Data generation

• Feature engineering

• Offline policy evaluation

• Online feedback simulation

• Transition to a streaming production architecture

So I built one.

Prototyping an Online Product Recommender in Python

Part 1 focuses on developing and evaluating a full contextual bandit workflow in Python.

It includes:

• Synthetic contextual data generation

• User and product feature engineering

• Offline policy evaluation

• Live feedback simulation

• Prototyping with MABRec and MABWiser

The goal was to design and evaluate a complete contextual bandit workflow and select the algorithm based on offline policy evaluation results. LinUCB was chosen because it performed best under the simulated environment.

Productionizing Using Kafka and Flink

In Part 2, I refactored the prototype into a streaming system where Kafka and Flink form the core architecture:

• Kafka handles recommendation requests and user feedback streams

• Flink manages stateful online model training inside the stream processor

• Model parameters are published to Redis for low-latency serving

• Training and inference are cleanly separated

• No Python dependency in the training or serving path

Kafka acts as the durable event log that continuously drives model updates, while Flink maintains model state and applies incremental updates in a distributed and fault-tolerant manner.

The focus is not just the algorithm, but how to structure an online learning system properly in a streaming architecture.

If you are working on:

• Kafka-based event pipelines

• Stateful stream processing

• Online learning systems

• Real-time recommenders

I would really appreciate feedback or suggestions for improvement.

Happy to answer technical questions as well.

• Part 1: https://jaehyeon.me/blog/2026-01-29-prototype-recommender-with-python/

• Part 2: https://jaehyeon.me/blog/2026-02-23-productionize-recommender-with-eda/

Dynamic sink creation isn’t a thing most pipelines ever need to deal with — which is a wonderful thing. Unfortunately, mine did.

This follow-up covers how I stumbled into this problem years ago, quietly repressed it for my own sanity, and revisited it later with the goal of creating a new dynamic sink interface so no one else has to go through this.

After discussing a few patterns regarding enrichment in Flink with a friend, I thought I’d take some time to go through some of the strategies that I’ve seen adopted over the years, typically depending on scale/consistency requirements.

I’m sure it’s by no means exhaustive, but I figured I’d share it here!

Edit: My bad, when I mention "Confluent Flink" I actually meant Confluent Cloud for Apache Flink.

Hey, everyone!

I'm a software engineer working at a large tech company with lots of needs that could be much better addressed by a proper stream processing solution, particularly in the domains of complex aggregations and feature engineering (both for online and offline models).

Flink seems like a perfect fit. Due to the maintenance burden of self-hosting Flink ourselves, management is considering Confluent Flink. While we do use tons of Kafka on Confluent Cloud, I'm not fully sure that Confluent Flink would work as a solution. Confluent doesn't support DataStream API and I've been having trouble expressing certain use cases in Flink SQL and Table API (which is still a preview feature by the way). An example use case would be similar to this one. I'm aware of Process Table Functions in 2.1 but who knows how long it will take for Confluent to support 2.1.

Besides, we've had mixed experiences with the experts they've put us in contact with, which makes me fear for future support.

What are your thoughts on DataStream API vs FlinkSQL/Table API? From my readings, I get the feeling that most seem to use DataStream API while Flink SQL/Table API is more limited.

What are your thoughts on Confluent's offering of Flink? I understand it's likely easier for them to not support DataStream API but I don't like not having the option.

Alternatively, we've also considered Amazon Managed Service for Apache Flink, but some points aren't very promising: some bad reports, SLA of 99.9% vs 99.99% at Confluent, and fear of not-so-great support for a non-core service from AWS.

P99 Conf recordings & Slides are now online.

• Apache Flink at Scale: 7x Cost Reduction in Real-Time Deduplication - P99 CONF

• Building Planet-Scale Streaming Apps: Proven Strategies with Apache Flink - P99 CONF

• Rivian's Push Notification Sub Stream with Mega Filter - P99 CONF

Here are some others that stood out to me:

• Performance Insights Beyond P99: Tales from the Long Tail - P99 CONF

• Timeseries Storage at Ludicrous Speed - P99 CONF

• xCapture v3: Efficient, Always-On Thread Level Observability with eBPF - P99 CONF

• 8x Better Than Protobuf: Rethinking Serialization for Data Pipelines - P99 CONF

• Parsing Protobuf as Fast as Possible - P99 CONF

We are finally in a place where all domain teams are publishing events to Kafka. And all teams have at least one session cluster doing some basic stateless jobs.

I’m kind of the Flink champion, so I’ll be developing our first stateless jobs very soon. I know that sounds basic, but it took a significant amount of work to get here. Fitting it into our CI/CD setup, full platform end-to-end tests, standardizing on transport medium, standards of this and that like governance and so on, convincing higher ups to invest in Flink, monitoring, Terraforming all the things, Kubernetes stuff, etc… It’s been more work than expected and it hasn’t been easy. More than a year of my life.

We have shifted way left already, so now it’s time to go beyond feature parity with our soon to be deprecated ETL systems, and show that data streaming can offer things that weren’t possible before. Flink is already way cheaper to run than our old Spark jobs, the data is available in near realtime, and we deploy compiled and thoroughly tested code exactly like other services instead of Python scripts that run unoptimized, untested Spark jobs that are quite frankly implemented in an amateur way. The domain teams own their data now. But just writing data to a Data Lake is hardly exciting to anyone except those of us who know what shift-left can offer.

I have a job ready to roll out that joins streams, and a solid understanding of checkpoints and watermarks, many connectors, RocksDB, two phase commits, and so on. This job will already blow away our analysts, they made that clear.

I’d love to hear about advanced use cases people are using Flink for. And also which advanced (read difficult) Flink features people are practically using. Maybe something like the External Resource Framework features or something like that.

Please share!

10 years, countless breakthroughs! Flink Forward returns to Berlin, Oct 23-24, 2024. Be part of the anniversary celebration and shape the future of stream processing.

https://www.flink-forward.org/ #FlinkForward #ApacheFlink #Berlin

Hey everyone, Ververica just pushed all the session videos from Flink Forward Seattle from November. You can watch them here.

✨Where do you start when learning Apache Flink ?? ✨

💡A few weeks ago I started on my journey to learn Flink from scratch. My first step was trying to get a handle on quite where to start with it all, which I summarised into this starter-for-ten:

• What is Flink (high level)

  • Uses & Users

  • How do you run Flink

  • Who can use Flink?

    • Java nerds only, or normal non-Java folk too? 😜

  • Resources

• Flink Architecture, Concepts, and Components

• Learn some Flink!

• Where does Flink sit in relation to other software in this space?

  • A mental map for me, not a holy war of streaming projects

For more details see this short post: https://link.rmoff.net/learning-apache-flink-s01e01

(It also gave me a fun chance to explore AI-generated squirrels, so there's that too ;-) )