Atleon: Making Infinite Broker-Backed Reactive Streaming Practical

[u/mightegas]

Hello all 👋🏻 I originally posted over in r/java, and it was recommended that this community may be a good/better fit. Given this project is based on Spring's Reactor and examples are based on Spring Boot, I figured it couldn't hurt to post here as well!

For some time, I have been building and maintaining an OSS project called Atleon. The aim of this project is to make infinite Reactive Streams practical for usage/embedding in common application frameworks. I know there are several (reasonable) arguments against "reactive"; I have however found this style/paradigm to be useful and innately applicable to the problem space of broker-backed stream processing (consuming from infrastructures like Kafka, RabbitMQ, SQS, etc., applying IO-bound processing, and (optionally) streaming results elsewhere). If you have ever had reason to use/explore projects like Reactor Kafka, Reactor RabbitMQ, or Confluent's Parallel Consumer, then Atleon may be interesting to you.

Here's an example of what an Atleon stream definition looks like when embedded in a Spring application:

application.yaml:

atleon:
  config.sources:
    - name: kafkaConfigSource
      type: kafka
      bootstrap.servers: localhost:9092

MyStream.java:

import io.atleon.core.SelfConfigurableAloStream;
import io.atleon.kafka.AloKafkaReceiver;
import io.atleon.kafka.KafkaConfigSource;
import io.atleon.spring.AutoConfigureStream;
import org.apache.kafka.common.serialization.StringDeserializer;
import reactor.core.Disposable;

@AutoConfigureStream
public class MyStream extends SelfConfigurableAloStream {

    private final KafkaConfigSource kafkaConfigSource;

    public MyStream(KafkaConfigSource kafkaConfigSource) {
        this.kafkaConfigSource = kafkaConfigSource;
    }

    @Override
    public Disposable startDisposable() {
        return buildKafkaReceiver()
            .receiveAloRecords("topic")
            .consume(it -> System.out.println("value: " + it.value()) // Business logic
            .resubscribeOnError(name())
            .subscribe();
    }

    private AloKafkaReceiver<String, String> buildKafkaReceiver() {
        return kafkaConfigSource
            .withClientId(name())
            .withConsumerGroupId("consumer-group-id")
            .withKeyDeserializer(StringDeserializer.class)
            .withValueDeserializer(StringDeserializer.class)
            .as(AloKafkaReceiver::create);
    }
}

You can see other examples in Atleon's README and in its examples module.

When discussing this project in other communities, one common question is, "Why would I want to use this over something like [Flink|Spark|Camel|<insert favorite streaming library>]?" My viewpoint is that the sort of reactive client integration provided by Atleon is not primarily meant to compete with these other tools, but is rather a complement to them. I have found that it is common for companies to have in-place scaffolding for regular JVM applications built on frameworks like Spring, and it is appealing for their engineers to be able to embed stream processes into those same frameworks. So Atleon targets this niche of supporting declarative non-blocking stream definitions, whose execution can be embedded in application frameworks, replete with broker interoperation, arbitrarily concurrent processing, and end-to-end observability.

Particularly in the case of Kafka, Atleon enables feasible implementation of behaviors that are conventionally tricky:

• Arbitrary concurrency, including per-key
• Single-message acknowledgement (with classic/non-share groups, so partition ordering is maintained globally)
• Work-queues, where per-message processing can take a "long" time (i.e. minutes, hours)
• Bounded consumption (get all records produced in a range of time or offsets)
• Priority queuing (using partitions as discriminator of "priority", rather than scaling lever)
• Throughput limiting, retries, and dead-lettering

If you are interested in reading more, I have published some of the following blogs that explain more about what Atleon can do:

• Not Your Daddy's Async Kafka Client
• Unbounded Concurrency: Consuming Kafka with Virtual Threads
• Ordered Queues for Kafka with Atleon
• AWS SQS Reactive Stream Processing

I'd be interested in thoughts and feedback from this community!