emitskevich/kafka-reo repository overview

⁠What is it

Kafka Replicator Exactly-Once - the tool to replicate data with exactly-once guarantee between different clusters of Apache Kafka.

GitHub repo⁠.

⁠The problem

⁠Why one need replication between clusters

1. Making a copy of data for disaster recovery purposes.
2. Gathering data from different regions to the central one for aggregation.
3. Data sharing between organizations.
4. ...

More in Confluent docs⁠.

⁠What tools exist for cross-cluster replication

1. MirrorMaker⁠ from Apache Kafka.
2. Replicator⁠ from Confluent.
3. Simple self-made "consume-produce in the loop" application.

All these tools can provide only either at-most-once or at-least-once delivery.

⁠What about exactly-once

Apache Kafka has transactional API⁠, which can be used for exactly-once delivery. The fundamental idea is to commit consumer offset and producer records in a single transaction. Kafka Streams uses it to provide high-level abstraction and easy access to exactly-once benefits. It may be enabled literally without code change⁠, with one config option.

This works only within the same Kafka cluster.

⁠What's not enough

Replication tools from the list above are not compatible with exactly-once delivery. The reason is in such case consumer offsets and producer records live in different clusters. Apache Kafka can't wrap operations with different clusters in one transaction. There is Kafka Improvement Proposal⁠ how to make it possible and good reading about it⁠, but it exists from 2020 and nothing of it is implemented in 2023.

⁠The solution

⁠Theory

1. Replicate messages to destination cluster with at-least-once guarantee. Wrap the messages with some metadata and apply repartitioning.
2. Apply deduplication, unwrap and restore initial partitioning, using exactly-once delivery within the destination cluster.

It requires more than 2x resources as compared with just at-least-once replication, but does the job.

⁠Design schema

⁠Launch options

⁠Docker run

Set your bootstrap servers and topic name for both clusters and run:

docker run \
    -e KAFKA_CLUSTERS_SOURCE_BOOTSTRAP_SERVERS=source-kafka-cluster:9092 \
    -e KAFKA_CLUSTERS_SOURCE_TOPIC=source-topic \
    -e KAFKA_CLUSTERS_DESTINATION_BOOTSTRAP_SERVERS=destination-kafka-cluster:9092 \
    -e KAFKA_CLUSTERS_DESTINATION_TOPIC=destination-topic \
    emitskevich/kafka-reo

⁠Docker compose & k8s

See instructions in GitHub repo⁠.

⁠Best practices

Launch as close to destination cluster as possible. It has notable performance boost, since the step of deduplication uses transactional API of destination cluster and is latency-sensible.