Kafka vs RabbitMQ: When to Choose Each

Kafka vs RabbitMQ becomes a useful system design question when I stop comparing product names and start asking what the application must remember, who must receive each message, and what happens after a worker fails.

A build service makes the distinction concrete. CompileRevision asks a worker to do something. BuildCompleted records something that happened. Both can travel through either broker. Their lifecycle requirements are what make one model fit better.

My starting point: choose Kafka for retained event history that independent consumers can read and replay. Choose RabbitMQ quorum queues for routed work distribution with a per-delivery acknowledgment lifecycle. Then check RabbitMQ Streams and Kafka Share Groups before treating that starting point as a product limitation.

This article compares those models, follows the failure paths, and applies a decision framework to concrete workloads. Product details were checked against Kafka 4.3 and RabbitMQ 4.4 documentation in October 2026. Availability still depends on the broker version, client, and managed service you deploy.

Start with one product and two requirements

Imagine a CI platform that builds a repository revision. A request creates a build record and schedules compilation. JVM workers and native compilation workers need different jobs. Once a build finishes, the platform records a completion event that status dashboards, capacity reporting, and failure analysis can use independently.

• Compilation work: one suitable worker should claim a job. A failed attempt needs a bounded retry policy. An acknowledged job does not need to remain available as a replayable queue entry.
• Completion history: several services need the same facts. A new reporting service may need last week’s events, and a broken projection may need rebuilding without running the builds again.

A command expresses requested work. An event describes a fact. Those names help clarify intent, but they do not choose the broker for you. A command may need an audit trail, and an event may only need short-lived delivery. Write down the required lifecycle first.

Kafka: retain the history, track each reader’s position

A Kafka topic is split into partitions. Each partition is an ordered log of records, and each record has an offset within that partition. In a standard consumer group, consumers divide the partitions between them. A different group can read the same topic independently. The KafkaConsumer API describes assignment, positions, and committed offsets.

For BuildCompleted, the status projection and capacity reporting use different group IDs. If capacity reporting pauses, the status projection can continue. If reporting code has a bug, I can rebuild a replacement projection from an earlier retained offset without asking the producer to publish those facts again.

Committing an offset records progress. It does not delete the corresponding event. Retention controls how long records remain available. A consumer that falls behind the available history cannot recover missing records just by resetting its offset. Retention must cover the outage and recovery window the product requires.

Compaction is a different requirement. A compacted topic keeps the latest value for each key as cleanup progresses. It is useful for reconstructing keyed state, but it is not a complete history of every intermediate change. Set retention and cleanup policy deliberately, using the topic configuration reference.

Where the partition boundary matters

Kafka provides order within a partition, not a total order across the topic. If builds for one repository need sequencing, a repository key can place their records together under a stable partitioning strategy. Changing partition counts can change key placement. Concurrent application handlers can also complete effects out of sequence even when records arrive in order.

With standard consumer groups, a partition is assigned to one consumer at a time, so the number of actively assigned consumers in a group is bounded by its partitions. Adding workers alone does not remove a hot partition. These are properties of the standard consumption model, not a universal limit on all Kafka workers. Kafka Share Groups, discussed below, use a different model.

RabbitMQ: route work, then acknowledge each delivery

In the RabbitMQ queue model, a publisher sends a message to an exchange. Bindings determine the destination queues. Consumers of one queue compete for its deliveries. Independent services that each need a copy generally use separate bound queues.

A direct exchange matches a routing key exactly, a topic exchange matches routing-key patterns, and a fanout exchange routes to its bound destinations without using the key. A direct exchange can still match several queues. Also, a RabbitMQ topic exchange is routing machinery, not the same abstraction as a Kafka retained topic. See RabbitMQ exchanges.

For compilation jobs, the routing key can distinguish JVM work from native compilation. Workers consume the appropriate queue. A worker manually acknowledges a job after handling it successfully, allowing that message to be removed from that queue. If another queue received its own routed copy, that copy has an independent lifecycle.

I would evaluate quorum queues when this work needs replicated queue storage. They use a replicated log internally, but expose a work-queue lifecycle to the application. Progress requires a majority of that queue’s members. A three-member queue can tolerate one member being unavailable. Classic queues can store messages on disk but do not provide the same replication model. Read the quorum queue and classic queue documentation before equating durability with replication.

Once a message is acknowledged and removed from an ordinary work queue, a newly added consumer cannot request it as historical data. A new queue also does not inherit messages that older queues already processed. That is a reason to consider retained streams when historical reads are a requirement.

Prefetch controls outstanding work, not business order

Manual acknowledgments and prefetch let me bound the number of outstanding deliveries held by a consumer. Too much prefetch can leave slow workers holding work that another worker could finish sooner. Tune it against job duration, memory, and execution concurrency. See RabbitMQ consumers.

Queue order also needs a precise definition. Multiple workers can finish in a different order from delivery. Redelivery, priorities, and concurrent publishers complicate the observed sequence. A single active consumer with sequential application processing can constrain ordering, at the cost of parallelism. FIFO storage alone does not establish the order of database effects. The queue ordering documentation explains these boundaries.

Compare the lifecycle, not a list of checkmarks

The following table compares Kafka standard consumer groups with RabbitMQ quorum work queues. Streams and share groups come next because they change several rows.

Neither column says one broker is always faster or simpler. Message size, batching, replication, routing fanout, retained bytes, handler cost, and required failure behavior change the result. A throughput number without those conditions is not a useful selection rule.

Two features that change the usual comparison

RabbitMQ Streams provide retained, replayable history

RabbitMQ Streams are persistent, replicated append-only logs with non-destructive consumption. Readers can revisit retained data. Super streams partition a logical stream, with ordering within each partition. This makes “RabbitMQ cannot replay messages” an incorrect product-wide claim.

Streams have their own feature and client model. Do not assume ordinary queue TTL, priority, or dead-letter behavior carries over. If RabbitMQ is already operated well by the team, evaluate Streams against the required retention, client support, partitioning, and processing ecosystem before adding Kafka. The Streams documentation is the relevant comparison point.

Kafka Share Groups provide competing-worker consumption

Kafka Queues, built on Share Groups, became production-ready in Kafka 4.2. Multiple consumers can share a partition, allowing more workers than partitions. Records use acquisition locks and individual acknowledgment outcomes rather than only a group’s contiguous offset progress. See the Kafka 4.2 upgrade notes.

That parallelism trades away processing order. Lock expiry can make a record available again, and acknowledgment handling still needs care. Rejection also does not automatically write the record to an application dead-letter topic. Check the KafkaShareConsumer API and supported client versions.

If the team already uses Kafka, Share Groups may satisfy a job-distribution requirement without a second broker. They do not make Kafka’s routing, retention, ordering, and operations identical to RabbitMQ queues. Compare the actual consumption models your application will use.

Both brokers stop short of your database transaction

Acknowledgment is an overloaded word. I separate three boundaries: the broker accepted the publication, the consumer recorded progress, and the business effect committed. Success at one boundary does not prove success at the next.

Kafka producer acknowledgments establish a configured broker acceptance boundary. With acks=all, the producer waits for the current in-sync replicas, subject to the relevant configuration. This does not mean a reporting service updated PostgreSQL. See the producer configuration.

RabbitMQ publisher confirms are separate from consumer acknowledgments. There is another subtlety: an unroutable publication can still receive a positive confirm. Use mandatory and handle returned messages, or a deliberately designed alternate-routing arrangement, when a missing destination must be detected. A confirm alone does not prove successful processing or even routing to a queue. See publisher confirms and consumer acknowledgments.

Consider this failure: a worker updates the build status in PostgreSQL and commits. It then crashes before its RabbitMQ acknowledgment or Kafka progress commit reaches the broker. The message can be processed again. If progress is recorded before the database work, the opposite failure can skip unfinished work.

For a database-only effect, one useful design is a stable event or job ID, a unique processed-message receipt scoped to the logical handler, and the receipt plus domain update in one local transaction. A duplicate then finds the receipt and does not repeat the update. A delivery tag or offset alone is not a cross-system business identity.

For compilation, email, or another external effect, that database receipt is not enough. Use a durable execution state machine, a stable request identity, and downstream idempotency or reconciliation where available. Record attempts separately from the logical job so retries do not accidentally create new work.

Kafka transactions can coordinate Kafka output records and standard consumer-group offsets. They do not automatically include an independent PostgreSQL transaction or an external API. The Kafka delivery semantics documentation explains the boundary. On the producer side, changing application state and publishing a message remains a dual-write problem with either broker. A transactional outbox is one way to address that publication boundary.

Retries need an ordering and recovery policy

Retry behavior should answer four questions: which failures are transient, how long can work wait, what is the attempt budget, and who repairs work after the budget expires? A dead-letter destination is useful only when something owns inspection and recovery.

Kafka: decide whether a failed record blocks its successors

For an ordered consumer, keeping a partition paused while retrying preserves sequence but delays unrelated keys in that partition. Moving a failed record to a retry topic allows forward progress, but later events for the same entity may be processed first. That can be acceptable for independent reports and unacceptable for a state transition that depends on its predecessor.

Commit only progress that is safe for the chosen processing model. In a regular consumer group, asynchronous handlers must not commit past unfinished earlier records. A retry topic also needs an intentional delay mechanism, attempt tracking, and a terminal recovery path.

“Kafka has no dead-letter support” is too broad. Kafka Connect has configurable dead-letter topics for supported error paths, and Kafka Streams added built-in dead-letter support in 4.2. A custom consumer still needs an explicit policy. See Connect error reporting and the Streams upgrade guide.

RabbitMQ: avoid immediate requeue loops

Repeatedly returning a failing message to a queue can create a busy retry loop. Quorum queues have native configurable delayed retry from RabbitMQ 4.3, using linear backoff and disabled by default. Delays do not universally require a plugin or a TTL queue arrangement.

Retry budgets also depend on the acknowledgment outcome. Since 4.3, explicit basic.nack returns do not increment the failed-delivery counter in the same way as rejected or genuinely failed deliveries. A delivery limit alone therefore does not bound every application requeue loop. Match the retry policy to the exact client and broker behavior. See versioned quorum queue retry documentation.

For terminal failures, configure dead-letter routing and its reliability separately. Default dead-letter republishing does not have the same safety as explicitly configured quorum at-least-once dead-lettering. The latter still allows duplicates and can retain backlog when the target is unavailable. See dead-letter exchanges.

For either broker, recovery should preserve the logical job ID, record why it failed, distinguish another attempt from a new request, and expose whether replay would repeat an external effect. A queue with no repair owner is a place where failed work accumulates.

Apply the choice to real requirements

1. Rebuildable build-status and capacity projections

I would start with Kafka standard consumer groups. The requirement is retained facts, independent readers, and the ability to rebuild a projection. Each service owns its progress. I would choose retention from the recovery requirement and test the time needed to catch up. RabbitMQ Streams remain a valid alternative if their clients and surrounding ecosystem fit the team.

2. Compilation jobs routed to specialized workers

I would start with RabbitMQ quorum queues. The requirement is targeted work distribution with acknowledgment, bounded outstanding deliveries, retry behavior, and terminal-failure handling. The job state still belongs in an application model. If Kafka already exists, I would evaluate Share Groups before adding another broker solely for competing consumers.

3. Several independent services need every event

Both systems can do this. With RabbitMQ ordinary queues, bind a queue for each independent subscriber. With Kafka, use a consumer group for each subscriber. If a subscriber added next month must read this month’s retained events, that historical-read requirement favors a log model: Kafka or RabbitMQ Streams. Current fanout alone does not settle the choice.

4. Existing infrastructure fits the workload well

A second broker introduces deployment, upgrades, monitoring, access control, capacity planning, and incident response. If the existing platform meets the required semantics and the team can operate it reliably, keeping one system may be the better decision. I would add another only for a concrete capability or operating advantage that justifies the extra boundary.

Using both requires a bridge failure design

A hybrid can retain BuildCompleted events in Kafka and dispatch specialized work through RabbitMQ. It is useful when both requirements are substantial. But a bridge that publishes a RabbitMQ job and then commits its Kafka offset can crash between those steps and publish the job again after restart. Waiting for a publisher confirm does not atomically include the Kafka commit. Preserve stable IDs and make the destination job handling idempotent.

What I would ask in a design review

A useful design explanation is short and causal: “I need independent historical readers, so I am choosing a retained log with separate consumer progress. The replay window is part of the storage policy. External effects remain idempotent.” For a work queue: “I need specialized workers to claim jobs, so I am choosing routed quorum queues with manual acknowledgment and bounded retries. Job status remains in the application database.”

Those statements explain why the choice fits. They are stronger than claiming that one product owns streaming and the other owns messaging.

Validate the decision under backlog and failure

Before committing to a broker, I would test a representative workload with equivalent durability assumptions. The following is a validation plan, not a benchmark result:

• Workload: realistic message sizes, routing fanout, slow and fast jobs, key skew, bursts, and the expected number of independent readers.
• Capacity: growth in retained data or ready and unacknowledged work, recovery throughput, replica storage, and time to drain the oldest backlog.
• Failures: worker crashes before and after a database commit, broker-member loss, unavailable destinations, poison messages, and rolling deployments.
• Product outcomes: duplicate effects, lost work, ordering violations, end-to-end latency percentiles, and time to restore a projection or failed job.

Consumer lag is useful for Kafka, but record counts alone do not reveal how old or expensive the work is. Queue depth is useful for RabbitMQ, but outstanding deliveries and the age of unfinished jobs matter too. Alert on the product’s recovery and latency objectives, not just whether the broker process is running.

My default recommendation is to start from the required lifecycle: retained history and independent replay, or routed work with explicit delivery completion. Name the exact model you chose. Then design ordering, retries, idempotency, and operations around that model. Kafka and RabbitMQ both become easier to evaluate when those requirements are explicit.

For more architecture and production topics, browse my Best Practices articles or the complete article index.