Compare a traditional message queue (RabbitMQ, SQS) with a distributed log (Kafka, Kinesis). When do you want each?
Key points
- A queue deletes a message once a consumer acknowledges it, a log retains messages and consumers track their own offsetIn plain words: A queue throws a message away once someone has handled it, while a log keeps everything and each reader remembers its own place
- Logs allow many independent consumer groups to read the same stream and to replay historyIn plain words: With a log, many separate readers can go through the same stream and go back to reread
- Queues offer per-message routing, priorities and simple competing consumers, logs offer ordering within a partition and very high throughputIn plain words: Queues are good at routing, priorities and sharing jobs among workers, and logs are good at strict order within a lane and sheer volume
- Replay and multiple readers make logs the backbone for event sourcing, CDC and stream processingIn plain words: Because it can be reread by many, a log is the base for rebuilding state, copying database changes and processing streams
- Queues are simpler for task distribution where each job should be done once and forgottenIn plain words: A queue is the simpler pick when each job should be done once and then forgotten
Model answer
Both move messages from producers to consumers asynchronously; they differ in what happens to a message after it is read.
Message queue (RabbitMQ, SQS, ActiveMQ). A message is delivered to one consumer, acknowledged, and deleted. Competing consumers share a queue to parallelise work. Brokers support routing rules, priorities, per-message TTL and delayed delivery. The mental model is a to-do list: each job is done by exactly one worker and then gone.
Distributed log (Kafka, Kinesis, Pulsar, Redpanda). Messages are appended to a partitioned, ordered log and retained for a configurable period (hours to forever). Consumers do not delete anything; each consumer group remembers its own offset. The same stream can be read by the billing team, the search indexer and a fraud model, each at its own pace, and any of them can rewind to reprocess. Throughput scales with partitions; ordering is guaranteed within a partition.
| Need | Queue | Log |
|---|---|---|
| Background jobs done once | yes | awkward |
| Several teams consuming the same events | fan-out exchanges, copies per queue | native, one topic |
| Replay last week's events | no | yes |
| Strict ordering for a key | per queue only | per partition (key-based) |
| Millions of events per second | hard | designed for it |
| Per-message delay, priority | yes | no |
Choosing. Image resizing, email sending, webhook delivery: a queue. Order events that feed analytics, search, notifications and an audit trail: a log. Many systems use both: a log as the system of record for events and queues as work buffers in front of specific workers.
A good closing line: "a queue is about distributing work; a log is about sharing a durable history."
Likely follow-ups
- How does a Kafka consumer group achieve parallelism, and what limits it?
- What happens in each system when a consumer is slow for an hour?