Message queues (Kafka, RabbitMQ, SQS)

A message queue holds messages between producers and consumers. Producers publish; consumers process asynchronously. This decouples services, buffers load, and enables at-least-once or exactly-once processing with retries and ordering guarantees (depending on the system).

Generic queue flow

flowchart LR P1[Producer 1] --> Q[Queue / Topic] P2[Producer 2] --> Q Q --> C1[Consumer 1] Q --> C2[Consumer 2]

Kafka vs RabbitMQ vs SQS (high level)

System	Model	Typical use
Kafka	Log/topic; consumers read offset; retention	Event streaming, high throughput, replay
RabbitMQ	Queues, exchanges, routing	Task queues, RPC, flexible routing
SQS	FIFO or standard queue; pull	AWS-native, decouple components, serverless

Kafka: topics and consumer groups

flowchart TB subgraph Topic["Topic: orders"] P0[Partition 0] P1[Partition 1] end Producer --> Topic Topic --> CG[Consumer Group] CG --> C1[Consumer A] CG --> C2[Consumer B] Note[Each partition consumed by one consumer in group]

Use a queue when you need async processing, load leveling, or decoupling. Choose Kafka for event log and replay; RabbitMQ for flexible routing and task queues; SQS for simple, managed queues on AWS.