Message Queues and Event Streaming

Why Message Queues Matter

In a distributed system, services need to communicate with each other. The simplest approach is synchronous HTTP calls: Service A sends a request to Service B and waits for a response. But this creates tight coupling — if Service B is slow or down, Service A is blocked or fails.

Message queues decouple services by introducing an intermediary. Instead of calling Service B directly, Service A publishes a message to a queue. Service B consumes messages from the queue at its own pace. This simple change provides enormous benefits for reliability, scalability, and system resilience.

Messaging Patterns

1. Point-to-Point (Queue)

A message is sent to a queue and consumed by exactly one consumer. If multiple consumers are listening, each message is delivered to only one of them. This is the classic work queue pattern used for distributing tasks across workers.

// Point-to-Point Architecture
//
// Producer ---> [ Queue ] ---> Consumer 1
//                         ---> Consumer 2  (each message goes to only ONE consumer)
//                         ---> Consumer 3
//
// Example: Order processing
// [Order Service] -> [order-processing-queue] -> [Payment Worker 1]
//                                             -> [Payment Worker 2]
//                                             -> [Payment Worker 3]
// Each order is processed by exactly one worker

2. Publish/Subscribe (Topic)

A message is published to a topic and delivered to all subscribers. Each subscriber receives a copy of every message. This is used when multiple services need to react to the same event.

// Publish/Subscribe Architecture
//
// Producer ---> [ Topic ] ---> Subscriber 1 (gets ALL messages)
//                         ---> Subscriber 2 (gets ALL messages)
//                         ---> Subscriber 3 (gets ALL messages)
//
// Example: User signup event
// [Auth Service] -> [user-signup topic] -> [Email Service]     (sends welcome email)
//                                       -> [Analytics Service] (tracks signup)
//                                       -> [CRM Service]       (creates lead)
// ALL services receive the signup event

3. Consumer Groups (Kafka Pattern)

Kafka combines both patterns with consumer groups. A topic is divided into partitions. Within a consumer group, each partition is consumed by exactly one consumer (point-to-point within the group). But multiple consumer groups can subscribe to the same topic (pub/sub across groups).

// Kafka Consumer Groups
//
// Topic: order-events (4 partitions)
//
// Consumer Group A (Order Processing):
//   Consumer A1 -> Partition 0, Partition 1
//   Consumer A2 -> Partition 2, Partition 3
//   (each message processed by exactly one consumer in group A)
//
// Consumer Group B (Analytics):
//   Consumer B1 -> Partition 0, Partition 1, Partition 2, Partition 3
//   (each message ALSO processed by group B independently)

Kafka, RabbitMQ, and SQS Compared

These are the three most popular messaging systems, each designed for different use cases.

Message Ordering and Delivery Guarantees

Understanding delivery guarantees is critical for building correct distributed systems. There are three levels:

At-Most-Once Delivery

Messages may be lost but are never delivered twice. The producer sends a message and does not wait for acknowledgment, or the consumer acknowledges before processing. If anything fails, the message is lost. This is the fastest option but only suitable when message loss is acceptable (e.g., metrics, logging).

At-Least-Once Delivery

Messages are guaranteed to be delivered at least once but may be delivered multiple times. The consumer acknowledges only after successfully processing the message. If the acknowledgment is lost, the message will be redelivered. This is the most common choice, but consumers must be idempotent — processing the same message twice must produce the same result.

// At-least-once delivery with idempotent consumer
class OrderProcessor {
  private processedIds: Set<string> = new Set();
  private db: Database;

  async processMessage(message: OrderMessage): Promise<void> {
    // Idempotency check - have we already processed this?
    const existing = await this.db.query(
      'SELECT id FROM processed_orders WHERE order_id = $1',
      [message.orderId]
    );

    if (existing.rows.length > 0) {
      console.log(`Order ${message.orderId} already processed, skipping`);
      return; // Idempotent - safely skip duplicate
    }

    // Process the order
    await this.db.transaction(async (tx) => {
      await tx.query('INSERT INTO orders VALUES ($1, $2, $3)',
        [message.orderId, message.userId, message.amount]);

      // Record that we processed this message
      await tx.query('INSERT INTO processed_orders (order_id) VALUES ($1)',
        [message.orderId]);
    });
  }
}

Exactly-Once Delivery

Every message is delivered exactly once — no loss, no duplicates. This is the holy grail of messaging but is extremely difficult to achieve in distributed systems. Kafka supports exactly-once semantics within the Kafka ecosystem using idempotent producers and transactional consumers, but true end-to-end exactly-once requires careful design across the entire pipeline.

Dead Letter Queues (DLQ)

A dead letter queue is a special queue where messages that cannot be successfully processed are sent. Instead of retrying forever or dropping failed messages, you route them to a DLQ for later inspection and manual intervention.

// Dead letter queue pattern
class MessageConsumer {
  private mainQueue: Queue;
  private deadLetterQueue: Queue;
  private maxRetries: number = 3;

  async consumeMessage(message: Message): Promise<void> {
    const retryCount = message.headers['x-retry-count'] || 0;

    try {
      await this.processMessage(message);
      await this.mainQueue.ack(message);
    } catch (error) {
      if (retryCount >= this.maxRetries) {
        // Max retries exceeded - send to dead letter queue
        console.error(`Message ${message.id} failed after ${this.maxRetries} retries`);
        await this.deadLetterQueue.publish({
          ...message,
          headers: {
            ...message.headers,
            'x-original-error': error.message,
            'x-failed-at': new Date().toISOString(),
          },
        });
        await this.mainQueue.ack(message); // Remove from main queue
      } else {
        // Retry with exponential backoff
        const delay = Math.pow(2, retryCount) * 1000; // 1s, 2s, 4s
        await this.mainQueue.nack(message, {
          delay,
          headers: { 'x-retry-count': retryCount + 1 },
        });
      }
    }
  }
}

When to Use Message Queues

Architectural Best Practices

• Always design idempotent consumers. In distributed systems, duplicate messages are inevitable. Your consumers must handle them gracefully.
• Set up dead letter queues. Do not let poison messages block your processing pipeline. Route them to a DLQ for investigation.
• Monitor queue depth. Growing queue depth means consumers are not keeping up. Set up alerts for queue depth thresholds.
• Use backpressure. If consumers are overwhelmed, slow down or pause producers rather than letting the queue grow unbounded.
• Choose the right tool. Use Kafka for high-throughput event streaming, RabbitMQ for complex routing and task queues, SQS for simple serverless integration.