05

Concepts

Messaging & Event Driven

How systems talk to each other without blocking. Queues, brokers, streams, and the patterns that make async pipelines reliable.

Why queues exist, what problem they solve, and the core model: producer puts work in, consumer takes it out, durably.

ProducerConsumerDurabilityAsync Processing

Background job execution — offloading CPU-heavy or slow work from the request path into a worker pool that processes at its own pace.

Background JobsWorker PoolJob SchedulingCelery

One publisher, many subscribers. The model that decouples event producers from consumers and enables fan-out without direct coupling.

PublisherSubscriberTopicFan-Out

Delivery Guarantees

At-most-once, at-least-once, exactly-once. What each guarantee costs and which systems actually deliver exactly-once end-to-end.

At-Most-OnceAt-Least-OnceExactly-OnceIdempotency

Dead Letter Queue

What happens to messages that can't be processed. DLQs prevent poison pills from blocking a queue and give you a recovery path for failed messages.

DLQPoison PillRetry LimitMessage Recovery

Message Ordering

Why ordering is hard in distributed queues, when it matters, and the tradeoff between strict ordering and throughput.

FIFOPartition KeyOrdering GuaranteesThroughput Tradeoff

Scheduling messages to be visible after a delay. The pattern behind retry backoff, scheduled jobs, and time-based workflows.

Visibility TimeoutScheduled MessagesRetry BackoffSQS Delay

Priority Queues

Processing high-priority messages before low-priority ones. The implementation patterns and the starvation problem that comes with them.

Priority LevelsStarvationMulti-QueueHeap-Based

Fan-Out on Write

Push updates to all followers at write time. Fast reads, expensive writes — the pattern Twitter uses for most users.

Write AmplificationPush ModelInbox PatternCelebrity Problem

Fan-Out on Read

Aggregate the feed at read time instead of write time. Cheaper writes, more expensive reads — used for celebrity accounts where fan-out on write breaks.

Pull ModelRead AggregationHybrid Fan-OutHot Users

Message Brokers

SQS and RabbitMQ in depth — architecture, delivery guarantees, routing, ACK/NACK, DLQ, scaling, and when to use each.

SQSRabbitMQACK / NACKExchange Types

Architecture, producers, consumers, partitions, ISR, offsets, consumer groups, rebalancing, retention, and exactly-once processing.

PartitionsConsumer GroupsOffsetsISR

Broker Comparison

Kafka vs SQS vs RabbitMQ — what fundamentally differs, when to choose what, and the tradeoffs around retention, ordering, and throughput.

Kafka vs SQSRetentionOrderingThroughput

Consumer lag, scaling consumers, load shedding, and backpressure signals. What to do when your consumers can't keep up with your producers.

Consumer LagLoad SheddingScalingBackpressure Signals

Event-Driven Architecture

Event sourcing, CQRS, dual-write problem, outbox pattern, inbox pattern, and CDC. The full toolkit for reliable event-driven systems.

Event SourcingCQRSOutbox PatternInbox Pattern

Stream Processing

Window types, watermarks, statefulness, and crash recovery. Processing infinite event streams in real time without losing data.

WindowingWatermarksStateful ProcessingCrash Recovery

Batch Processing

MapReduce and Spark — how to process petabytes of data in bounded jobs. Map, shuffle, reduce, DAG optimization, and hot key handling.

MapReduceSparkDAGHot Keys

Architectures for combining batch and stream processing. Lambda's two-layer approach vs Kappa's stream-only simplification.

Lambda ArchitectureKappa ArchitectureBatch LayerSpeed Layer

Schema Evolution

Schema Registry, Protobuf, and Avro. How you change message formats over time without breaking producers and consumers that deploy independently.

Schema RegistryProtobufAvroBackward Compatibility