Write-heavy Systems

Updated Nov 16, 2026 · 5 min read
On this page

    A video platform records a "view" event every time someone watches a few seconds, plus heartbeats every 10 seconds while they keep watching. At peak that is 50,000 writes per second. The first design inserts each event as a row and runs UPDATE videos SET views = views + 1 for each one. The database's disk and lock queue saturate, the hottest video's row becomes a single point of contention, and inserts start timing out, so events are lost.

    Systems where the main load is writes (logs, metrics, clicks, sensor data, likes, chat messages, location updates) are write-heavy. Caching does not help them much, because every write must land somewhere. Instead, they rely on a different set of tools.

    How it works

    Absorb, then apply

    Log then batch: clients go through the API Gateway and load balancer to an ingest service that appends to a partitioned log and acknowledges in about 5 ms; writers read the log in batches of 1,000 and
    Log then batch: clients go through the API Gateway and load balancer to an ingest service that appends to a partitioned log and acknowledges in about 5 ms; writers read the log in batches of 1,000 and write to three database shards.
    1. Requests come through the API Gateway and load balancer to a thin ingest service.
    2. It validates the event and appends it to a partitioned log. Appending is sequential I/O, the cheapest kind of write, and a log cluster takes hundreds of thousands of events per second.
    3. Once the log has stored it (on several brokers), the service acknowledges the client. The client is done in milliseconds.
    4. Writer consumers read the log and apply events to storage in batches: one insert of 1,000 rows is far cheaper than 1,000 single inserts.

    The log is also a buffer: if storage slows down or is being upgraded, events wait in the log (for days if needed) instead of being rejected.

    Combine writes

    Combining writes: many +1 view events for video 9 are summed in an in-memory counter that flushes once per second as a single UPDATE adding 48,213, so 50,000 increments per second become one write per
    Combining writes: many +1 view events for video 9 are summed in an in-memory counter that flushes once per second as a single UPDATE adding 48,213, so 50,000 increments per second become one write per hot key per second.

    Many writes are increments or overwrites of the same key. Combine them first:

    • Counters: sum increments in memory per key and flush every second. 50,000 +1s become one +48,213.
    • Latest value wins: for location updates, keep only the newest position per driver in the buffer.
    • Windows: stream processors (Flink, Kafka Streams) aggregate per minute before writing results.

    If the aggregator crashes, it replays from the last log offset it committed, so nothing is lost. Commit the offset only after the flush succeeds, and make the flush idempotent (for example, write "total for minute 12:03" rather than "add N").

    Spread the load

    • Partition by a key with even traffic: device_id, user_id, or a hash. Avoid time-only keys, which send all current writes to one partition.
    • Salt hot keys: a celebrity's like counter can be split into 16 sub-counters (likes:post7:0 … :15), summed on read.
    • Scale partitions with consumers: one consumer per log partition; more partitions allow more parallel writers.

    Storage built for writes

    • LSM-tree databases (Cassandra, ScyllaDB, RocksDB-based stores) turn random writes into sequential ones: write to memory and a log, flush sorted files, merge later. High write throughput, at some cost to reads.
    • Time-series databases for metrics: compress well, expire old data automatically.
    • Object storage for raw events, written as large batched files (for example, Parquet every 5 minutes) for analytics.
    • A relational database still works for moderate rates (thousands per second) with batching and partitioning.

    Trade-offs and costs

    • Eventual consistency: counts and dashboards lag by seconds. Say how much is acceptable.
    • Durability vs latency: acknowledging after the log write (replicated) is safe; acknowledging before it is faster but can lose data. Choose per write: payments must be durable; a "user is typing" signal can be lost.
    • Duplicates: logs redeliver after failures, so writes must be idempotent (event IDs, upserts).
    • Ordering: guaranteed only within a partition. Partition by the key whose order matters.
    • Read cost: write-optimized stores and pre-aggregated data make some reads harder; build separate read models if needed.
    • Operational load: the log cluster, consumers and lag monitoring are real systems to run.

    When to reach for it

    Signals in the prompt: "every click", "every heartbeat", "sensor readings", "logs from all servers", "location updates every 4 seconds", "count views or likes", "ad impressions". Ask:

    • What is the write rate at peak, and the event size?
    • Must every event be stored, or only aggregates?
    • How fresh must reads be?
    • How long is data kept?

    Example

    Video view counting. Assumptions: 50,000 view events per second at peak, 200 bytes each, counts may lag by 10 seconds, raw events kept 30 days for analytics.

    • Ingest: gateway → ingest service → log topic views, 32 partitions keyed by video_id. About 10 MB per second, comfortable for a small log cluster.
    • Counting: 8 aggregator instances, each owning 4 partitions. They sum views per video in memory and flush every 5 seconds with UPDATE video_counts SET views = views + :n WHERE video_id = :id, plus a dedupe table of (aggregator, offset range) so a replayed batch isn't added twice.
    • Hot video: a viral video is one key, but it costs one update per flush no matter how many views it gets, so no special handling is needed for the counter. Its page reads go through cache.
    • Raw events: a second consumer group writes 5-minute Parquet files to object storage, about 860 GB per day, with a 30-day lifecycle rule.
    • Freshness: the count on screen is at most about 10 seconds behind.

    Common mistakes

    • Writing every event synchronously to the main database.
    • One row updated by everyone (a global counter) with no aggregation or salting.
    • Partitioning by time so all current writes hit one partition.
    • Assuming exactly-once delivery and double-counting on replay.
    • Adding a cache to a write-heavy path and expecting it to help.
    • No retention plan, so raw data grows forever.

    Say this in the interview

    "This is write-heavy: 50,000 events a second. The ingest service appends each event to a log partitioned by video_id and acks after the log write, about 5 ms. Aggregators sum views in memory and flush one update per video every few seconds, so the database sees thousands of writes a second, not fifty thousand. Offsets are committed after each flush and the flush is idempotent, so a crash just replays. Raw events also go to object storage in batched files for analytics."

    Used in