You add read replicas to scale a read-heavy app. What user-facing bugs does replication lag create and how do you prevent them?
Key points
- Read-your-writes violations, a user saves and immediately sees the old value on the next pageIn plain words: You save a change, reload, and see the old value, because the copy you read from has not caught up
- Monotonic read violations, refreshing flips between new and old data when requests hit different replicasIn plain words: You refresh and the data flips between new and old, because each request reached a different copy
- Route reads to the primary for a short window after a user's write, or when the request carries a recent write tokenIn plain words: Right after someone writes, send that person's reads to the main database for a short while
- Compare replica LSN or a version with the one returned by the write and wait or fall back if the replica is behindIn plain words: Or carry a marker of how far the write got and only read from a copy that has reached it, waiting or falling back otherwise
- Pin a session to one replica and monitor lag so lagging replicas drop out of the read poolIn plain words: Keep each person on one copy, and stop sending reads to any copy that falls too far behind
Model answer
Read replicas are the cheapest way to multiply read capacity, and they come with one catch: a replica is always slightly behind the primary. Usually milliseconds, under load or during a schema change sometimes minutes.
Bugs lag creates
- Read-your-writes. A user updates their display name, the app redirects to the profile page, and the profile page reads from a replica that has not applied the change yet. The user sees the old name and assumes the save failed.
- Monotonic reads. Two refreshes hit two replicas with different lag; the comment appears, disappears, and reappears.
- Causal violations. A reply is replicated before the message it answers, so a reader sees an orphaned reply.
Preventions, cheapest first
- Read from the primary after a write. Set a short-lived cookie or session flag when the user writes; route that user's reads to the primary for the next few seconds. Simple and covers most cases.
- Write position tokens. The write returns the primary's log position (LSN, GTID). The client sends it back on subsequent requests; the router picks a replica that has applied at least that position, or waits briefly, or falls back to the primary.
- Session pinning. Hash the user to one replica so at least their view is monotonic.
- Lag-aware pools. Health checks measure replica lag and remove replicas beyond a threshold from the read pool; alert when lag grows.
- Design for staleness where it is fine. Product listings, public feeds and analytics can be stale by seconds with no one noticing; reserve primary reads for the user's own data.
Mention that writes still all go to one primary, so replicas do not help a write-heavy workload, and that cross-region replicas add tens of milliseconds of lag by physics alone.
Likely follow-ups
- How would you pass a "last write position" from the client through to the read routing layer?
- When is reading stale data completely fine?