Designing a Chat System from First Principles

Building a chat application sounds deceptively simple until you look beyond the happy path. Messages need to arrive almost instantly, remain available when users go offline, stay in order, and scale to millions of concurrent connections. This article builds a modern chat system from first principles, introducing one architectural component at a time as each new problem emerges. 

The Problem

Picture a messaging app. Two people, or a group, exchange messages that need to show up within milliseconds, in the right order, and exactly once as far as anyone can tell. On top of that, if someone's phone is off for a week, every message sent to them in that time still has to be there when they turn it back on.

The scope here covers sending and receiving one-on-one and group messages, delivering messages that arrived while someone was offline, keeping messages in order within a conversation, showing sent, delivered, and read receipts, and tracking who's currently online or when they were last seen. End-to-end encryption is worth a separate look, since it changes what the server is allowed to see.

A few questions shape almost everything about how this gets built. Are groups small, or do they run into the tens of thousands? A group of fifty behaves nothing like a group of fifty thousand. What delivery guarantee actually matters? Promising true exactly-once delivery across a distributed system is expensive, and in practice, promising at-least-once delivery with the client throwing away duplicates gets you the same experience for a fraction of the cost. Does ordering need to hold across every conversation someone's part of, or just within each one? Per-conversation ordering is achievable and is genuinely what people need. And what happens to a message sent to someone who's been offline for a week? That question alone is what forces durable storage and a way to catch up on reconnect into the design.

What Makes This Hard

The naive version of chat is simple to describe: the client sends a message, the server passes it along. That falls apart the moment you notice the recipient is often not connected when the message arrives. The message can't just vanish because nobody was listening, and it can't sit around waiting to be polled for either, because that isn't real-time anymore.

Every message has to be saved and pushed at the same time, to someone who might be sitting on any server in the fleet, or might not be connected to anything at all. Figuring out how a message reaches someone who could be offline is the actual design problem here, not the easy case of two people who happen to be online together at the same moment.

The idea that ties the whole system together: chat is a durable log, one per conversation. Clients read new messages as they arrive, and re-read whatever they missed the moment they reconnect. The real-time delivery path and the durable storage underneath it are kept as two separate concerns.

The Concepts This Design Rests On

Persistent connections and knowing where everyone is

Clients hold a long-lived connection, a WebSocket, open to a gateway server, so the server can push messages down to them instead of the client constantly asking "anything new?" With hundreds of millions of clients connected at once, that means a large fleet of gateways, and since any given user could be sitting on any one of them, the system needs a connection registry: a map from user to gateway, rebuilt every time someone connects or disconnects, living in a fast in-memory store so lookups stay cheap.

Storing the message before trying to deliver it

A message that's been acknowledged to its sender is already safely stored. The message service writes it to the durable store and assigns it its place in the sequence before it acknowledges anything or attempts delivery. Live delivery, then, isn't the system of record. It's a speed optimization sitting on top of a record that already exists.

At-least-once delivery, with duplicates quietly dropped

Networks retry. That means the same message can show up twice on the wire. The sender attaches an id to each message it sends, and both server and client use that id to recognize and discard duplicates. The underlying guarantee is at-least-once, but because repeats get filtered out, what the person actually experiences is exactly-once. Chasing true exactly-once delivery across separate systems costs a lot more than this approach and buys very little extra in return.

Ordering within a conversation, not across all of them

Every conversation gets its own steadily increasing sequence number, assigned the moment a message is stored. Clients sort and deduplicate using that number, which keeps everyone in a conversation seeing the same order without needing any kind of global clock. Ordering across unrelated conversations isn't something anyone needs, and it isn't something you'd want to pay for even if they did.

Sending a message to a group

A message to a group of N people becomes N individual deliveries: pushed straight to whoever's online, queued for whoever isn't. For a small group, that write-time fan-out is cheap. Once a group gets large, the more sensible move is a shared log that members read from on their own, which we'll come back to.

Presence

Whether someone's online, and when they were last seen if they're not, is short-lived and doesn't need to be perfectly accurate at every instant. It has no business living in the durable message path. The mechanism is a heartbeat with an expiration: each user gets an in-memory key that expires after a few seconds unless the client keeps refreshing it. As long as heartbeats keep arriving, the user reads as online. If the app crashes, the heartbeats stop, the key expires a few seconds later, and the person flips to offline with a last-seen timestamp attached. Because this whole thing gets rebuilt on connect and kept alive purely by heartbeats, losing the entire presence store costs nothing more than a brief refresh.

The pattern underneath all of this: store before you deliver, order each conversation with its own sequence number, deduplicate using a client-supplied id, and keep routing and presence information ephemeral and off the durable path entirely.

Requirements

What it needs to do. Send and receive one-on-one and group messages in real time. Deliver messages sent while someone was offline, once they reconnect. Keep messages ordered within a conversation. Provide sent, delivered, and read receipts, plus online and last-seen presence.

What it needs to guarantee. Durability comes first: once a message is acknowledged, it must never be lost. Latency matters too, online-to-online delivery should land well under a second. Ordering needs to be consistent across everyone in a conversation. Availability matters because connections drop constantly, and a client should be able to reconnect and catch up within seconds. And the whole thing needs to hold up at hundreds of millions of concurrent connections and millions of messages a second.

The property worth protecting above all else is durability: once something is acknowledged, it's stored, full stop, which is exactly why the message service writes to durable storage before it acknowledges or delivers anything. The thing that actually shapes the architecture, though, is the demand for real-time delivery to someone who could be on any gateway, or on none at all. That single requirement is what forces the persistent-connection fleet, the connection registry, and the split between the live delivery path and the durable store.

Sizing the System

A few rough numbers help size the pieces. Say there are 100 million concurrent users, roughly 100,000 connections per gateway server, a million messages sent per second, an average of three recipients per message, and messages averaging a kilobyte each, with a year of history retained.

That works out to about a thousand gateway servers, just from dividing concurrent connections by connections per gateway. Deliveries land around three million a second, since each of the million messages sent per second reaches an average of three people. Writing all of that to durable storage comes out to roughly a gigabyte a second before replication, and a year of retained history, replicated three times over, adds up to somewhere in the neighborhood of 95 petabytes.

Two things fall out clearly from this. The number of concurrent connections is what determines how big the gateway fleet needs to be, not the message volume. And it's the message rate multiplied by how many people each message reaches that determines both the delivery rate and how fast the durable store needs to absorb writes.

The Interface

Imagine a client sends a message, the acknowledgment gets lost on the way back, and the client retries the send. The two ways to handle that: have the server drop anything that arrives within a suspiciously short window of an identical message, or have the client attach an id to the message and let the server recognize and discard the duplicate by that id. The second one is the approach that actually holds up, since timing windows are a guess and an explicit id isn't.

The interface itself is a persistent connection plus four operations. connect() opens that connection. send(conversation_id, client_msg_id, body) sends a message, carrying that client-supplied id so the server can deduplicate it, and returns the sequence number it was assigned within the conversation. sync(conversation_id, since_seq) asks for everything that's happened in a conversation since a given point, which is exactly what a client calls after reconnecting from being offline. ack(conversation_id, up_to_seq) tells the server how far the client has actually read or delivered up to.

The Data Model

A message needs to belong to some conversation and needs a sequence number to be ordered by. Beyond that, each additional field earns its place by covering something the record couldn't otherwise represent.

A conversation has an id, a type (one-on-one or group), a list of member ids, and the sequence number of its most recent message. A message has the conversation id it belongs to, its own sequence number within that conversation, the sender's id, the client-supplied id used for deduplication, the body, and a creation timestamp. A cursor, tracked per user per conversation, records that user's id, the conversation id, how far messages have been delivered to them, and how far they've actually read.

Messages themselves are partitioned by conversation id inside a durable log or key-value store, ordered by that per-conversation sequence number, which is what gives ordering without needing any kind of global clock. The connection registry, mapping users to gateways, and presence, tracking who's online or when they were last seen, live entirely in memory. Both get rebuilt on connect and never touch the durable message path.

Building the Architecture, One Failure at a Time

The easiest way to understand why this system looks the way it does is to start with the simplest possible version and watch where it breaks.

One server. A single server holds every connection and passes messages directly between them. This falls apart almost immediately: one machine can't hold hundreds of millions of sockets, and the moment there's more than one server, a sender's server has no way of knowing which server the recipient happens to be connected to.

A gateway fleet and a registry. Spread connections across many gateway servers, and add a connection registry that records which gateway each user is on, so a message can find its way to the right place. This solves the connection-scale problem, but it introduces a new one: gateways now need a way to talk to each other, and having every gateway hold a connection to every other gateway doesn't scale past a handful of machines, let alone a thousand.

A message service and an internal bus. Add a message service that owns each message as it moves through the system, and an internal publish-subscribe bus, essentially a message broker where a sender publishes to a named topic and whichever servers are subscribed to that topic receive the message. The mechanism is one topic per gateway. Gateway 17 subscribes to its own topic and nothing else. To reach a user sitting on Gateway 17, the message service checks the registry, publishes to that gateway's topic, and Gateway 17, the only subscriber, receives the message and pushes it down that user's socket. No gateway ever sees traffic meant for connections it doesn't hold.

This gets a message to an online recipient's gateway just fine. If the recipient is offline, though, no gateway is subscribed to receive anything on their behalf, and the message has nowhere to go. Without more than this, it would simply be dropped.

A durable store and offline delivery. The message service writes every message to durable storage, assigning it its sequence number, before it acknowledges the sender. If the recipient happens to be offline, the message is already safely stored regardless. It gets marked pending, a push notification goes out, and the recipient's client pulls the actual message down using sync the next time it reconnects.

Put together, sending a message now looks like this: the client sends it to its gateway, the gateway forwards it to the message service, the message service persists it and assigns a sequence number, and only then acknowledges the sender. Meanwhile it checks the registry for the recipient. If they're online, the message goes out over the bus to their gateway and down their socket. If they're offline, a push notification goes out instead, and the message waits in storage until the recipient reconnects and syncs.

Each piece of this architecture exists to answer one specific failure: the gateway fleet solves for connection scale, the registry solves for finding the recipient, the bus solves for getting a message across gateways, and the durable store solves for a message surviving the recipient being offline entirely.

A Closer Look at a Few Parts

How a message actually finds someone

Say a message has been stored and acknowledged, and the recipient is sitting somewhere in a fleet of a thousand gateways, or possibly nowhere at all. The registry answers that question: it maps user to gateway, gets updated on every connect and disconnect, and lives in a store fast enough to check on every single message. If the person's online, the message travels over the bus to their gateway and down their socket. If they're offline, it's already sitting in durable storage, so the system marks it pending, sends a push notification, and waits for the client to call sync once it reconnects. Heartbeats catch dead connections the server hasn't noticed on its own yet, and clear out the registry entry so nothing gets routed to a gateway that isn't actually holding that connection anymore.

Here's what that looks like concretely. Alice and Bob's conversation holds messages numbered 41 through 45. Bob's phone drops its connection right after his client has seen message 42, so it remembers that it's seen up to sequence 42. While he's offline, Alice sends three more messages, 43, 44, and 45, each one persisted to the log as it arrives. When Bob reconnects, his client calls sync asking for everything since 42. The server hands back messages 43, 44, and 45 straight from the log, and his client moves its cursor up to 45. Nothing gets missed, nothing shows up twice, because it's the cursor, not the live socket, that defines what he still needs to see. A week offline works exactly the same way. The gap is just bigger.

Keeping order and handling repeats

Imagine a client reads a message, the acknowledgment back to the server gets lost, and the client retries. Without care, the person on the other end sees that message twice, possibly out of order. The actual contract here is at-least-once delivery paired with deduplication on the receiving end, which together look like exactly-once delivery from the outside. The steadily increasing sequence number assigned to each conversation gives everything its order: clients sort and deduplicate by that number, so anything delivered twice or out of sequence still ends up exactly where it belongs, exactly once. Receipts travel through the same mechanism, moving from sent, meaning stored, to delivered, meaning it reached the device, to read, meaning it was opened, each one just a small update to that user's cursor for that conversation.

Large groups and presence at scale

Picture a group with fifty thousand members posting a single message. Treating that like fifty thousand individual sends is a lot of work to repeat on every single message. For a group of fifty, pushing the message out to each member on write is cheap, turning one send into fifty delivery tasks, which is entirely manageable. Do the same thing for a group of fifty thousand, though, and every message becomes fifty thousand delivery tasks plus fifty thousand sets of cursor bookkeeping, repeated every single time anyone posts.

Past a certain size, the better move is to flip the model: store the message once in a shared log, and let each member read it on their own schedule, tracked by their own cursor. That turns fifty thousand writes back into one, at the cost of each client doing a bit more work to catch up. It's the same tradeoff large feed systems make for accounts with huge followings, choosing to fan out on read instead of on write once the numbers get big enough. Presence rides alongside all of this without ever touching the durable path. It's in-memory, it doesn't need to be perfectly precise, and at large scale it gets sampled aggressively, so an online indicator that's a few seconds stale barely matters and corrects itself on the next signal.

Variants Worth Naming

For very large groups and public broadcast channels with thousands of members, the shared-log approach carries over directly: participants read from one durable log rather than each receiving their own pushed copy.

For end-to-end encryption, the server ends up routing ciphertext it can't actually read. The architecture underneath doesn't change, but anything that depended on reading the plaintext, like search or smart reply suggestions, has to move onto the client instead.

Scaled up another ten times, to billions of connections, the gateway fleet grows accordingly, the message store and registry get sharded by conversation and user, the publish-subscribe bus gets partitioned, and presence leans even harder into aggressive in-memory sampling. A stale presence indicator is a reasonable price to pay at that scale. A lost message never is.

The Pattern Underneath It All

A chat system, at its core, is a durable log kept per conversation, read live as messages arrive and re-read whenever a client reconnects. Store the message first, then deliver it. Give each conversation its own sequence for ordering. Deduplicate using an id the client provides. And keep the durable message path entirely separate from the ephemeral routing and presence information sitting alongside it.

That same shape shows up anywhere real-time delivery and durability both matter at once: notifications, activity feeds, collaborative editing, order processing pipelines. A durable log underneath a live delivery layer, with a registry pointing live traffic at whichever connection actually needs it.