Design a file sync service

A file sync service keeps a folder identical across a user's laptop, phone and the web, and shares parts of it with other people. Edit a file on one machine and it appears on the others within seconds, even when the file is 4 GB and only one paragraph changed. This lesson designs the core of such a service, and the ideas (content-addressed chunks, a metadata journal and cursor-based sync) show up anywhere data must be kept in sync across devices.

#Requirements

Functional:

• Upload, download, update, delete and move files and folders; changes sync to all of a user's devices.
• Files up to 50 GB. Uploads and downloads resume after interruption.
• Sharing folders with other users, with view or edit permission.
• Version history: restore any version from the last 30 days.
• Offline edits sync when the device reconnects; concurrent edits never silently lose data.

Non-functional:

• Durability above all: once the service says a file is saved, it must never be lost. Eleven nines is the usual target for the stored bytes.
• Sync latency of a few seconds for small changes between online devices.
• Bandwidth efficiency: changing a small part of a large file shouldn't re-upload the whole file.
• Strong consistency of the metadata (what files exist, which version is current), even though the bytes can be eventually consistent.

#Numbers

users:                    100 million, 3 devices each
stored data:              average 20 GB per user → 2 EB logical; after deduplication ~1 EB
file changes:             each user changes ~20 files/day → 2 billion changes/day ≈ 23,000/s avg
changed bytes:            most changes are small edits; ~1 PB/day of new chunks uploaded
metadata:                 ~1,000 files per user → 100 billion file records

Two very different workloads appear. Bytes: an exabyte of immutable data written once and read occasionally, which is exactly what object storage is for. Metadata: 100 billion small records that are read and updated constantly and must be strongly consistent. The central design decision is to keep these two apart.

#Chunks: the unit of storage and transfer

Files are split into chunks (blocks), each identified by the hash of its contents, for example SHA-256. A file version is then just an ordered list of chunk hashes:

report.pdf, version 7 = [a3f1…, 9bc4…, 77e0…, 1d2a…]

This one idea gives you several properties at once:

• Delta sync: edit one part of the file, and only the chunks that changed have new hashes. The client uploads those and nothing else.
• Deduplication: identical chunks have identical hashes, so the same chunk is stored once, whether it appears twice in one file, in two versions of a file, or in two users' copies of the same video.
• Resumable transfers: an interrupted 50 GB upload resumes from the first chunk the server doesn't have.
• Integrity: the client and server can verify every chunk against its hash.

Fixed-size chunks (say 4 MB) have a weakness: insert one byte at the start of a file, and every chunk boundary shifts, so every chunk changes. Content-defined chunking fixes this by choosing boundaries where a rolling hash of the last few dozen bytes matches a pattern (for example, its low 22 bits are zero, giving roughly 4 MB average chunks). Boundaries then move with the content, and an insertion only changes the chunks around it.

#Uploading a change

Clients talk to two services: a metadata service (what exists) and object storage (the bytes), with uploads going directly to storage via presigned URLs so the servers never proxy file data.

Order matters: chunks are uploaded before the commit, and the metadata service verifies that every referenced chunk exists before committing. So the metadata can never point at bytes that aren't stored. If the client dies halfway, the uploaded chunks are orphans that nothing references, and a garbage collector deletes them later.

The commit includes the base version the client edited (v7). If the current version is still v7, the commit succeeds. If another device committed v8 in the meantime, it's a conflict, handled below. This is optimistic concurrency control.

#Metadata: a journal per namespace

The metadata service is the heart of the system. Model it around a namespace: a user's root folder, or a shared folder. Every change in a namespace is appended to that namespace's journal with a sequence number:

journal(namespace_id, seq, path, file_id, version, chunk_list | deleted, modified_by, at)
files(namespace_id, file_id, path, current_version, size, ...)   -- current state, derived from the journal

The journal is the source of truth and gives you sync almost for free: a device remembers the last sequence number it has applied, its cursor, and asks "give me everything in namespace 88 after seq 50190". Version history is the journal too.

Storage: 100 billion files and constant updates need a sharded database. Shard by namespace ID, so every operation on one namespace (commits, listing a folder, reading its journal) touches one shard, and transactions within a namespace stay simple and strongly consistent. A relational database sharded this way (or a distributed SQL database) works well, because commits need conditional writes ("only if current version is v7") and listing needs ordered queries.

A very large shared namespace, such as a company-wide folder with millions of files and thousands of editors, becomes a hot partition. Large organisations get dedicated shards, and the journal can be split per top-level folder inside the namespace.

#Telling devices something changed

Devices need to learn about new journal entries quickly without polling every few seconds. The usual pattern is a notification channel that carries no data, only "namespace 88 has changed":

1. Each device holds a long-lived connection (a long-poll HTTP request or a WebSocket) to a notification service, subscribing to its namespaces.
2. When a commit succeeds, the metadata service publishes namespace 88 → seq 50211 to the notification service.
3. The notification service tells subscribed devices, which then call the metadata service with their cursor to fetch the actual changes, then download missing chunks.

Keeping notifications data-free makes them simple and safe to lose: if a notification is dropped or a device was offline, it catches up with its cursor on reconnect. Devices also do a cursor check every few minutes as a safety net.

#Conflicts

Two devices edit the same file while one of them is offline. When the offline device reconnects, its commit says "base v7", but the current version is v8.

Merging arbitrary files automatically isn't possible: the service doesn't understand the contents of a spreadsheet or a Photoshop file. The safe, standard behaviour is:

• The first commit wins and becomes v8.
• The losing commit is saved as a new file next to it: report (Ana's conflicted copy 2026-11-02).pdf.
• Both users see both files. No data is lost, and a human decides how to merge.

Folder operations need similar rules: a file edited on one device while its folder was deleted on another is restored rather than deleted, because losing an edit is worse than an unwanted folder.

Collaborative editors such as Google Docs avoid file-level conflicts entirely by syncing operations on a structured document (with techniques such as operational transformation or CRDTs). That's a different system, built for one file type.

#Durability, versions and deletion

• Bytes: object storage replicates or erasure-codes each chunk across failure domains; store chunks in at least two regions for disaster recovery (RPO and RTO).
• Metadata: the journal is replicated synchronously within a region and backed up continuously. Losing metadata is worse than losing bytes: without it, the bytes are an unlabelled pile of chunks.
• Version history and trash: deleting a file appends a "deleted" entry; the old versions' chunk lists stay in the journal for 30 days.
• Garbage collection: a chunk can be deleted only when no version anywhere references it. Track reference counts (or periodically mark reachable chunks from all live versions), and delete unreferenced chunks after a grace period, so a slow in-flight commit can't reference a chunk that was just collected.

#Practice

Users report two problems: (1) syncing a folder of 200,000 small files (a code repository with node_modules) takes hours, and (2) a user shares a 30 GB video folder with a team of 500 people, and when they all open it, the office internet connection saturates.

Propose design changes for each.

#Key takeaways

• Separate bytes (immutable chunks in object storage) from metadata (a strongly consistent, sharded database).
• Content-addressed chunks give delta sync, deduplication, resumable transfers and integrity checks; content-defined boundaries keep small edits small.
• Upload chunks first, then commit a version that references them, conditional on the base version.
• A per-namespace journal with cursors makes sync, catch-up after offline periods and version history the same mechanism.
• Resolve conflicts by keeping both versions ("conflicted copy"), and garbage-collect chunks only when nothing references them.