# Design a file sync service > Design Dropbox-style file sync: chunking and deduplication, uploads to object storage, a metadata journal, change notification, conflicts and versions. System Design roadmap · Case studies · 20 min read · https://levelcode.org/roadmaps/system-design/design-file-sync A file sync service keeps a folder identical across a user's laptop, phone and the web, and shares parts of it with other people. Edit a file on one machine and it appears on the others within seconds, even when the file is 4 GB and only one paragraph changed. This lesson designs the core of such a service, and the ideas (content-addressed chunks, a metadata journal and cursor-based sync) show up anywhere data must be kept in sync across devices. ## Requirements **Functional:** - Upload, download, update, delete and move files and folders; changes sync to all of a user's devices. - Files up to 50 GB. Uploads and downloads resume after interruption. - Sharing folders with other users, with view or edit permission. - Version history: restore any version from the last 30 days. - Offline edits sync when the device reconnects; concurrent edits never silently lose data. **Non-functional:** - **Durability above all**: once the service says a file is saved, it must never be lost. Eleven nines is the usual target for the stored bytes. - Sync latency of a few seconds for small changes between online devices. - Bandwidth efficiency: changing a small part of a large file shouldn't re-upload the whole file. - Strong consistency of the **metadata** (what files exist, which version is current), even though the bytes can be eventually consistent. ## Numbers ```text users: 100 million, 3 devices each stored data: average 20 GB per user → 2 EB logical; after deduplication ~1 EB file changes: each user changes ~20 files/day → 2 billion changes/day ≈ 23,000/s avg changed bytes: most changes are small edits; ~1 PB/day of new chunks uploaded metadata: ~1,000 files per user → 100 billion file records ``` Two very different workloads appear. **Bytes**: an exabyte of immutable data written once and read occasionally, which is exactly what [object storage](https://levelcode.org/concepts/object-storage) is for. **Metadata**: 100 billion small records that are read and updated constantly and must be strongly consistent. The central design decision is to keep these two apart. ## Chunks: the unit of storage and transfer Files are split into **chunks** (blocks), each identified by the hash of its contents, for example SHA-256. A file version is then just an ordered list of chunk hashes: ```text report.pdf, version 7 = [a3f1…, 9bc4…, 77e0…, 1d2a…] ``` This one idea gives you several properties at once: - **Delta sync**: edit one part of the file, and only the chunks that changed have new hashes. The client uploads those and nothing else. - **Deduplication**: identical chunks have identical hashes, so the same chunk is stored once, whether it appears twice in one file, in two versions of a file, or in two users' copies of the same video. - **Resumable transfers**: an interrupted 50 GB upload resumes from the first chunk the server doesn't have. - **Integrity**: the client and server can verify every chunk against its hash. Fixed-size chunks (say 4 MB) have a weakness: insert one byte at the start of a file, and every chunk boundary shifts, so every chunk changes. **Content-defined chunking** fixes this by choosing boundaries where a rolling hash of the last few dozen bytes matches a pattern (for example, its low 22 bits are zero, giving roughly 4 MB average chunks). Boundaries then move with the content, and an insertion only changes the chunks around it. > [!NOTE] > Cross-user deduplication has a privacy cost: if uploading a chunk is instant because it already exists, an attacker can test whether *someone* has a specific file. Services either deduplicate only within a user (or organisation), or always make the client upload the bytes regardless. ## Uploading a change Clients talk to two services: a **metadata service** (what exists) and **object storage** (the bytes), with uploads going directly to storage via presigned URLs so the servers never proxy file data. ```sequence title: Saving an edited file participants: Client, Metadata service, Object storage Client -> Client: Chunk file, hash each chunk Client -> Metadata service: Which of [a3f1, 9bc4, 77e0, 1d2a] do you have? Metadata service --> Client: Missing: [9bc4], with a presigned upload URL > Only the changed chunk is transferred. Client -> Object storage: PUT chunk 9bc4 Object storage --> Client: 200 OK Client -> Metadata service: Commit report.pdf v8 = [a3f1, 9bc4, 77e0, 1d2a], base v7 Metadata service --> Client: Committed as journal entry 50211 ``` Order matters: chunks are uploaded **before** the commit, and the metadata service verifies that every referenced chunk exists before committing. So the metadata can never point at bytes that aren't stored. If the client dies halfway, the uploaded chunks are orphans that nothing references, and a garbage collector deletes them later. The commit includes the **base version** the client edited (v7). If the current version is still v7, the commit succeeds. If another device committed v8 in the meantime, it's a conflict, handled below. This is [optimistic concurrency control](https://levelcode.org/concepts/optimistic-vs-pessimistic-locking). ## Metadata: a journal per namespace The metadata service is the heart of the system. Model it around a **namespace**: a user's root folder, or a shared folder. Every change in a namespace is appended to that namespace's **journal** with a sequence number: ```text journal(namespace_id, seq, path, file_id, version, chunk_list | deleted, modified_by, at) files(namespace_id, file_id, path, current_version, size, ...) -- current state, derived from the journal ``` The journal is the source of truth and gives you sync almost for free: a device remembers the last sequence number it has applied, its **cursor**, and asks "give me everything in namespace 88 after seq 50190". Version history is the journal too. Storage: 100 billion files and constant updates need a sharded database. Shard by **namespace ID**, so every operation on one namespace (commits, listing a folder, reading its journal) touches one shard, and transactions within a namespace stay simple and strongly consistent. A relational database sharded this way (or a distributed SQL database) works well, because commits need conditional writes ("only if current version is v7") and listing needs ordered queries. A very large shared namespace, such as a company-wide folder with millions of files and thousands of editors, becomes a [hot partition](https://levelcode.org/concepts/hot-partitions). Large organisations get dedicated shards, and the journal can be split per top-level folder inside the namespace. ## Telling devices something changed Devices need to learn about new journal entries quickly without polling every few seconds. The usual pattern is a **notification channel** that carries no data, only "namespace 88 has changed": 1. Each device holds a long-lived connection (a long-poll HTTP request or a WebSocket) to a notification service, subscribing to its namespaces. 2. When a commit succeeds, the metadata service publishes `namespace 88 → seq 50211` to the notification service. 3. The notification service tells subscribed devices, which then call the metadata service with their cursor to fetch the actual changes, then download missing chunks. Keeping notifications data-free makes them simple and safe to lose: if a notification is dropped or a device was offline, it catches up with its cursor on reconnect. Devices also do a cursor check every few minutes as a safety net. ## Conflicts Two devices edit the same file while one of them is offline. When the offline device reconnects, its commit says "base v7", but the current version is v8. Merging arbitrary files automatically isn't possible: the service doesn't understand the contents of a spreadsheet or a Photoshop file. The safe, standard behaviour is: - The first commit wins and becomes v8. - The losing commit is saved as a new file next to it: `report (Ana's conflicted copy 2026-11-02).pdf`. - Both users see both files. **No data is lost**, and a human decides how to merge. Folder operations need similar rules: a file edited on one device while its folder was deleted on another is restored rather than deleted, because losing an edit is worse than an unwanted folder. Collaborative editors such as Google Docs avoid file-level conflicts entirely by syncing *operations* on a structured document (with techniques such as operational transformation or CRDTs). That's a different system, built for one file type. ## Durability, versions and deletion - **Bytes**: object storage replicates or erasure-codes each chunk across failure domains; store chunks in at least two regions for disaster recovery ([RPO and RTO](https://levelcode.org/concepts/rpo-rto)). - **Metadata**: the journal is replicated synchronously within a region and backed up continuously. Losing metadata is worse than losing bytes: without it, the bytes are an unlabelled pile of chunks. - **Version history and trash**: deleting a file appends a "deleted" entry; the old versions' chunk lists stay in the journal for 30 days. - **Garbage collection**: a chunk can be deleted only when no version anywhere references it. Track reference counts (or periodically mark reachable chunks from all live versions), and delete unreferenced chunks after a grace period, so a slow in-flight commit can't reference a chunk that was just collected. ## Practice Users report two problems: (1) syncing a folder of 200,000 small files (a code repository with `node_modules`) takes hours, and (2) a user shares a 30 GB video folder with a team of 500 people, and when they all open it, the office internet connection saturates. Propose design changes for each. **Solution** **1. Many small files.** - The cost is per file, not per byte: one "which chunks do you have?" request, one upload and one commit per file, each with a network round trip. Batch at every step: hash many files locally, ask about thousands of chunk hashes in one request, upload small chunks in batches (or pack many small files into one upload), and **commit many files in one journal transaction**. - Downloading in the other direction, the cursor fetch should return changes in large pages, and the client should pipeline chunk downloads with high concurrency rather than one at a time. - Many tools (and users) prefer that some directories don't sync at all: support ignore rules (for example a `.syncignore` file, with sensible defaults for `node_modules` and build outputs). **2. Many devices downloading the same large files on one network.** - The chunks are content-addressed, so a device can fetch a chunk from **any source that has it** and verify it by hash. Add **LAN sync**: clients on the same network discover each other and fetch chunks from peers before going to the internet. The first device downloads from the cloud, the other 499 mostly get it locally. Hash verification means a peer can't serve corrupted or malicious data. - Serve chunks through a **CDN** with long cache lifetimes: content-addressed chunks never change, so they can be cached forever. - On the client, **selective sync** and on-demand files (placeholders that download when opened) avoid pulling 30 GB to every laptop that will never play the videos. ## Key takeaways - Separate bytes (immutable chunks in object storage) from metadata (a strongly consistent, sharded database). - Content-addressed chunks give delta sync, deduplication, resumable transfers and integrity checks; content-defined boundaries keep small edits small. - Upload chunks first, then commit a version that references them, conditional on the base version. - A per-namespace journal with cursors makes sync, catch-up after offline periods and version history the same mechanism. - Resolve conflicts by keeping both versions ("conflicted copy"), and garbage-collect chunks only when nothing references them. ## Quiz 1. Why does content-defined chunking beat fixed-size chunks for delta sync? A. Its chunks are smaller B. Inserting bytes near the start of a file doesn't shift every later boundary, so only nearby chunks change C. It doesn't need hashes D. It encrypts each chunk **Answer: B.** With fixed offsets, one inserted byte changes every following chunk. Boundaries chosen from the content itself move with the data, so only chunks around the edit get new hashes. 2. Why are chunks uploaded before the metadata commit, rather than after? A. So committed metadata never references bytes that aren't stored; a failed upload only leaves orphan chunks to garbage-collect B. Object storage requires it C. It makes uploads faster D. So conflicts can't happen **Answer: A.** If metadata were committed first, other devices could see a version whose bytes don't exist yet, or never will. Orphan chunks are harmless and cleaned up later. 3. A laptop was offline for a week. How does it catch up? A. It re-downloads every file B. It waits for the next notification C. It asks the metadata service for all journal entries after its stored cursor for each namespace D. It compares file sizes with the server **Answer: C.** The journal is ordered by sequence number, and the device's cursor records where it stopped. Notifications only say that something changed; the cursor is how a device finds out what. 4. Two devices edit the same spreadsheet offline. What does a file sync service typically do when both sync? A. Keep the larger file B. Merge the cells automatically C. Reject the second device's edit and discard it D. Keep the first commit as the current version and save the other as a conflicted copy **Answer: D.** The service can't safely merge arbitrary file formats, and it must never lose data. Saving the loser as a conflicted copy keeps both and lets a person decide.