# Design a payment system > Design checkout payments where correctness beats speed: end-to-end idempotency, a payment state machine, unknown outcomes, webhooks, a double-entry ledger, sagas and daily reconciliation. System Design roadmap · Case studies · 22 min read · https://levelcode.org/roadmaps/system-design/design-payment-system Most systems in this roadmap trade a little correctness for speed or availability: a stale feed, an approximate count. Payments don't get that option. Charging a customer twice, losing a payment, or being unable to explain where a sum of money went are each worse than being slow or briefly unavailable. This lesson designs the payment backend of an online marketplace that charges customers through a payment service provider (PSP) such as Stripe or Adyen. The scale is modest; the interesting part is making every step correct when networks time out and messages repeat. ## Requirements **Functional:** - Charge a customer for an order using a card, through a PSP. - Handle extra authentication steps (3-D Secure) that redirect the customer to their bank. - Issue full and partial refunds. - Record every movement of money so balances for the marketplace and each seller can be explained to the cent. - Pay out sellers' balances weekly (we'll focus on pay-ins). **Non-functional:** - **Exactly-once charging.** No customer is charged twice for one order, whatever retries happen. - **No lost payments.** Every successful charge is recorded and leads to a fulfilled order or a refund. - **Auditability.** Money records are immutable; corrections are new records, never edits. - **Correctness over availability.** If we can't be sure, we stop and reconcile rather than guess. - **Minimal card-data exposure.** Card numbers never reach our servers (keeping us out of most PCI DSS scope). ## Numbers One million orders a day is about 12 payments per second on average, perhaps 100 per second during a sale. A single PostgreSQL primary handles that easily. **Scale is not the design driver; failure handling is.** That's a useful conclusion in itself: it means we can keep payments, idempotency records and the ledger in one database and use real transactions across them. ## The components - **Checkout service:** creates the order and asks for a payment. - **Payment service:** owns payments and their state machine, and talks to the PSP. - **PSP:** charges the card. The customer's browser enters card details into the PSP's hosted fields, so our servers only ever see a token. - **Ledger:** an append-only, double-entry record of every money movement. - **Reconciliation job:** compares our records with the PSP's settlement reports every day. ```sequence title: Paying for an order participants: Browser, Checkout, Payment service, Database, PSP Browser -> Checkout: POST /orders/981/pay (Idempotency-Key k1) Checkout -> Payment service: create payment for order 981 (key k1) Payment service -> Database: INSERT payment (state: created) Payment service -> PSP: create charge, idempotency key pay_77 PSP --> Payment service: requires_action (3-D Secure) Payment service --> Browser: redirect to bank challenge Browser -> PSP: customer completes 3-D Secure PSP -> Payment service: webhook charge.succeeded (pay_77) Payment service -> Database: succeeded + ledger + outbox (one tx) > The state change, the balanced ledger entries and the event for the order service commit together. ``` ## Idempotency, end to end Every hop in that diagram can time out *after* the other side did the work. The defence is an idempotency key at **every** hop, so a retry is recognised and returns the original result instead of repeating the action. See [idempotency keys](https://levelcode.org/concepts/idempotency-keys). **Client → us.** The browser generates a key per payment attempt and sends it in an `Idempotency-Key` header. We store the key with a hash of the request and, once finished, the response: ```sql CREATE TABLE idempotency_keys ( key text PRIMARY KEY, request_hash bytea NOT NULL, status text NOT NULL CHECK (status IN ('in_progress', 'completed')), response_code int, response_body jsonb, created_at timestamptz NOT NULL DEFAULT now() ); ``` ```go func (s *Server) withIdempotency(ctx context.Context, key string, reqHash []byte, do func() (int, []byte, error)) (int, []byte, error) { // Claim the key. Only one request can insert it. tag, err := s.db.Exec(ctx, `INSERT INTO idempotency_keys (key, request_hash, status) VALUES ($1, $2, 'in_progress') ON CONFLICT (key) DO NOTHING`, key, reqHash) if err != nil { return 0, nil, err } if tag.RowsAffected() == 0 { var status string var hash, body []byte var code *int if err := s.db.QueryRow(ctx, `SELECT status, request_hash, response_code, response_body FROM idempotency_keys WHERE key = $1`, key, ).Scan(&status, &hash, &code, &body); err != nil { return 0, nil, err } switch { case !bytes.Equal(hash, reqHash): return http.StatusUnprocessableEntity, []byte(`{"error":"key reused with a different request"}`), nil case status == "in_progress": return http.StatusConflict, []byte(`{"error":"request still in progress"}`), nil default: return *code, body, nil // replay the original response } } code, body, err := do() if err != nil { // Let the client retry: release the key so the next attempt runs again. s.db.Exec(ctx, `DELETE FROM idempotency_keys WHERE key = $1`, key) return 0, nil, err } _, err = s.db.Exec(ctx, `UPDATE idempotency_keys SET status = 'completed', response_code = $2, response_body = $3 WHERE key = $1`, key, code, body) return code, body, err } ``` Releasing the key on error is only safe because the work inside `do` is itself safe to repeat — which brings us to the next hop. One gap to close in production: if the process crashes inside `do`, the key stays `in_progress` and every retry gets `409` forever. Store when the claim was made and let a later request take over a claim older than the longest the work can take. **Us → PSP.** Every PSP call carries an idempotency key derived from *our* payment ID (for example `pay_77`), never a fresh random value per attempt. If our process crashes after the PSP charged the card and we retry with the same key, the PSP returns the existing charge instead of creating another. Major PSPs, including Stripe and Adyen, support this. **PSP → us.** Webhooks arrive at least once and sometimes out of order. Verify each one's signature, store its event ID with a unique constraint and skip duplicates ([idempotent consumers](https://levelcode.org/concepts/idempotent-consumers)), and apply it through the state machine, which ignores transitions that don't make sense. ## The payment state machine A payment is a small state machine, persisted, with every transition recorded: ```text created ──► processing ──► succeeded ──► (partially_)refunded │ │ │ ├──► requires_action ──► processing │ │ └─────────────┴──► failed └──► unknown ──► (resolved by webhook or status query) ``` Transitions are **conditional updates**, so concurrent or replayed events can't move a payment backwards: ```sql UPDATE payments SET state = 'succeeded', psp_charge_id = $2, updated_at = now() WHERE id = $1 AND state IN ('processing', 'requires_action', 'unknown'); -- 0 rows updated → duplicate or out-of-order event: log and ignore. ``` ### The unknown outcome The most dangerous moment is a **timeout calling the PSP**. The charge may or may not have happened. Treating it as a failure and letting the customer "try again" with a new key is the classic cause of double charges. Instead: 1. Move the payment to `unknown` and tell the customer the payment is being confirmed. 2. Resolve it: retry the same request with the **same** idempotency key, query the PSP for the charge's status, and listen for the webhook. 3. Only when the PSP confirms there's no charge does the payment become `failed`, and only then may a new attempt start. ## A double-entry ledger Payments say what *happened*; the **ledger** says where the *money* is. Use double-entry bookkeeping, which accountants have relied on for centuries because it catches errors structurally: every transaction consists of entries that **sum to zero** — money is never created or destroyed, only moved between accounts. ```sql CREATE TABLE ledger_accounts ( id bigint PRIMARY KEY, name text NOT NULL UNIQUE, -- 'psp_clearing', 'seller:42:payable', 'fee_revenue' currency char(3) NOT NULL ); CREATE TABLE ledger_transactions ( id bigint PRIMARY KEY, kind text NOT NULL, -- 'capture', 'refund', 'payout' payment_id bigint, created_at timestamptz NOT NULL DEFAULT now() ); CREATE TABLE ledger_entries ( transaction_id bigint NOT NULL REFERENCES ledger_transactions, account_id bigint NOT NULL REFERENCES ledger_accounts, amount_minor bigint NOT NULL, -- signed, in cents; never floating point PRIMARY KEY (transaction_id, account_id) ); ``` A customer pays $100.00 for a seller's item; the marketplace keeps a $3.00 fee: | Account | Amount | |---|---| | `psp_clearing` (money held for us at the PSP) | +10000 | | `seller:42:payable` (what we owe the seller) | −9700 | | `fee_revenue` | −300 | | **Sum** | **0** | The entries are inserted in the **same database transaction** as the payment's move to `succeeded` and an outbox event for the order service. Rules that keep the ledger trustworthy: - **Append only.** A refund is a new transaction with reversed entries; nothing is updated or deleted. - **Balances are derived** by summing entries (with periodic snapshots for speed), so they can always be recomputed and explained. - **Integer minor units** per currency, with currency on the account; conversions are explicit transactions. - **Check the invariant** continuously: `SELECT transaction_id FROM ledger_entries GROUP BY transaction_id HAVING sum(amount_minor) <> 0` must return no rows. See [event sourcing and CQRS](https://levelcode.org/concepts/event-sourcing-cqrs) for the general pattern: an immutable record of events from which current state is derived. ## Coordinating the order: a saga Paying for an order spans services: reserve inventory, charge the card, confirm the order, notify the seller. There's no transaction across them, and [two-phase commit](https://levelcode.org/concepts/two-phase-commit) across a PSP isn't an option. Use a **saga**: a sequence of local steps, each with a **compensating action** if a later step fails. See [sagas](https://levelcode.org/concepts/saga-pattern). | Step | Compensation | |---|---| | Reserve inventory (with an expiry) | Release the reservation | | Charge the card | Refund the charge | | Confirm the order | Cancel the order and notify the customer | An orchestrator (often the checkout service) drives the steps, persists the saga's progress, and consumes events published through [transactional outboxes](https://levelcode.org/concepts/transactional-outbox) so no step's completion is lost. If inventory can't be confirmed after a successful charge — the item sold out in the meantime — the saga issues an automatic refund rather than leaving the customer charged for nothing. ## Reconciliation Even with all of the above, records drift: a webhook never arrives, a bug writes the wrong fee, a PSP reverses a charge after a dispute. **Reconciliation** catches it. Every day, download the PSP's settlement report and match it line by line against the ledger: - **Matched:** amounts and IDs agree. - **Missing on our side:** the PSP charged and we have no record. Create the payment, fulfil or refund, and find the bug. - **Missing on the PSP side:** we recorded success the PSP doesn't know. Investigate before paying out the seller. - **Amount mismatch:** fees or currency conversion differ; correct with an adjusting ledger transaction. Mismatches go to a queue for people to review, and seller payouts wait until the period reconciles. This is the "correctness over availability" requirement in practice: when in doubt, stop the money and look. ## Practice After a deploy, support receives reports of customers charged twice for one order. Logs show that during a PSP slowdown, calls to create charges timed out after 5 seconds; the checkout page then showed "Payment failed, try again"; customers clicked again; and the payment service created a new charge each time, with a new random idempotency key per call. Some of the timed-out charges had actually succeeded. (1) Explain exactly how the double charges happened. (2) List the fixes, at every hop. (3) Write the ledger entries for refunding $40.00 of the $100.00 order above as a partial refund, where the marketplace returns its fee proportionally. **Solution** **(1) How it happened.** The PSP received the first request and charged the card, but answered after our 5-second timeout. We treated the timeout as a failure, so the UI invited a retry. The retry used a new idempotency key, so the PSP saw a brand-new request and charged again. Nothing on our side recorded that the first outcome was unknown rather than failed. **(2) Fixes.** - *PSP hop:* derive the PSP idempotency key from our payment ID, so every retry of the same payment uses the same key and the PSP returns the original charge. - *Unknown outcomes:* on timeout, set the payment to `unknown`, show "We're confirming your payment", and resolve it by retrying with the same key, querying the PSP's status, and handling the webhook. Never allow a new attempt until the old one is known to have failed. - *Client hop:* the browser sends an `Idempotency-Key` per payment attempt, and the payment endpoint stores and replays responses, so a double click or a page retry returns the first result. - *Order level:* allow at most one active payment per order (a partial unique index on `payments (order_id) WHERE state NOT IN ('failed')`), so even a bug can't start two concurrent charges for one order. - *Clean up:* use reconciliation to find all orders with two successful charges, refund the duplicates automatically, and add an alert on this invariant. **(3) Partial refund of $40.00, fee returned proportionally (3% of $40 = $1.20).** A new `refund` transaction, with entries summing to zero: | Account | Amount | |---|---| | `psp_clearing` | −4000 | | `seller:42:payable` | +3880 | | `fee_revenue` | +120 | The seller's payable balance drops by $38.80 and the marketplace gives back $1.20 of its fee, together funding the $40.00 that leaves the PSP balance to the customer. The original capture transaction is untouched, and the payment moves to `partially_refunded`. ## Key takeaways - In payments, failure handling — not scale — drives the design; keep payments, idempotency and the ledger in one transactional database. - Use idempotency keys at every hop, and derive the PSP's key from your payment ID so retries can never create a second charge. - Model payments as a persisted state machine with conditional transitions; treat timeouts as unknown, not failed. - Record money in an append-only double-entry ledger in integer minor units, where every transaction sums to zero. - Coordinate multi-service checkout with a saga and compensations, driven by outbox events. - Reconcile against the PSP daily, and hold payouts until the books agree. ## Quiz 1. A call to the PSP to create a charge times out. What should the payment service do? A. Mark the payment failed and let the customer try again with a new attempt B. Mark it unknown, then resolve it by retrying with the same idempotency key, querying the PSP, or waiting for the webhook C. Immediately issue a refund D. Create a new charge with a fresh idempotency key **Answer: B.** A timeout means the outcome is unknown: the charge may have succeeded. Treating it as a failure and retrying with a new key is how double charges happen. 2. In a double-entry ledger, what must be true of every transaction's entries? A. They must all be positive B. They must sum to zero C. There must be exactly two of them D. They must be in the same account **Answer: B.** Money moves between accounts but is never created or destroyed, so debits and credits balance. A transaction that doesn't sum to zero signals a bug immediately. 3. Why are refunds recorded as new ledger transactions instead of editing the original entries? A. Databases can't update ledger rows B. An append-only ledger keeps a complete, auditable history and lets every balance be recomputed and explained C. Editing would make refunds faster D. It allows floating-point amounts **Answer: B.** Immutability means the history of what happened — the charge and then its refund — is preserved for audits and reconciliation. Corrections are additional transactions, never edits. 4. The card was charged, but the item sold out before the order was confirmed. In a saga, what happens? A. The whole operation is rolled back by a distributed transaction B. The compensating action for the charge runs: the customer is refunded and the order cancelled C. The order is confirmed anyway and the seller must find stock D. Nothing; the customer must contact support **Answer: B.** Sagas replace a cross-service transaction with local steps plus compensations. A failure after a successful charge triggers the refund, leaving the system consistent.