Design a payment system

Most systems in this roadmap trade a little correctness for speed or availability: a stale feed, an approximate count. Payments don't get that option. Charging a customer twice, losing a payment, or being unable to explain where a sum of money went are each worse than being slow or briefly unavailable. This lesson designs the payment backend of an online marketplace that charges customers through a payment service provider (PSP) such as Stripe or Adyen. The scale is modest; the interesting part is making every step correct when networks time out and messages repeat.

#Requirements

Functional:

• Charge a customer for an order using a card, through a PSP.
• Handle extra authentication steps (3-D Secure) that redirect the customer to their bank.
• Issue full and partial refunds.
• Record every movement of money so balances for the marketplace and each seller can be explained to the cent.
• Pay out sellers' balances weekly (we'll focus on pay-ins).

Non-functional:

• Exactly-once charging. No customer is charged twice for one order, whatever retries happen.
• No lost payments. Every successful charge is recorded and leads to a fulfilled order or a refund.
• Auditability. Money records are immutable; corrections are new records, never edits.
• Correctness over availability. If we can't be sure, we stop and reconcile rather than guess.
• Minimal card-data exposure. Card numbers never reach our servers (keeping us out of most PCI DSS scope).

#Numbers

One million orders a day is about 12 payments per second on average, perhaps 100 per second during a sale. A single PostgreSQL primary handles that easily. Scale is not the design driver; failure handling is. That's a useful conclusion in itself: it means we can keep payments, idempotency records and the ledger in one database and use real transactions across them.

#The components

• Checkout service: creates the order and asks for a payment.
• Payment service: owns payments and their state machine, and talks to the PSP.
• PSP: charges the card. The customer's browser enters card details into the PSP's hosted fields, so our servers only ever see a token.
• Ledger: an append-only, double-entry record of every money movement.
• Reconciliation job: compares our records with the PSP's settlement reports every day.

#Idempotency, end to end

Every hop in that diagram can time out after the other side did the work. The defence is an idempotency key at every hop, so a retry is recognised and returns the original result instead of repeating the action. See idempotency keys.

Client → us. The browser generates a key per payment attempt and sends it in an Idempotency-Key header. We store the key with a hash of the request and, once finished, the response:

CREATE TABLE idempotency_keys (
  key           text PRIMARY KEY,
  request_hash  bytea NOT NULL,
  status        text NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code int,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now()
);

func (s *Server) withIdempotency(ctx context.Context, key string, reqHash []byte, do func() (int, []byte, error)) (int, []byte, error) {
	// Claim the key. Only one request can insert it.
	tag, err := s.db.Exec(ctx,
		`INSERT INTO idempotency_keys (key, request_hash, status) VALUES ($1, $2, 'in_progress')
		 ON CONFLICT (key) DO NOTHING`, key, reqHash)
	if err != nil {
		return 0, nil, err
	}
	if tag.RowsAffected() == 0 {
		var status string
		var hash, body []byte
		var code *int
		if err := s.db.QueryRow(ctx,
			`SELECT status, request_hash, response_code, response_body FROM idempotency_keys WHERE key = $1`, key,
		).Scan(&status, &hash, &code, &body); err != nil {
			return 0, nil, err
		}
		switch {
		case !bytes.Equal(hash, reqHash):
			return http.StatusUnprocessableEntity, []byte(`{"error":"key reused with a different request"}`), nil
		case status == "in_progress":
			return http.StatusConflict, []byte(`{"error":"request still in progress"}`), nil
		default:
			return *code, body, nil // replay the original response
		}
	}

	code, body, err := do()
	if err != nil {
		// Let the client retry: release the key so the next attempt runs again.
		s.db.Exec(ctx, `DELETE FROM idempotency_keys WHERE key = $1`, key)
		return 0, nil, err
	}
	_, err = s.db.Exec(ctx,
		`UPDATE idempotency_keys SET status = 'completed', response_code = $2, response_body = $3 WHERE key = $1`,
		key, code, body)
	return code, body, err
}

Releasing the key on error is only safe because the work inside do is itself safe to repeat — which brings us to the next hop. One gap to close in production: if the process crashes inside do, the key stays in_progress and every retry gets 409 forever. Store when the claim was made and let a later request take over a claim older than the longest the work can take.

Us → PSP. Every PSP call carries an idempotency key derived from our payment ID (for example pay_77), never a fresh random value per attempt. If our process crashes after the PSP charged the card and we retry with the same key, the PSP returns the existing charge instead of creating another. Major PSPs, including Stripe and Adyen, support this.

PSP → us. Webhooks arrive at least once and sometimes out of order. Verify each one's signature, store its event ID with a unique constraint and skip duplicates (idempotent consumers), and apply it through the state machine, which ignores transitions that don't make sense.

#The payment state machine

A payment is a small state machine, persisted, with every transition recorded:

created ──► processing ──► succeeded ──► (partially_)refunded
   │             │
   │             ├──► requires_action ──► processing
   │             │
   └─────────────┴──► failed
                 └──► unknown ──► (resolved by webhook or status query)

Transitions are conditional updates, so concurrent or replayed events can't move a payment backwards:

UPDATE payments SET state = 'succeeded', psp_charge_id = $2, updated_at = now()
WHERE id = $1 AND state IN ('processing', 'requires_action', 'unknown');
-- 0 rows updated → duplicate or out-of-order event: log and ignore.

#The unknown outcome

The most dangerous moment is a timeout calling the PSP. The charge may or may not have happened. Treating it as a failure and letting the customer "try again" with a new key is the classic cause of double charges. Instead:

1. Move the payment to unknown and tell the customer the payment is being confirmed.
2. Resolve it: retry the same request with the same idempotency key, query the PSP for the charge's status, and listen for the webhook.
3. Only when the PSP confirms there's no charge does the payment become failed, and only then may a new attempt start.

#A double-entry ledger

Payments say what happened; the ledger says where the money is. Use double-entry bookkeeping, which accountants have relied on for centuries because it catches errors structurally: every transaction consists of entries that sum to zero — money is never created or destroyed, only moved between accounts.

CREATE TABLE ledger_accounts (
  id       bigint PRIMARY KEY,
  name     text NOT NULL UNIQUE,     -- 'psp_clearing', 'seller:42:payable', 'fee_revenue'
  currency char(3) NOT NULL
);

CREATE TABLE ledger_transactions (
  id         bigint PRIMARY KEY,
  kind       text NOT NULL,          -- 'capture', 'refund', 'payout'
  payment_id bigint,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE ledger_entries (
  transaction_id bigint NOT NULL REFERENCES ledger_transactions,
  account_id     bigint NOT NULL REFERENCES ledger_accounts,
  amount_minor   bigint NOT NULL,    -- signed, in cents; never floating point
  PRIMARY KEY (transaction_id, account_id)
);

A customer pays $100.00 for a seller's item; the marketplace keeps a $3.00 fee:

The entries are inserted in the same database transaction as the payment's move to succeeded and an outbox event for the order service. Rules that keep the ledger trustworthy:

• Append only. A refund is a new transaction with reversed entries; nothing is updated or deleted.
• Balances are derived by summing entries (with periodic snapshots for speed), so they can always be recomputed and explained.
• Integer minor units per currency, with currency on the account; conversions are explicit transactions.
• Check the invariant continuously: SELECT transaction_id FROM ledger_entries GROUP BY transaction_id HAVING sum(amount_minor) <> 0 must return no rows.

See event sourcing and CQRS for the general pattern: an immutable record of events from which current state is derived.

#Coordinating the order: a saga

Paying for an order spans services: reserve inventory, charge the card, confirm the order, notify the seller. There's no transaction across them, and two-phase commit across a PSP isn't an option. Use a saga: a sequence of local steps, each with a compensating action if a later step fails. See sagas.

An orchestrator (often the checkout service) drives the steps, persists the saga's progress, and consumes events published through transactional outboxes so no step's completion is lost. If inventory can't be confirmed after a successful charge — the item sold out in the meantime — the saga issues an automatic refund rather than leaving the customer charged for nothing.

#Reconciliation

Even with all of the above, records drift: a webhook never arrives, a bug writes the wrong fee, a PSP reverses a charge after a dispute. Reconciliation catches it. Every day, download the PSP's settlement report and match it line by line against the ledger:

• Matched: amounts and IDs agree.
• Missing on our side: the PSP charged and we have no record. Create the payment, fulfil or refund, and find the bug.
• Missing on the PSP side: we recorded success the PSP doesn't know. Investigate before paying out the seller.
• Amount mismatch: fees or currency conversion differ; correct with an adjusting ledger transaction.

Mismatches go to a queue for people to review, and seller payouts wait until the period reconciles. This is the "correctness over availability" requirement in practice: when in doubt, stop the money and look.

#Practice

After a deploy, support receives reports of customers charged twice for one order. Logs show that during a PSP slowdown, calls to create charges timed out after 5 seconds; the checkout page then showed "Payment failed, try again"; customers clicked again; and the payment service created a new charge each time, with a new random idempotency key per call. Some of the timed-out charges had actually succeeded.

(1) Explain exactly how the double charges happened. (2) List the fixes, at every hop. (3) Write the ledger entries for refunding $40.00 of the $100.00 order above as a partial refund, where the marketplace returns its fee proportionally.

#Key takeaways

• In payments, failure handling — not scale — drives the design; keep payments, idempotency and the ledger in one transactional database.
• Use idempotency keys at every hop, and derive the PSP's key from your payment ID so retries can never create a second charge.
• Model payments as a persisted state machine with conditional transitions; treat timeouts as unknown, not failed.
• Record money in an append-only double-entry ledger in integer minor units, where every transaction sums to zero.
• Coordinate multi-service checkout with a saga and compensations, driven by outbox events.
• Reconcile against the PSP daily, and hold payouts until the books agree.