Capstone: Designing a Cloud Storage System for a Workload

• Given a realistic workload description, identify which of this discipline's systems (or which specific mechanism from one of them) is the right fit, and justify that choice in terms of the discipline's two opening axes: consistency versus availability, and throughput versus latency.
• Explain, for each design chosen, precisely which alternative system from this discipline would have been the wrong fit, and what specific guarantee that alternative would have sacrificed or failed to provide.
• Synthesize the discipline's central recurring lesson, that no single system studied here is universally correct, into a concrete, defensible design recommendation for three genuinely different workloads.

This discipline opened by arguing that Distributed Systems I's theory (consensus, replication, consistency models) is necessary but not sufficient, real engineering judgment about a specific workload's actual requirements is what determines which real, working system is the right one to build or adopt. Every concept since has studied one specific system, GFS, MapReduce, consistent hashing, Dynamo, Bigtable, Chubby, ZooKeeper, two-phase commit, Chain Replication, Spanner, Memcached, and serverless computing, as a case study in a deliberate, specific trade-off, never as a universally correct answer. This capstone is the discipline's payoff: given three genuinely different, realistic workloads, the task is to pick and justify a real design for each, using the specific vocabulary and trade-offs this discipline developed, and, just as importantly, to be explicit about which alternative designs would have been the wrong choice, and precisely why.

Workload 1: a shopping cart that must never lose a write

A shopping cart write (adding or removing an item) must always succeed, even during a network partition or a brief node outage, since a rejected write directly costs a customer's ability to complete a purchase, and an occasional stale read (briefly seeing a slightly outdated cart) is a far smaller cost than a rejected write. This is, almost by definition, the exact workload Dynamo was built for, and its own paper uses precisely this example. The right design choice is Dynamo's approach: consistent hashing to partition carts across nodes, sloppy quorums and hinted handoff so a write always succeeds even when some of a cart's designated replica nodes are briefly unreachable, and vector clocks to detect and surface genuinely concurrent writes (two devices adding different items to the same cart nearly simultaneously) so the application can perform a domain-correct merge, a union of items, rather than silently losing one write.

The wrong choice here would be Spanner's design: Spanner's commit wait and cross-shard two-phase commit are built specifically to guarantee external consistency, at the cost of a transaction sometimes not completing at all during a severe partition, exactly the "please try again later" outcome this workload's actual requirement (a write must always succeed) rules out. Using Spanner here would mean occasionally refusing a cart write specifically to protect a strong consistency guarantee this workload does not actually need, choosing the wrong point on the discipline's first axis for this specific requirement.

Workload 2: a global financial ledger that must never show two conflicting balances

A financial ledger has the opposite priority from Workload 1: two different views of an account's balance, visible to two different observers at the same moment, is a serious, unacceptable correctness failure (it could let the same funds be spent twice, from two different places, before either debit is visible to the other), while a small amount of added latency on each transaction is an acceptable, minor cost. This is exactly Spanner's target workload, and its own paper cites financial and similarly consistency-critical data as a primary use case. The right design choice is Spanner's approach: each shard of the ledger replicated via Paxos for fault tolerance, cross-shard transfers coordinated via two-phase commit with a Paxos-replicated coordinator (avoiding the plain two-phase-commit blocking problem a single, individually-crashable coordinator would introduce), and TrueTime's commit wait to guarantee external consistency, a transaction that completes before another begins is guaranteed a strictly earlier timestamp, everywhere in the globally distributed system.

The wrong choice here would be Dynamo's approach: Dynamo's sloppy quorums and vector-clock-surfaced conflicts are built specifically to let a write succeed even when replicas disagree, exactly the property that would let two conflicting balance updates both succeed independently, precisely the double-spend risk this workload cannot tolerate. Dynamo's own conflict-resolution model assumes the application can correctly merge concurrent versions (as with a shopping cart's union-of-items), but there is no domain-correct way to "merge" two conflicting account balances the way there is for a cart's contents, making Dynamo's whole conflict model a poor fit here, not merely a stylistic mismatch.

Workload 3: batch analytics over petabytes of logs, with a deadline but no live user waiting on any single operation

A nightly batch job that scans petabytes of log data to compute daily aggregate statistics has an entirely different shape from either workload above: no individual read or write is latency-sensitive (nothing is waiting synchronously on any single operation the way a live user waits on a cart write or a balance check), but the job as a whole needs to finish within some deadline (say, before the next business day), and the workload is overwhelmingly about aggregate throughput over huge sequential data, not any per-operation guarantee at all. This is exactly the workload GFS and MapReduce were built for: log data stored durably and cheaply in GFS, with its large 64MB chunks and relaxed consistency model matched to this data's append-heavy, rarely-randomly-accessed write pattern, and the actual aggregation computation expressed as MapReduce's map and reduce functions, with the runtime's locality-aware scheduling, deterministic re-execution on failure, and backup task execution for stragglers, absorbing the routine machine failures this discipline's opening concept established as the expected, steady-state condition at this scale.

The wrong choice here would be Chain Replication or Spanner: both are built to optimize strong consistency for individual, latency-sensitive operations, exactly the property this workload does not need (nothing here is a single, latency-sensitive read or write waiting on a strongly consistent answer), while GFS and MapReduce's throughput-first design, explicitly discussed on this discipline's opening throughput-versus-latency axis, is the actual bottleneck this workload cares about. Using Spanner's transactional, externally-consistent, per-operation-latency-paying design to process a petabyte-scale batch scan would add substantial, entirely unnecessary overhead for a guarantee (external consistency of individual operations) this workload never asked for.

Example 1: a fourth workload, worked through explicitly, to practice the same reasoning

Problem: A service needs a coordination mechanism to ensure exactly one instance of a background job runner is active at a time across a fleet of redundant instances, with automatic failover if the active instance crashes. Using this discipline's systems, identify the right design and justify it.

Reasoning: This is not a data-storage workload at all, it is a coordination problem, specifically leader election with automatic failover on crash, exactly the problem ZooKeeper (or Chubby, its more narrowly-scoped predecessor) is built for. The right design uses ZooKeeper's ephemeral, sequential znodes: each instance creates an ephemeral sequential znode under a shared path on startup, the instance holding the lowest-numbered znode is the active leader, and if that instance crashes, its ephemeral znode is automatically removed when its session expires, triggering the next-lowest instance (which had set a watch specifically on its immediate predecessor, per the leader-election pattern developed in the ZooKeeper concept) to become the new leader. Neither Dynamo nor Spanner nor GFS is built for this kind of problem at all, they solve data-storage and data-processing problems, not general-purpose coordination and leader election, confirming that the right tool here is not simply "whichever system in this discipline sounds the most sophisticated," but specifically the one built for this exact kind of problem.

Example 2: recognizing when a single workload actually needs to combine two of this discipline's systems

Problem: A social media service needs to store user posts (structured, sparse data, since posts have wildly varying optional fields like attached images, location tags, or poll options) at petabyte scale, with the underlying storage itself needing to be durable and fault-tolerant. Explain why this workload's answer draws on two of this discipline's systems together, rather than just one.

Resolution: The sparse, semi-structured data model, posts with wildly varying, sparse sets of optional fields, is exactly Bigtable's data model (a sparse, distributed, sorted map with column families, rather than a rigid, fully-populated schema), making Bigtable the right choice for how this data should be organized and queried. But Bigtable itself, as its own concept in this discipline develops, does not implement its own durable storage layer or its own replication and fault tolerance for actual byte data, it stores its SSTables and commit logs in GFS, relying on GFS's chunk replication for the actual durability of the underlying bytes, and relies on Chubby (or, in an open, non-Google deployment, a ZooKeeper-based equivalent) purely for master election and tablet-server liveness coordination. The correct answer to this workload is therefore not "pick exactly one system from this discipline," it is recognizing that Bigtable itself is already a composition of GFS (durable storage) and a coordination service (Chubby or its equivalent), precisely the layered architecture the Bigtable concepts in this discipline developed, rather than something to reinvent from scratch.

• "The 'best' system studied in this discipline is whichever one provides the strongest guarantees, so it is always the safest default choice." Spanner's external consistency is a strictly stronger guarantee than Dynamo's eventual consistency, but Workload 1 shows concretely why choosing the stronger guarantee is the wrong choice when the workload's actual requirement is availability, not consistency, a system's strongest available guarantee is not automatically the right one to reach for, the workload's actual requirements determine which guarantee is actually worth paying for.
• "Each of this discipline's systems solves a completely separate, non-overlapping problem, so there is always exactly one obviously correct answer for any given workload." Example 2 shows a realistic workload whose correct answer is a composition of two systems (Bigtable built on top of GFS and a coordination service), not a single, cleanly separate choice; recognizing when a workload's correct answer is itself a layered composition, rather than a single system picked off a list, is part of the actual skill this capstone is asking for.
• "Once a system is chosen for a workload, its specific parameters (like Dynamo's N, R, and W, or GFS's chunk size) do not need to be reconsidered, the system's default design is already fully specified." This discipline's concepts developed several tunable parameters precisely because real deployments adjust them for their specific situation, Dynamo's own paper describes different N, R, and W settings for different use cases even within Amazon, and a real design exercise should include reasoning about these parameters for the specific workload at hand, not just naming which system to use in the abstract.

This capstone applies the discipline's entire case-study sequence to three genuinely different, realistic workloads. A shopping cart that must never lose a write calls for Dynamo's availability-first design, consistent hashing, sloppy quorums, hinted handoff, and vector clocks, deliberately choosing availability over Spanner's stronger but availability-costing external consistency. A global financial ledger that must never show conflicting balances calls for the opposite choice, Spanner's Paxos-replicated shards, two-phase commit with a fault-tolerant coordinator, and TrueTime's commit wait, deliberately paying a small latency cost for a strong consistency guarantee Dynamo's design cannot provide. Batch analytics over petabytes of logs calls for neither of those transactional designs at all, but for GFS and MapReduce's throughput-first architecture, matched to a workload with no individual latency-sensitive operation but a real aggregate deadline. Across all three, and the two additional worked examples (coordination via ZooKeeper, and a layered Bigtable-on-GFS-and-Chubby design), the same lesson this discipline opened with holds: no single system studied here is the universally correct choice, each is a deliberate, well-reasoned answer to a specific workload's actual requirements, and real engineering judgment means matching the workload to the system, and being able to say precisely what would go wrong with any of the other choices.