Computer Science › Specialization
The theory of consensus, replication, and consistency from Distributed Systems I is necessary but not sufficient: real systems serving billions of requests a day are built by combining those primitives into specific, opinionated designs, each trading correctness, availability, and performance differently for a specific workload.
A function-as-a-service platform runs a stateless handler on demand, scaling from zero to thousands of concurrent invocations and billing per invocation, a genuine step forward for autoscaling, but one that a systems-level critique shows is disappointingly stateless and I/O-poor exactly where the case studies in this discipline (data-parallel jobs, distributed storage) need state and fast networking most.
Mapping both nodes and keys onto the same hash ring, and assigning each key to the next node clockwise, means adding or removing one node only reshuffles the keys adjacent to it on the ring, not the whole keyspace, the enabling trick behind every partitioned store that needs to scale its node count up or down without a full data reshuffle.
A cache that is trivial at small scale, look up a key, fall back to the database on a miss, breaks in several specific, real ways at billions of requests per second: thundering herds on a popular miss, stale reads racing a database write, and cross-region replication lag, each addressed by a distinct, deliberate mechanism (leases, an invalidation daemon reading the DB commit log, and regional pools) rather than one generic fix.
Dynamo replaces a strict quorum (which blocks writes whenever a designated replica is down) with a sloppy quorum: a write goes to the first N healthy nodes walking the ring, even if that means a temporary, hinted stand-in for an unavailable node, choosing availability for writes over always hitting the theoretically correct replica set.
Because sloppy quorums let two clients write the same key without either seeing the other's write, Dynamo tags every value with a vector clock and hands genuinely concurrent versions back to the application to reconcile, semantically honest about the eventual consistency it is choosing rather than hiding the conflict and picking a version arbitrarily.
A single master holding only metadata, many chunkservers holding 64MB chunks replicated three ways, and a design that deliberately optimizes for huge sequential reads and append-only writes over the small random-access files a single-machine file system is built for.
A sparse, distributed, sorted map keyed by (row, column family:qualifier, timestamp), split into row-range tablets and persisted as immutable, sorted SSTables on GFS, an entirely different storage layout from a B+Tree-indexed relational engine, optimized for extremely wide, sparse tables and high-throughput sequential writes.
Bigtable does not solve distributed coordination itself; it leans entirely on Chubby, a separate lock service, to elect a single live master, let tablet servers claim ownership of the tablets they serve, and store schema and access-control metadata, a clean separation of "serve data fast" from "agree on who is in charge."
ZooKeeper generalizes Chubby's idea into an open, reusable coordination service, a hierarchy of znodes with client-set watches for notifications, all writes ordered by ZAB, a leader-based atomic broadcast protocol that solves exactly the same replicated-log problem Raft does; a client library links this concept directly back to the consensus theory already covered.
When a single transaction touches data partitioned across multiple machines, a coordinator must get every participant to vote "prepared to commit" before any of them actually commits; the protocol is simple but has one real weakness, a participant that is prepared and then loses contact with the coordinator is stuck holding its locks, blocked, until the coordinator comes back.
Spanner buys external consistency across data centers spread around the planet not by inventing new consensus theory but by exposing clock uncertainty directly: TrueTime returns an interval guaranteed to contain the true time, and a transaction simply waits out that interval (commit wait) before releasing its result, converting a hard distributed-systems problem into a bounded, physical one.
Arrange replicas in a line instead of around a single primary: writes enter at the head and flow node to node to the tail, which alone answers reads, giving the same strong consistency as primary-backup replication while spreading write throughput across every node in the chain rather than bottlenecking on one primary.
Given three realistic workloads (a shopping cart that must never lose a write, a global financial ledger that must never show two conflicting balances, and a batch analytics job over petabytes of logs), pick and justify a real system design from this discipline for each, and explain precisely which guarantee each alternative design would have given up.
A programmer supplies only a map function (turning one input record into intermediate key/value pairs) and a reduce function (combining all values for one intermediate key); the runtime supplies everything else, parallelization, scheduling, network shuffling, and fault tolerance, over data already sitting in a distributed file system like GFS.
A single master re-executes any task whose worker stops heartbeating, schedules map tasks on (or near) the chunkserver already holding their input to avoid saturating the network, and launches redundant backup executions of the last few straggling tasks so one slow machine cannot hold up an entire job.
GFS deliberately exposes a relaxed, sometimes-undefined-but-consistent view of a file after concurrent writes, in exchange for a specialized atomic record append operation that lets thousands of producers write to the same file concurrently without a distributed lock, precisely the trade-off its target workload (MapReduce output, log-style writes) actually needs.
Part of the Computer Science curriculum.
The theory of consensus, replication, and consistency from Distributed Systems I is necessary but not sufficient: real systems serving billions of requests a day are built by combining those primitives into specific, opinionated designs, each trading correctness, availability, and performance differently for a specific workload.
A function-as-a-service platform runs a stateless handler on demand, scaling from zero to thousands of concurrent invocations and billing per invocation, a genuine step forward for autoscaling, but one that a systems-level critique shows is disappointingly stateless and I/O-poor exactly where the case studies in this discipline (data-parallel jobs, distributed storage) need state and fast networking most.
Mapping both nodes and keys onto the same hash ring, and assigning each key to the next node clockwise, means adding or removing one node only reshuffles the keys adjacent to it on the ring, not the whole keyspace, the enabling trick behind every partitioned store that needs to scale its node count up or down without a full data reshuffle.
A cache that is trivial at small scale, look up a key, fall back to the database on a miss, breaks in several specific, real ways at billions of requests per second: thundering herds on a popular miss, stale reads racing a database write, and cross-region replication lag, each addressed by a distinct, deliberate mechanism (leases, an invalidation daemon reading the DB commit log, and regional pools) rather than one generic fix.
Dynamo replaces a strict quorum (which blocks writes whenever a designated replica is down) with a sloppy quorum: a write goes to the first N healthy nodes walking the ring, even if that means a temporary, hinted stand-in for an unavailable node, choosing availability for writes over always hitting the theoretically correct replica set.
Because sloppy quorums let two clients write the same key without either seeing the other's write, Dynamo tags every value with a vector clock and hands genuinely concurrent versions back to the application to reconcile, semantically honest about the eventual consistency it is choosing rather than hiding the conflict and picking a version arbitrarily.
A single master holding only metadata, many chunkservers holding 64MB chunks replicated three ways, and a design that deliberately optimizes for huge sequential reads and append-only writes over the small random-access files a single-machine file system is built for.
A sparse, distributed, sorted map keyed by (row, column family:qualifier, timestamp), split into row-range tablets and persisted as immutable, sorted SSTables on GFS, an entirely different storage layout from a B+Tree-indexed relational engine, optimized for extremely wide, sparse tables and high-throughput sequential writes.
Bigtable does not solve distributed coordination itself; it leans entirely on Chubby, a separate lock service, to elect a single live master, let tablet servers claim ownership of the tablets they serve, and store schema and access-control metadata, a clean separation of "serve data fast" from "agree on who is in charge."
ZooKeeper generalizes Chubby's idea into an open, reusable coordination service, a hierarchy of znodes with client-set watches for notifications, all writes ordered by ZAB, a leader-based atomic broadcast protocol that solves exactly the same replicated-log problem Raft does; a client library links this concept directly back to the consensus theory already covered.
When a single transaction touches data partitioned across multiple machines, a coordinator must get every participant to vote "prepared to commit" before any of them actually commits; the protocol is simple but has one real weakness, a participant that is prepared and then loses contact with the coordinator is stuck holding its locks, blocked, until the coordinator comes back.
Spanner buys external consistency across data centers spread around the planet not by inventing new consensus theory but by exposing clock uncertainty directly: TrueTime returns an interval guaranteed to contain the true time, and a transaction simply waits out that interval (commit wait) before releasing its result, converting a hard distributed-systems problem into a bounded, physical one.
Arrange replicas in a line instead of around a single primary: writes enter at the head and flow node to node to the tail, which alone answers reads, giving the same strong consistency as primary-backup replication while spreading write throughput across every node in the chain rather than bottlenecking on one primary.
Given three realistic workloads (a shopping cart that must never lose a write, a global financial ledger that must never show two conflicting balances, and a batch analytics job over petabytes of logs), pick and justify a real system design from this discipline for each, and explain precisely which guarantee each alternative design would have given up.
A programmer supplies only a map function (turning one input record into intermediate key/value pairs) and a reduce function (combining all values for one intermediate key); the runtime supplies everything else, parallelization, scheduling, network shuffling, and fault tolerance, over data already sitting in a distributed file system like GFS.
A single master re-executes any task whose worker stops heartbeating, schedules map tasks on (or near) the chunkserver already holding their input to avoid saturating the network, and launches redundant backup executions of the last few straggling tasks so one slow machine cannot hold up an entire job.
GFS deliberately exposes a relaxed, sometimes-undefined-but-consistent view of a file after concurrent writes, in exchange for a specialized atomic record append operation that lets thousands of producers write to the same file concurrently without a distributed lock, precisely the trade-off its target workload (MapReduce output, log-style writes) actually needs.