curiositech/ongaro-ousterhout-2014-raft

SKILL: Raft Consensus Design Principles

license: Apache-2.0

Metadata

• Name: raft-consensus-design
• Version: 1.0.0
• Description: Apply Raft's design principles for building understandable, reliable distributed and multi-agent systems
• Source: "In Search of an Understandable Consensus Algorithm" (Ongaro & Ousterhout)

Activation Triggers:

• Designing multi-agent coordination systems
• Debugging distributed system failures or edge cases
• Evaluating system architecture trade-offs between complexity and capability
• Implementing consensus, orchestration, or leader election
• Making systems more maintainable or debuggable
• Decomposing complex problems with interdependent parts

When to Use This Skill

Load this skill when you need to:

1. Design for human comprehension: When system correctness depends on implementers understanding it completely (not just theoretically)
2. Coordinate distributed agents: When multiple agents/processes must agree on state despite failures
3. Reduce edge cases: When a system has too many special cases, corner conditions, or mysterious bugs
4. Choose between design alternatives: When multiple approaches work theoretically but you need criteria beyond performance
5. Prevent rather than handle failures: When you want to make certain failure modes structurally impossible
6. Balance centralization vs. distribution: When deciding between hierarchical control and peer-to-peer coordination

This skill is NOT for: Systems where theoretical elegance matters more than implementation correctness, or where perfect optimization trumps maintainability.

Core Mental Models

1. Understandability as First-Class Design Objective

Principle: Optimize for the cognitive limitations of implementers, not the aesthetic preferences of designers.

When faced with design alternatives that are theoretically equivalent:

• Ask: "Which approach can a competent engineer explain completely in 15 minutes?"
• Ask: "Which design has fewer hidden implications and subtle interdependencies?"
• Measure: Test whether people can actually implement it correctly (Raft's empirical study showed 77% better comprehension than Paxos)

Why it matters: Complex systems fail not because the theory is wrong but because humans misunderstand the theory. A "simpler" algorithm that's harder to explain produces more bugs than a "complex" algorithm with clear component boundaries.

Application: Before adding abstraction, generality, or optimization, ask if the complexity cost exceeds the capability gain. Make the common case trivial even if it makes edge cases more explicit.

2. Decomposition by Interdependency Minimization

Principle: The joints between components matter more than the components themselves.

Good decomposition separates concerns such that:

• Each component can be explained with minimal reference to others
• Failure in one component has limited blast radius
• Implementation order is flexible (loose coupling in time)
• The interface between components is smaller than the component internals

Raft's example: Leader election, log replication, and safety are nearly independent subproblems. You can understand log replication without understanding election mechanics.

Bad decomposition signs:

• "You can't understand X without first understanding Y and Z"
• Changing one component requires coordinated changes in others
• The interface documentation is as complex as the implementation
• Edge cases span multiple components

3. State Space Reduction Through Strategic Constraints

Principle: Adding constraints makes systems more reliable by eliminating states humans can't reason about.

Raft's constraints:

• Logs have no holes (entries are contiguous)
• Leaders never overwrite their own entries (append-only)
• Log entries flow unidirectionally (leader → followers)
• Only candidates with up-to-date logs can be elected

Each constraint eliminates combinatorial possibilities. The Leader Append-Only Property alone prevents entire classes of consistency violations.

How to apply:

1. Identify the full state space your system could theoretically occupy
2. Ask which states are genuinely necessary vs. artifacts of implementation freedom
3. Add invariants that make unnecessary states impossible (not just invalid)
4. Enforce constraints structurally (type systems, immutability) rather than procedurally (validation logic)

Trade-off: Strategic inflexibility reduces capability less than it reduces complexity. Raft is "less general" than Paxos but handles the practical case more reliably.

4. Asymmetric Coordination Simplifies Common Paths

Principle: Hierarchical coordination with clear authority is simpler than democratic consensus, even when both achieve the same outcome.

Raft's strong leader model:

• Leader decides log entry placement unilaterally
• Followers validate and replicate (they don't negotiate)
• Complex election logic is isolated to rare failure scenarios
• Most system operation follows: leader proposes → majority accepts

When to use strong leadership:

• The common case (normal operation) is much more frequent than failures
• Decision quality matters less than decision latency
• Coordination overhead exceeds the cost of occasional wrong decisions
• System must remain available during partial failures

When NOT to use it:

• Decisions are irreversible and must be optimal
• No single agent has sufficient information
• Trust is distributed (no agent should have unilateral authority)

5. Strategic Nondeterminism Eliminates Corner Cases

Principle: Reducing guarantees can paradoxically increase reliability by eliminating pathological cases humans can't anticipate.

Raft's randomized election timeouts: Each server picks a random timeout value. If deterministic, split votes become likely in certain timing scenarios. Randomization handles all possible timings uniformly: "Pick any; it doesn't matter."

Application pattern:

• Use randomization when: (1) optimal choice is unknowable, (2) any adequate choice works, (3) deterministic approaches create timing/ordering dependencies
• Don't randomize when: (1) choices have irreversible consequences, (2) you need reproducible behavior for debugging

Examples beyond Raft:

• Load balancing: Random selection avoids thundering herd better than deterministic rules
• Retry timing: Exponential backoff with jitter prevents synchronized retry storms
• Agent task selection: "Pick any ready task randomly" is more robust than complex priority optimization

Decision Frameworks

Framework 1: Decomposition Quality Check

When decomposing a system into components, evaluate:

IF explanation requires saying "but first you need to understand..."
THEN decomposition boundaries are wrong

IF changing component A forces changes in component B
THEN they share hidden state or assumptions

IF the interface between components is complex
THEN consider merging them or finding different boundary

IF failure in component A cascades to component B
THEN they're not truly independent subproblems

Good test: Can you implement components in any order? If implementation order is forced, your decomposition creates artificial dependencies.

Framework 2: State Space Constraint Selection

When system has too many edge cases:

1. Enumerate actual states system occupies during correct operation
2. Identify theoretical states that implementation allows but are never needed
3. For each unnecessary state, ask: "What constraint would make this impossible?"
4. Add constraints that maximize (edge cases eliminated / functionality reduced)
5. Enforce through types, invariants, or architectural patterns—not validation

Example: If logs can have holes, you must handle: holes during replication, holes during recovery, hole compaction, hole replication retry. Constraint "no holes allowed" eliminates all these cases.

Framework 3: Centralization vs. Distribution

When choosing coordination model:

PREFER strong leader (asymmetric) IF:
- Common case is 100x more frequent than failures
- Decision latency matters more than optimal correctness
- Single agent has sufficient context for good decisions
- Coordination overhead exceeds cost of occasional suboptimal choice

PREFER symmetric peer coordination IF:
- No agent has privileged information
- System must tolerate arbitrary leader failure
- Decision quality matters more than latency
- Trust should be distributed

Hybrid option: Leader for common case, consensus for critical decisions (this is Raft's actual pattern).

Framework 4: Determinism vs. Randomization

When facing coordination deadlocks or pathological timing:

USE randomization IF:
- All adequate choices are roughly equivalent
- Deterministic approach creates timing dependencies
- Corner cases proliferate despite attempts to handle them
- System needs to break symmetry quickly

STAY deterministic IF:
- Debugging requires reproducibility
- Choices have irreversible consequences
- Compliance requires audit trails
- Performance optimization depends on predictability

Warning sign: If you keep finding "one more edge case" in deterministic design, randomization may eliminate the entire class.

Framework 5: Specification Completeness

When documenting systems or designing agents:

Specification is incomplete if implementer must guess:
- What happens during partial failures
- Resource limits and timing assumptions
- Recovery procedures after crashes
- How to handle concurrent operations

Complete specification includes:
- Normal operation (the algorithm)
- All failure modes and recovery paths
- Timing assumptions and resource limits
- Invariants that must be maintained
- Extension points and what they can assume

Test: Can someone implement correctly without asking clarifying questions? Raft's completeness enabled independent implementations that interoperated.

Reference Documentation

| Reference File | When to Load | Key Content | |---------------|--------------|-------------| | decomposition-as-coordination-strategy.md | Designing multi-component systems; experiencing tight coupling between modules | How Raft's three-way split (election/replication/safety) reduces interdependency; criteria for good vs. bad decomposition boundaries | | strong-leadership-vs-democratic-coordination.md | Choosing between centralized orchestrator and peer coordination; designing leader election | Deep analysis of strong leader trade-offs; when asymmetric coordination simplifies systems; failure scenario handling | | state-space-reduction-through-constraints.md | System has too many edge cases; debugging mysterious state corruption; designing invariants | Raft's specific constraints (no holes, append-only, etc.); how restrictions eliminate bug classes; constraint selection methodology | | randomization-and-coordination-simplicity.md | Facing split-brain or coordination deadlocks; deterministic approach creating corner cases | Why Raft uses randomized timeouts; when nondeterminism reduces complexity; trade-offs vs. reproducibility | | failure-modes-and-recovery-design.md | Implementing fault tolerance; system experiencing undefined failure states; designing recovery logic | Raft's failure taxonomy; how to design for failure from the start; making recovery paths first-class design elements | | understandability-as-measurable-goal.md | Evaluating design alternatives; system is "theoretically correct" but brittle in practice; measuring design quality | Raft's empirical study comparing comprehension; metrics for understandability; how to make it a design objective not afterthought | | implementing-vs-specifying-distributed-systems.md | Writing specifications or documentation; gap between design and implementation; distributed system design | The completeness gap in academic literature; what Raft includes that others omit; how to write implementable specs |

Anti-Patterns

1. Optimization Before Comprehension

Mistake: Adding performance optimizations or generalizations before the core algorithm is fully understood and correctly implemented.

Why it fails: Optimization introduces special cases. If the base case isn't crystal clear, optimizations create compound complexity that's exponentially harder to debug.

Raft's lesson: The paper explicitly omits optimizations to maintain focus on understandability. Add performance later, after correctness is proven.

2. Symmetric Coordination for Common Cases

Mistake: Using consensus/voting for every decision because it's "more distributed" or "more fault-tolerant."

Why it fails: Coordination overhead dominates when most decisions are uncontroversial. Symmetric protocols require multiple round-trips even when all agents would agree immediately.

Raft's lesson: Use strong leader for common case, consensus only for critical decisions (like leader election itself).

3. Handling Invalid States Instead of Preventing Them

Mistake: Writing validation and error-handling logic for states that shouldn't be possible.

Why it fails: Error handlers become part of the state space, introducing their own edge cases. Recovery logic is rarely tested and often wrong.

Raft's lesson: Make invalid states impossible through constraints. The system should structurally prevent holes in logs, not detect and repair them.

4. Decomposition by Load Instead of Concern

Mistake: Splitting system into components based on performance characteristics, code size, or team boundaries rather than semantic independence.

Why it fails: Creates components with high semantic coupling and low cohesion. Changes ripple across boundaries; understanding one requires understanding others.

Raft's lesson: Decompose by failure modes and logical independence. Raft's three components fail independently and can be reasoned about separately.

5. Specification Without Operational Detail

Mistake: Describing the algorithm's correctness properties without timing assumptions, resource limits, or failure handling.

Why it fails: Implementers must guess about edge cases, leading to divergent implementations that don't interoperate.

Raft's lesson: Include timing (election timeouts), resources (log compaction), and failure modes (network partitions, crashes) in the spec itself.

6. Determinism at All Costs

Mistake: Insisting on deterministic behavior even when it creates cascading corner cases.

Why it fails: Deterministic coordination in asynchronous systems requires complex tie-breaking, ranking, or additional rounds. Corner cases multiply.

Raft's lesson: The authors tried deterministic election first, found "subtle issues" that kept proliferating, and switched to randomization. Sometimes nondeterminism is the simpler choice.

Shibboleths: True Understanding vs. Surface Knowledge

Surface-Level Understanding Says:

• "Raft is just Paxos made simple"
• "We chose Raft because it's easier to implement"
• "Strong leader means single point of failure"
• "Randomization makes the system unpredictable"

Deep Understanding Says:

• "Raft demonstrates that understandability is a measurable design objective that trades off against other goals—the paper's contribution is making that trade-off explicit and empirically validated"
• "Raft's decomposition strategy—separating leader election, log replication, and safety into nearly independent subproblems—is more important than any individual component's simplicity"
• "Strong leader simplifies the common case at the cost of more complex leader election; this is a deliberate trade-off based on failure frequency"
• "Randomized timeouts eliminate pathological timing scenarios that deterministic approaches must explicitly handle; it reduces corner cases, not predictability"

Tells That Someone Has Internalized Raft:

1. They explain constraints before algorithms

• Surface: "Here's how log replication works..."
• Deep: "First, we enforce three invariants: logs have no holes, leaders append-only, and entries flow unidirectionally. Given these constraints, replication becomes..."

2. They evaluate designs by understandability

• Surface: "Option A is more elegant/efficient"
• Deep: "Option A requires understanding X, Y, and Z simultaneously. Option B separates concerns. Even though A is theoretically simpler, B will produce fewer implementation bugs."

3. They recognize decomposition quality

• Surface: "We split it into microservices for scalability"
• Deep: "These components share hidden state through the database—they're not truly independent. Real decomposition would separate by failure mode, not by resource usage."

4. They see the tension between specification completeness and abstraction

• Surface: "The spec should be implementation-agnostic"
• Deep: "Raft includes timing windows and resource limits in the spec because implementers need them for correctness. Abstraction that omits operational details creates the gap between theory and practice."

5. They know when NOT to apply Raft's lessons

• Surface: "Always use strong leader for coordination"
• Deep: "Strong leader works when common case dominates. For systems where every decision is critical or trust must be distributed, symmetric consensus is worth the complexity cost."

The Ultimate Shibboleth

Ask: "Why did the Raft authors conduct an empirical study comparing comprehension with Paxos?"

Surface answer: "To prove Raft is simpler."

Deep answer: "To demonstrate that understandability can be operationalized and measured, which makes it a legitimate design objective rather than a subjective preference. The study's real contribution is showing that design-for-comprehension produces measurably better outcomes—it's not just about aesthetics, it's about reliability. This challenges the assumption that theoretical elegance and practical understandability are aligned."

---

Note: This skill emphasizes design principles over implementation details. Load reference files for deeper dives into specific mechanisms, trade-offs, and applications.