klod68/circuit-breaker-state-machine

---

name: circuit-breaker-state-machine description: "Use when protecting a service from repeatedly calling a failing dependency, when retry logic alone keeps hammering an already-down service, or when you need to fail fast after sustained failures and probe cautiously for recovery. Covers the three-state circuit breaker (Closed → Open → Half-Open), Polly v8 ResiliencePipeline configuration with FailureRatio and MinimumThroughput tuning, and correct retry + circuit breaker composition order. Domain: Resilience, Fault Tolerance. Level: Intermediate. Tags: circuit-breaker, resilience, polly, fault-tolerance, state-machine."

Circuit Breaker State Machine

Problem

When a downstream dependency fails, retry logic keeps hammering it — wasting resources, delaying responses, and preventing the dependency from recovering. The system needs to "fail fast" once it detects sustained failure, then cautiously probe for recovery.

Solution: Circuit Breaker (Three States)

  ┌──────────┐    failure threshold    ┌──────────┐
  │  CLOSED  │ ──────────────────────► │   OPEN   │
  │ (normal) │                         │ (reject) │
  └──────────┘                         └──────────┘
       ▲                                    │
       │  success in half-open              │ break duration expires
       │                                    ▼
       │                              ┌───────────┐
       └───────────────────────────── │ HALF-OPEN │
                                      │  (probe)  │
                                      └───────────┘

| State | Behavior | |-------|----------| | Closed | Requests pass through. Failures are counted. When the failure ratio exceeds the threshold within a sampling window, transition to Open. | | Open | Requests are immediately rejected with BrokenCircuitException. After the break duration, transition to Half-Open. | | Half-Open | A limited number of probe requests are allowed through. If they succeed, transition to Closed. If they fail, transition back to Open. |

Implementation with Polly v8

using Polly;

var pipeline = new ResiliencePipelineBuilder()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,                           // Open after 50% failures
        SamplingDuration = TimeSpan.FromSeconds(30),   // Within a 30-second window
        MinimumThroughput = 10,                        // Need at least 10 calls to evaluate
        BreakDuration = TimeSpan.FromSeconds(15),      // Stay open for 15 seconds
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
            .Handle<TimeoutException>(),
        OnOpened = args =>
        {
            // Log at Warning — this is a meaningful state transition
            logger.LogWarning("Circuit opened for {Duration} after {FailureRatio:P0} failure rate",
                args.BreakDuration, args.Outcome.Exception?.Message);
            return ValueTask.CompletedTask;
        },
        OnClosed = args =>
        {
            logger.LogInformation("Circuit closed — dependency recovered");
            return ValueTask.CompletedTask;
        },
        OnHalfOpened = args =>
        {
            logger.LogInformation("Circuit half-opened — probing dependency");
            return ValueTask.CompletedTask;
        }
    })
    .Build();

Combining Retry + Circuit Breaker

Order matters: retry wraps circuit breaker — retries happen inside the circuit.

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        Delay = TimeSpan.FromMilliseconds(500),
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,
        SamplingDuration = TimeSpan.FromSeconds(30),
        MinimumThroughput = 10,
        BreakDuration = TimeSpan.FromSeconds(15)
    })
    .Build();

With HttpClientFactory

builder.Services.AddHttpClient("MyApi")
    .AddStandardResilienceHandler(); // Includes retry + circuit breaker + timeout

Configuration as Policy

public sealed class CircuitBreakerOptions
{
    public double FailureRatio { get; set; } = 0.5;
    public TimeSpan SamplingDuration { get; set; } = TimeSpan.FromSeconds(30);
    public int MinimumThroughput { get; set; } = 10;
    public TimeSpan BreakDuration { get; set; } = TimeSpan.FromSeconds(15);
}

When to Use

• HTTP calls to external APIs that may be down or degraded
• Database connections to replicas that may fall behind
• Any dependency where sustained failure wastes resources
• Protecting upstream services from cascading failures

When NOT to Use

• In-process method calls (no network involved)
• Operations that fail for business reasons (validation errors, not-found)
• One-off scripts or batch jobs with manual retry

Gotchas

1. MinimumThroughput: Set high enough to avoid opening on a small sample. 2 failures out of 3 calls is not a pattern — it's noise.
2. BrokenCircuitException: Callers must handle this explicitly — it means "don't even try." Map it to 503 Service Unavailable in API responses.
3. Shared circuit state: A circuit breaker instance is shared across all calls through the same pipeline. Register one pipeline per dependency, not per call site.
4. Health checks: Expose circuit state in a health check — an open circuit is an Unhealthy / Degraded signal.
5. Fallback: Consider a fallback strategy for open-circuit scenarios: cached data, default response, or graceful degradation.

Related Skills

• Retry with Exponential Backoff (retry before circuit opens)
• Health Check Endpoint (expose circuit state)
• Resilience Configuration Agent (review policy settings)