jaktestowac/testing-performance-and-load

Testing Performance and Load

Use this skill when a system's speed or capacity is in question and the answer needs to be a number someone can act on.

Most performance testing produces numbers nobody uses. The two causes are always the same: a workload that does not resemble reality, and a result reported as an average. A test that hammers one endpoint with a flat 500 virtual users tells you how the system responds to something that will never happen, and a mean response time of 200ms is compatible with one user in twenty waiting four seconds.

When to Use

• a feature has a stated latency or throughput requirement
• "the app feels slow" needs to become a measurement
• a launch, campaign, or migration needs a capacity check
• a performance regression is suspected between two releases
• a performance result exists and needs interpreting
• a CI check should catch obvious regressions before they ship

Operating Principles

• Model the workload before writing the script. Test shape comes from real traffic: which endpoints, in what ratio, with what think time, at what concurrency.
• Percentiles, never averages. Report p50, p95, p99. The mean describes nobody's experience and hides the tail entirely.
• A threshold without an SLO is a guess. Derive the number from what users need or what the business promised, and say which.
• One variable per run. Comparing two runs that differ in load, data volume, and code changes tells you nothing about any of them.
• Environment differences invalidate the number, not just weaken it. A result from a quarter-size environment is a shape, not a capacity.
• Find the bottleneck, do not just report the symptom. "p95 is 3 seconds" is an observation; "p95 is 3 seconds because the product query has no index on category_id" is a finding.

Workflow

Phase 0: Name the question

Different questions need different tests. Pick one per run.

| Question | Test type | Shape | | --- | --- | --- | | Is it fast enough under normal load? | Load test | Steady state at expected concurrency | | Where does it break? | Stress test | Ramp until failure | | Does it survive a sudden surge? | Spike test | Step change up, then down | | Does it degrade over hours? | Soak test | Steady load for 4 to 24 hours | | Did this release make it slower? | Regression comparison | Identical shape, two builds | | How much capacity do we have? | Capacity test | Ramp to the SLO breach point |

A test that tries to answer all six answers none. If "it is slow" is all you have, start with a load test at expected concurrency and let the result narrow the question.

Phase 1: Model the workload

The phase that decides whether the result means anything. Work through ./resources/workload-model.md.

From real traffic where possible:

• Endpoint mix: the actual ratio, from access logs or analytics. Reads usually outnumber writes by an order of magnitude, and synthetic tests usually get this wrong.
• Concurrency: derived from throughput and think time, not guessed. concurrent users ≈ arrival rate × session duration.
• Think time: real users pause. A test with zero think time is a different system entirely.
• Data distribution: most requests hit a small hot set. A test with uniformly random ids has a cache hit rate nothing like production.
• Session shape: users log in, browse, and act. A test that only calls the checkout endpoint skips the state that makes checkout slow.
• Peak profile: daily peak, weekly peak, and the campaign or event shape if there is one.

When real traffic is unavailable, state the assumptions explicitly and mark the result as assumption-dependent. An assumed workload is workable; an unstated one is not.

Phase 2: Set thresholds from the SLO

Each threshold traces to something: a user need, a documented SLO, a competitor benchmark, or a current baseline you intend to hold.

./resources/thresholds-and-slos.md covers deriving them. The shape:

thresholds: {
  'http_req_duration{name:checkout}': ['p(95)<800', 'p(99)<2000'],
  'http_req_failed': ['rate<0.001'],
  'checkout_completed': ['count>0'],
}

Rules:

• per-endpoint, not global. A global p95 mixes a health check with a report export and describes neither.
• include an error rate threshold. A fast 500 is not a pass, and a load test with no error threshold will report one as success.
• include at least one business-outcome counter. "Requests succeeded" is not the same as "orders were created", and a test that never completes a checkout can look perfect.

Phase 3: Prepare the environment

Record every difference from production before running anything. The differences bound what the result can claim.

• Sizing: instance count, CPU, memory, relative to production
• Data volume: a database with 2,000 rows and one with 4 million behave differently, and the difference is not linear
• Data shape: long-lived accounts, large carts, deep histories
• Caches: cold or warm, and warmed how
• Dependencies: real, stubbed, or rate-limited. A stubbed dependency that responds in 1ms removes the queueing that production has.
• Network: the load generator's location relative to the system
• Autoscaling: enabled or fixed, and with what policy

Then decide where the load generator runs. A generator on the same machine as the system under test competes with it for CPU, and the result measures the contention.

Phase 4: Warm up, then measure

• Warmup: run at low load until JIT compilation, connection pools, and caches settle. Exclude the warmup from the results, and say how long it was.
• Ramp: reach target load over minutes, not instantly, unless the spike is the subject.
• Steady state: long enough for the metric to stabilize. Ten minutes is usually the minimum for anything meaningful.
• Ramp-down: watch for errors during the descent; connection pool problems often surface here.

Monitor the system, not only the client. Client-side response times tell you something is slow; server CPU, memory, database connections, queue depth, and GC pauses tell you why. A run with no server-side observation produces a symptom and no cause.

Phase 5: Analyze

Report distributions.

• p50 is the typical experience
• p95 is the experience of a noticeable minority
• p99 is where the timeouts and abandonments live
• max is one data point, and it is usually a GC pause or a cold start

Look for the shapes in ./resources/perf-report-template.md:

• the knee: throughput plateaus while response time climbs; that is the capacity limit
• the sawtooth: periodic latency spikes, usually GC or a cache expiry
• the drift: response time climbing over a soak; usually a leak or unbounded growth
• the cliff: sudden failure past a threshold, usually a pool or a connection limit

Correlate every latency feature with a server-side metric before naming a cause. A hypothesis without a correlating metric is a guess, and performance work built on a guessed bottleneck is expensive.

Phase 6: Report and act

Use ./resources/perf-report-template.md. It requires:

• the workload model, so the reader can judge whether it resembles reality
• environment differences from production, stated up front
• results as percentiles per endpoint, with the error rate
• the bottleneck, with the evidence for it
• what the result does not tell you
• a recommendation

Add a lightweight CI check when there is something worth protecting: a short run at modest load, thresholds set generously enough not to be flaky, and a comparison against the last release. Recipes in ./resources/k6-recipes.md. Its job is catching an order-of-magnitude regression, not measuring capacity.

Common Failure Modes

• reporting the average, which describes nobody and hides the tail
• a flat load with no think time and no ramp, which no real traffic resembles
• uniformly random test data, giving a cache behaviour production never has
• testing one endpoint in isolation when the slowness comes from contention between several
• no error rate threshold, so a run that returned 500s at speed is reported as a pass
• a test that never completes a business transaction while every request succeeds
• results from an environment a quarter the size, presented as capacity
• no warmup, so the first minute's cold-start latency poisons the percentiles
• naming a bottleneck from client-side numbers alone
• a CI perf gate so tight it flakes, then gets disabled and never re-enabled

Resource Map

• ./resources/workload-model.md - deriving the endpoint mix, concurrency, think time, and data distribution from real traffic, with a worked example
• ./resources/thresholds-and-slos.md - deriving thresholds from SLOs, percentile choice, error budgets, and per-endpoint targets
• ./resources/k6-recipes.md - k6 scripts for load, stress, spike, and soak shapes, plus the CI regression check and an Artillery equivalent
• ./resources/perf-report-template.md - report structure, the four curve shapes and what each means, and a worked example

Related Skills

• assessing-release-readiness - when a performance result feeds a go/no-go decision
• analyzing-quality-metrics - when performance should be trended across releases rather than measured once
• testing-api-contracts - when the endpoints under load also need shape verification
• designing-test-data - when the load test needs a realistic hot set and data distribution
• handling-sensitive-test-data - when load test data is derived from production traffic
• automating-ci-test-pipelines (planned) - for running the CI regression check and storing its history
• tech-debt-analysis - when the bottleneck is structural rather than a single query

Definition of Done

This skill is complete when:

• the question is one of load, stress, spike, soak, regression, or capacity, and only one
• the workload model states endpoint mix, concurrency, think time, and data distribution, with its source or its assumptions
• thresholds are per-endpoint and each traces to an SLO, a user need, or a stated baseline
• an error rate threshold and at least one business-outcome counter are included
• environment differences from production are recorded before the run
• warmup is excluded from the results and its duration stated
• results are reported as percentiles, never as averages
• every latency finding is correlated with a server-side metric before a cause is named
• the report says what the result does not tell you