sentenz/go-benchmark-testing

Benchmark Testing

Instructions for AI coding agents on automating benchmark test creation using consistent software testing patterns in this Go project.

• 1. Benefits
• 2. Principles
  • 2.1. FIRST
• 3. Patterns
  • 3.1. Microbenchmarking
  • 3.2. Comparative Benchmarking
  • 3.3. Memory Profiling
  • 3.4. Statistical Benchmarking
  • 3.5. Sub-benchmarks
  • 3.6. Table-Driven Testing
• 4. Workflow
• 5. Commands
• 6. Style Guide
• 7. Template
  • 7.1. Multi-Scenario Benchmarks
  • 7.2. Simple Benchmarks
  • 7.3. Benchmarks with Validation
• 8. References

1. Benefits

• Performance Measurement

  Benchmark tests measure the execution time and memory allocation of functions, providing quantifiable metrics for performance analysis.

• Regression Detection

  Continuous benchmarking helps identify performance regressions early in the development cycle before they reach production.

• Optimization Guidance

  Benchmark results guide optimization efforts by identifying bottlenecks and quantifying the impact of performance improvements.

• Comparative Analysis

  Benchmarks enable comparison of different implementations or algorithms to make informed decisions about performance trade-offs.

• Resource Profiling

  Memory allocation tracking helps identify unnecessary allocations and optimize memory usage patterns.

2. Principles

2.1. FIRST

The FIRST principles for benchmark testing focus on creating reliable and meaningful measurements.

• Fast

  Benchmark setup and teardown should be minimal and excluded from timing to ensure accurate measurement of the function under test.

• Independent

  Each benchmark should be self-contained and not depend on shared state or results from other benchmarks to ensure isolated performance measurements.

• Repeatable

  Benchmarks should produce consistent, comparable results across runs and environments by controlling inputs and avoiding non-deterministic operations.

• Self-Validating

  Benchmarks should optionally validate results to prevent the compiler from optimizing away the code under measurement.

• Timely

  Benchmarks should be established before optimization work begins to provide a performance baseline and measure the impact of changes.

3. Patterns

3.1. Microbenchmarking

Microbenchmarking is a software testing technique that measures the performance of small, isolated code units to identify performance characteristics and bottlenecks.

3.2. Comparative Benchmarking

Comparative Benchmarking is a testing approach that compares the performance of different implementations or algorithms side-by-side using consistent workloads.

3.3. Memory Profiling

Memory Profiling is the process of measuring memory allocations and usage patterns during benchmark execution using -benchmem flag.

3.4. Statistical Benchmarking

Statistical Benchmarking uses multiple iterations to calculate statistical measures (mean, variance) to ensure reliable and reproducible results.

3.5. Sub-benchmarks

Sub-benchmarks organize related benchmark cases using b.Run() to group variations of the same function with different input scenarios.

3.6. Table-Driven Testing

Table-Driven Testing is a software testing technique in which benchmark cases are organized in a tabular format to systematically cover different input scenarios.

4. Workflow

1. Identify

   Identify performance-critical functions in pkg/ or internal/ that benefit from performance tracking (e.g., pkg/<package>/<file>.go).

2. Add/Create

   Create benchmark tests in the same package (e.g., pkg/<package>/<file>_test.go).

3. Benchmark Test Coverage Requirements

   Focus on functions that:

   • Are called frequently in hot paths
   • Perform mathematical operations or calculations
   • Process data structures or collections
   • Have multiple implementation approaches to compare
   • Are candidates for optimization

4. Apply Templates

   Structure all benchmark tests using the template pattern.

5. Baseline Measurements

   Establish performance baselines by running benchmarks on stable code before making changes.

5. Commands

| Command | Description | | --------------------------------------------------------------- | -------------------------------------------------- | | make go-test-bench | Execute all benchmarks with memory statistics | | go test -bench=BenchmarkPercent -benchmem ./pkg/percent | Execute a specific benchmark function | | go test -bench=. -benchmem -cpuprofile=cpu.prof ./pkg/percent | Generate CPU profile for performance analysis | | go test -bench=. -benchmem -memprofile=mem.prof ./pkg/percent | Generate memory profile for allocation analysis | | go test -bench=. -benchtime=10s ./pkg/percent | Run benchmarks for a specific duration | | benchstat old.txt new.txt | Compare benchmark results before and after changes |

6. Style Guide

• Test Framework

  Use the standard Go testing package with testing.B for benchmark tests.

• Include Imports

  Include testing and any packages needed for the function under test.

• Benchmark Function Naming

  Name benchmark functions with the Benchmark prefix followed by the function name (e.g., BenchmarkPercent for testing Percent()).

• Benchmark Loop

  Use b.Loop() to control the number of iterations. The testing framework automatically adjusts the loop iterations to get reliable timing measurements. b.Loop() is preferred over b.N as it provides better integration with the testing framework and more accurate measurements. Unlike b.N-style benchmarks, b.Loop() integrates timer management, it automatically handles b.ResetTimer() at the loop's start and b.StopTimer() at its end, eliminating the need to manually manage the benchmark timer for setup and cleanup code.

• Timer Control

  When using b.Loop(), timer management is automatic and no manual b.ResetTimer(), b.StopTimer(), or b.StartTimer() calls are needed for typical benchmarks. For advanced scenarios not using b.Loop(), use b.ResetTimer() to exclude setup time from measurements and b.StopTimer()/b.StartTimer() to exclude specific operations.

• Sub-benchmarks

  Use b.Run() to organize related benchmark cases with different input scenarios. Each sub-benchmark runs independently with its own b.N iterations.

• Memory Reporting

  Use b.ReportAllocs() to report memory allocations per operation when not using -benchmem flag.

• Result Validation

  Optionally validate results in benchmarks to prevent compiler optimizations from eliminating dead code.

7. Template

Use this template for new benchmark test functions. Replace placeholders with actual values and adjust as needed for the use case.

7.1. Multi-Scenario Benchmarks

For benchmarking multiple scenarios or input variations, use sub-benchmarks with table-driven approach.

func Benchmark<FunctionName>(b *testing.B) {
	// Define benchmark cases with different scenarios
	benchmarks := []struct {
		name   string
		param1 <type>
		param2 <type>
		// Add more parameters as needed
	}{
		{
			name:   "scenario description 1",
			param1: <value1>,
			param2: <value2>,
		},
		{
			name:   "scenario description 2",
			param1: <value1>,
			param2: <value2>,
		},
		// Add more benchmark cases
	}

	for _, bm := range benchmarks {
		b.Run(bm.name, func(b *testing.B) {
			// Arrange
			// Setup code here (automatically excluded from timing by b.Loop)

			// Act
			for b.Loop() {
				_, _ = <Function>(bm.param1, bm.param2)
			}
		})
	}
}

7.2. Simple Benchmarks

For benchmarking a single scenario, use a simple loop without sub-benchmarks.

func Benchmark<FunctionName>(b *testing.B) {
	// Arrange
	// Setup code here (automatically excluded from timing by b.Loop)
	param1 := <value1>
	param2 := <value2>

	// Act
	for b.Loop() {
		_, _ = <Function>(param1, param2)
	}
}

7.3. Benchmarks with Validation

For benchmarks that need to prevent compiler optimizations, store results in package-level variables.

var (
	benchResult <type>
	benchError  error
)

func Benchmark<FunctionName>(b *testing.B) {
	// Arrange
	// Setup code here (automatically excluded from timing by b.Loop)
	param1 := <value1>
	param2 := <value2>

	// Act
	for b.Loop() {
		benchResult, benchError = <Function>(param1, param2)
	}
}

8. References

• Go Benchmarks documentation.
• Go testing.B package documentation.
• Go benchstat tool documentation.