adaptationio/test-observability

Test Observability

Overview

Connect your Playwright tests to your observability stack (Grafana, Prometheus, Loki, Tempo) for:

• Trace Correlation: See full trace from browser click → backend → database
• Test Metrics: Dashboard with pass rates, durations, flakiness
• Log Aggregation: Test logs alongside application logs
• Failure Analysis: Quickly identify root cause across distributed systems

---

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Playwright     │────▶│  OTEL Collector  │────▶│  Tempo (Traces) │
│  Tests          │     │                  │────▶│  Loki (Logs)    │
│                 │     │                  │────▶│  Prometheus     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                          │
                                                          ▼
                                                  ┌─────────────────┐
                                                  │    Grafana      │
                                                  │  Test Dashboard │
                                                  └─────────────────┘

---

Quick Start: OTEL Reporter

1. Install Dependencies

npm install playwright-opentelemetry-reporter @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

2. Configure Reporter

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-opentelemetry-reporter', {
      serviceName: 'playwright-tests',
      endpoint: process.env.OTEL_ENDPOINT || 'http://localhost:4318/v1/traces',
    }],
  ],
});

3. Run Tests

# With local OTEL collector
OTEL_ENDPOINT=http://localhost:4318/v1/traces npx playwright test

# With Railway OTEL collector
OTEL_ENDPOINT=http://otel-collector.railway.internal:4318/v1/traces npx playwright test

4. View in Grafana

• Open Grafana → Explore → Tempo
• Search for service.name = "playwright-tests"
• See test spans with duration, status, steps

---

Tracetest Integration (Advanced)

Tracetest enables trace-based testing - assert on any span in the distributed trace.

Installation

npm install @tracetest/playwright

Configuration

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['@tracetest/playwright', {
      serverUrl: process.env.TRACETEST_URL || 'http://localhost:11633',
      apiKey: process.env.TRACETEST_API_KEY,
    }],
  ],
});

Trace-Based Assertions

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';
import { Tracetest } from '@tracetest/playwright';

test('checkout creates order', async ({ page }) => {
  const tracetest = new Tracetest();

  // Start trace capture
  await tracetest.capture();

  // Perform user action
  await page.goto('/checkout');
  await page.click('#place-order');
  await expect(page.locator('.order-confirmation')).toBeVisible();

  // Assert on backend trace!
  await tracetest.assertOnTrace({
    assertions: [
      // API response time
      {
        selector: 'span[name="POST /api/orders"]',
        assertion: 'attr:http.status_code = 201',
      },
      // Database query time
      {
        selector: 'span[name="INSERT orders"]',
        assertion: 'attr:db.duration < 100ms',
      },
      // Payment service
      {
        selector: 'span[name="payment.process"]',
        assertion: 'attr:payment.status = "success"',
      },
    ],
  });
});

Benefits

• 80% faster debugging: See exactly where failures occur
• Full visibility: Browser → API → Database → Services
• Backend assertions: Test database queries, service calls
• Performance validation: Assert on span durations

---

Grafana Dashboard

Import Dashboard

1. Open Grafana → Dashboards → Import
2. Upload dashboards/test-results-dashboard.json
3. Select Prometheus and Tempo data sources

Key Panels

| Panel | Query | Purpose | |-------|-------|---------| | Pass Rate | sum(playwright_test_passed) / sum(playwright_test_total) | Overall health | | Test Duration (p95) | histogram_quantile(0.95, playwright_test_duration_bucket) | Performance | | Failed Tests | playwright_test_failed{status="failed"} | Quick triage | | Flaky Tests | playwright_test_retries > 1 | Identify flakiness | | Slowest Tests | topk(10, playwright_test_duration) | Optimization targets |

Custom Metrics

// Export custom metrics from tests
import { test } from '@playwright/test';
import { metrics } from '@opentelemetry/api';

const testMeter = metrics.getMeter('playwright-tests');
const testDuration = testMeter.createHistogram('test.duration');
const testCounter = testMeter.createCounter('test.count');

test('custom metrics', async ({ page }) => {
  const start = Date.now();

  await page.goto('/');
  await page.click('.action');

  // Record custom metric
  testDuration.record(Date.now() - start, {
    test: 'homepage-action',
    browser: 'chromium',
  });

  testCounter.add(1, { status: 'passed' });
});

---

Railway Integration

Connect to your existing Railway observability stack.

Environment Variables

# Set in Railway service or .env
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.railway.internal:4318
OTEL_SERVICE_NAME=playwright-tests

Playwright Config

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['playwright-opentelemetry-reporter', {
      serviceName: process.env.OTEL_SERVICE_NAME || 'playwright-tests',
      endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
      headers: {
        // Add auth if needed
        'Authorization': `Bearer ${process.env.OTEL_AUTH_TOKEN}`,
      },
    }],
  ],
});

Verify Connection

# Test OTEL collector is receiving data
curl -X POST http://otel-collector.railway.internal:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'
# Should return 200 OK

---

Log Correlation

Send test logs to Loki alongside traces.

Winston + OTEL

// tests/fixtures/logger.ts
import winston from 'winston';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';

export const logger = winston.createLogger({
  transports: [
    new winston.transports.Console(),
    // Send to OTEL collector → Loki
    new OTLPLogTransport({
      endpoint: process.env.OTEL_ENDPOINT,
    }),
  ],
});

// In tests
import { logger } from './fixtures/logger';

test('with logging', async ({ page }) => {
  logger.info('Starting test', { test: 'checkout' });

  await page.goto('/checkout');
  logger.debug('Page loaded', { url: page.url() });

  await page.click('#submit');
  logger.info('Order submitted');
});

View Logs in Grafana

1. Grafana → Explore → Loki
2. Query: {service="playwright-tests"}
3. Filter: |= "error" or |json | level="error"

---

Alerting

Set up alerts for test failures.

Prometheus Alert Rules

# alerts/test-alerts.yml
groups:
  - name: playwright-tests
    rules:
      - alert: TestFailureRate
        expr: |
          (sum(rate(playwright_test_failed[5m])) / sum(rate(playwright_test_total[5m]))) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High test failure rate (>10%)"

      - alert: TestDurationSpike
        expr: |
          histogram_quantile(0.95, playwright_test_duration_bucket) > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Test duration p95 > 30 seconds"

Grafana Alert

1. Edit dashboard panel
2. Alert tab → Create alert
3. Condition: avg() of A is above 0.1
4. Notifications: Slack, email, PagerDuty

---

Workflow: Debugging with Traces

1. Test Fails in CI

❌ tests/checkout.spec.ts:15 - Order creation failed
   Expected: Order confirmation visible
   Actual: Error page shown

2. Find Trace in Grafana

Grafana → Explore → Tempo
Query: service.name="playwright-tests" AND name="checkout.spec.ts"

3. Analyze Trace

Browser click "Place Order" (50ms)
  └─ POST /api/orders (200ms)
       └─ Validate cart (10ms)
       └─ Process payment ❌ (timeout after 30s)
           └─ Payment gateway unreachable
       └─ Create order (skipped)

4. Root Cause

Found: Payment gateway timeout caused order failure.

Without traces: Would have debugged browser-side for hours.

---

Best Practices

1. Add Trace Context to Tests

import { context, trace } from '@opentelemetry/api';

test('with trace context', async ({ page }) => {
  const tracer = trace.getTracer('playwright');

  await tracer.startActiveSpan('checkout-flow', async (span) => {
    span.setAttribute('test.name', 'checkout');
    span.setAttribute('test.browser', 'chromium');

    await page.goto('/checkout');
    await page.click('#submit');

    span.setStatus({ code: 1 }); // OK
    span.end();
  });
});

2. Propagate Context to Backend

// Ensure trace ID passes from browser to backend
test('trace propagation', async ({ page }) => {
  // Get current trace ID
  const traceId = trace.getActiveSpan()?.spanContext().traceId;

  // Add to request headers
  await page.route('**/*', route => {
    route.continue({
      headers: {
        ...route.request().headers(),
        'traceparent': `00-${traceId}-${spanId}-01`,
      },
    });
  });
});

3. Tag Tests for Filtering

test('with tags', async ({ page }) => {
  const span = trace.getActiveSpan();
  span?.setAttribute('test.suite', 'e2e');
  span?.setAttribute('test.priority', 'critical');
  span?.setAttribute('test.feature', 'checkout');

  // Now filter in Grafana: test.feature="checkout"
});

---

References

• references/tracetest-integration.md - Full Tracetest setup
• references/otel-reporter-setup.md - OTEL reporter configuration
• references/grafana-dashboards.md - Dashboard creation guide
• dashboards/test-results-dashboard.json - Ready-to-import dashboard

---

Test observability transforms debugging from guesswork to precision - see the full picture from click to database.