openshift-eng/fetch-test-report

Fetch Test Report

This skill fetches a report for an OpenShift CI test by its full name using the Sippy tests API. It returns test metadata including the BigQuery/Component Readiness test ID, Jira component, pass rates for the current and previous reporting periods, and open bug counts. The open_bugs field counts Jira bugs that mention this test by name, which can help surface bugs that have been filed but not yet triaged in Component Readiness.

When to Use This Skill

Use this skill when you need to:

• Look up a test's Component Readiness / BigQuery test ID (test_id)
• Check current and previous pass rates for a test
• Find the Jira component associated with a test
• Determine if a test is flaking, failing, or stable
• Get run counts and failure/flake breakdowns for a test
• Check if there are open Jira bugs mentioning this test (may not yet be triaged in Component Readiness)
• See a per-variant breakdown of pass rates to identify if the test fails only in certain job types

Prerequisites

1. Network Access: The Sippy API must be accessible at https://sippy.dptools.openshift.org

   • No authentication required
   • Check: curl -s https://sippy.dptools.openshift.org/api/health?release=4.22

2. Python 3: Python 3.6 or later

   • Check: python3 --version
   • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Determine the Release

If the user did not specify a release, use the fetch-releases skill to determine the latest OCP release:

release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)

If the user specified a release, use that directly.

Step 2: Run the Python Script

The skill uses a Python script to query the Sippy tests API:

# Path to the Python script
script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py"

# Fetch test report in JSON format (collapsed — one row for the test across all variants)
python3 "$script_path" "<test_name>" --release "$release" --format json

# Or fetch as human-readable summary
python3 "$script_path" "<test_name>" --release "$release" --format summary

# Fetch per-variant breakdown (one row per variant combo — useful for identifying
# if the test is failing only in certain job types, e.g., aws vs metal, upgrade vs new-install)
python3 "$script_path" "<test_name>" --release "$release" --no-collapse --format json

Step 3: Parse the Output

The script outputs a JSON array (typically with one element for an exact name match):

# Store JSON output in a variable for processing
test_data=$(python3 "$script_path" "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" --format json)

# Extract specific fields using jq
test_id=$(echo "$test_data" | jq -r '.[0].test_id')
component=$(echo "$test_data" | jq -r '.[0].jira_component')
pass_rate=$(echo "$test_data" | jq -r '.[0].current_pass_percentage')
runs=$(echo "$test_data" | jq -r '.[0].current_runs')

API Details

Endpoint

GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}
GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}&collapse=false

When collapse=false is specified, the API returns one row per variant combination instead of a single collapsed row. Each row includes a variants array showing the specific variant combo (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]). This helps identify if the test is failing in certain types of jobs and not others.

Filter Format

The filter parameter is a JSON object using Sippy's standard filter syntax:

{
  "items": [
    {
      "columnField": "name",
      "operatorValue": "equals",
      "value": "<test_name>"
    }
  ]
}

Response Schema

The API returns a JSON array of test objects:

[
  {
    "id": 5987,
    "test_id": "openshift-tests:e8f7491a505095b6356dfd8d4cf7218b",
    "name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
    "suite_name": "openshift-tests",
    "variants": null,
    "jira_component": "kube-apiserver",
    "jira_component_id": "12367637",
    "current_successes": 3425,
    "current_failures": 22,
    "current_flakes": 0,
    "current_pass_percentage": 99.36,
    "current_failure_percentage": 0.64,
    "current_flake_percentage": 0,
    "current_working_percentage": 99.36,
    "current_runs": 3447,
    "previous_successes": 2570,
    "previous_failures": 43,
    "previous_flakes": 0,
    "previous_pass_percentage": 98.35,
    "previous_failure_percentage": 1.65,
    "previous_flake_percentage": 0,
    "previous_working_percentage": 98.35,
    "previous_runs": 2613,
    "net_failure_improvement": 1.01,
    "net_flake_improvement": 0,
    "net_working_improvement": 1.01,
    "net_improvement": 1.01,
    "tags": null,
    "open_bugs": 0
  }
]

Key Fields:

• id: Sippy PostgreSQL internal ID (less commonly used)
• test_id: BigQuery / Component Readiness test ID (format: suite:hash). This is the ID used with modern skills like fetch-regression-details and fetch-test-runs.
• name: Full test name
• suite_name: Test suite (e.g., openshift-tests)
• variants: Array of variant strings (only present when --no-collapse / collapse=false is used). Shows the specific variant combo for this row (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]).
• jira_component: Associated OCPBUGS Jira component
• jira_component_id: Jira component numeric ID
• current_*: Metrics for the current 7-day reporting period
• previous_*: Metrics for the 7 days before the current period
• net_working_improvement: Change in working percentage (positive = improving)
• open_bugs: Number of open Jira bugs that mention this test by name. This can surface bugs that have been filed but not yet triaged in Component Readiness. Useful for finding existing work before filing a duplicate.

Error Handling

Case 1: Sippy Not Reachable

python3 fetch_test_report.py "[sig-api-machinery] ..."
# Error: Failed to connect to Sippy API: [Errno 61] Connection refused
# Check network connectivity to sippy.dptools.openshift.org.

Case 2: No Test Found

If the test name doesn't match any test, the script returns an empty JSON array [] or "No tests found matching the given name." in summary mode.

Case 3: Missing Arguments

python3 fetch_test_report.py
# usage: fetch_test_report.py [-h] [--release RELEASE] [--format {json,summary}] test_name
# fetch_test_report.py: error: the following arguments are required: test_name

Exit Codes:

• 0: Success
• 1: Error (API error, network error, etc.)

Examples

Example 1: Fetch Test Report in JSON Format

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.22 --format json

Example 2: Get a Human-Readable Summary

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.22 --format summary

Expected Output:

Test: [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]
  Test ID:         openshift-tests:abc123def456
  Suite:           openshift-tests
  Jira Component:  kube-apiserver
  Open Bugs:       0

  Current Period (last 7 days):
    Runs:       3447
    Pass Rate:  99.36%
    Failures:   22 (0.64%)
    Flakes:     0 (0.00%)

  Previous Period (7 days before current):
    Runs:       2613
    Pass Rate:  98.35%
    Failures:   43 (1.65%)
    Flakes:     0 (0.00%)

  Trend:  improved (+1.01%)

Example 3: Fetch Per-Variant Breakdown

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.22 --no-collapse --format summary

This returns one row per variant combination, showing which specific job types the test is passing or failing in. Useful for regression analysis to determine if the failure is platform-specific, upgrade-specific, etc.

Example 4: Fetch Test Report for a Different Release

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.21 --format json

Notes

• The test name must be an exact match — use the full test name including sig prefix and suite tags
• The API is accessed via the production Sippy URL; no authentication required
• If --release is not specified, use the fetch-releases skill to determine the latest release
• current_* fields cover the last 7 days; previous_* fields cover the 7 days before that
• current_working_percentage = pass rate + flake rate (tests that ultimately passed, possibly after retries)
• The test_id field is the one used in Component Readiness URLs and with the fetch-regression-details and fetch-test-runs skills
• An empty result ([]) means no test matched — double-check the exact test name spelling

See Also

• Related Skill: fetch-releases (determines the latest OCP release)
• Related Skill: fetch-regression-details (uses test_id to fetch regression details)
• Related Skill: fetch-test-runs (uses test name to fetch individual test run results)
• Related Command: /ci:fetch-test-report (command that invokes this skill)
• Related Command: /ci:analyze-regression (full regression analysis workflow)