openshift-eng/prow-job-analyze-test-failure

Prow Job Analyze Test Failure

This skill analyzes test failures by downloading Prow CI artifacts, checking test logs, inspecting resources and events, analyzing test source code, and optionally integrating cluster diagnostics from must-gather data.

Prerequisites

Identical with "prow-job-analyze-resource" skill.

Input Format

The user will provide:

1. Prow job URL - gcsweb URL containing test-platform-results/

   • Example: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_hypershift/6731/pull-ci-openshift-hypershift-main-e2e-aws/1962527613477982208
   • URL may or may not have trailing slash

2. Test name (optional) - specific test name that failed

   • When provided, focus the analysis on that test's stack trace and logs
   • When omitted, analyze all failed CI steps by inspecting the JUnit XML, build logs, and step artifacts to identify the root cause — this is the common case for multi-step CI workflows (e.g. HyperShift, bare-metal OADP jobs) where the failure is in a step rather than a named unit test
   • Examples:
     • TestKarpenter/EnsureHostedCluster/ValidateMetricsAreExposed
     • TestCreateClusterCustomConfig
     • The openshift-console downloads pods [apigroup:console.openshift.io] should be scheduled on different nodes

3. Optional flags (optional):

   • --fast - Skip must-gather extraction and analysis; continue with log/JUnit/step-artifact analysis only (scope is preserved: test-level when test-name is provided, step-level across all failed CI steps when omitted)

Implementation Steps

Step 1: Parse and Validate URL

Use the "Parse and Validate URL" steps from "prow-job-analyze-resource" skill

Step 2: Create Working Directory

1. Check for existing artifacts first

   • Check if .work/prow-job-analyze-test-failure/{build_id}/logs/ directory exists and has content
   • If it exists with content:
     • Use AskUserQuestion tool to ask:
       • Question: "Artifacts already exist for build {build_id}. Would you like to use the existing download or re-download?"
       • Options:
         • "Use existing" - Skip to step Analyze Test Failure
         • "Re-download" - Continue to clean and re-download
     • If user chooses "Re-download":
       • Remove all existing content: rm -rf .work/prow-job-analyze-test-failure/{build_id}/logs/
       • Also remove tmp directory: rm -rf .work/prow-job-analyze-test-failure/{build_id}/tmp/
       • This ensures clean state before downloading new content
     • If user chooses "Use existing":
       • Skip directly to Step 4 (Analyze Test Failure)
       • Still need to download prowjob.json if it doesn't exist

2. Create directory structure

   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/tmp
   

   • Use .work/prow-job-analyze-test-failure/ as the base directory (already in .gitignore)
   • Use build_id as subdirectory name
   • Create logs/ subdirectory for all downloads
   • Create tmp/ subdirectory for temporary files (intermediate JSON, etc.)
   • Working directory: .work/prow-job-analyze-test-failure/{build_id}/

Step 3: Download and Validate prowjob.json

Use the fetch-prowjob-json skill to fetch the prowjob.json for this job. See plugins/ci/skills/fetch-prowjob-json/SKILL.md for complete implementation details.

1. Fetch prowjob.json using the Prow job URL (convert to gcsweb URL per the fetch-prowjob-json skill)
2. Save locally to .work/prow-job-analyze-test-failure/{build_id}/logs/prowjob.json
3. Parse and validate
   • Search for pattern: --target=([a-zA-Z0-9-]+) in the ci-operator args
   • If not found:
     • Display: "This is not a ci-operator job. The prowjob cannot be analyzed by this skill."
     • Explain: ci-operator jobs have a --target argument specifying the test target
     • Exit skill
4. Extract target name and JOB_NAME
   • Capture the target value (e.g., e2e-aws-ovn) from the --target= arg
   • Extract JOB_NAME from .spec.job in prowjob.json (the artifact directory key):

     JOB_NAME=$(jq -r '.spec.job' .work/prow-job-analyze-test-failure/{build_id}/logs/prowjob.json)
     

   • Note: on PR jobs {target} and {JOB_NAME} often differ. All artifact-path lookups (JUnit XML, step logs, step artifacts) must use {JOB_NAME}, not {target}

Step 4: Analyze Test Failure

Step 4.0: Detect Aggregated Jobs

Aggregated jobs run the same job in parallel (typically 10 times) and perform statistical analysis of test results. Detect aggregation by checking for aggregated- prefix in the job name or an aggregator container/step in prowjob.json.

If the job is aggregated, the failure modes are different from normal jobs:

1. Statistically significant test failure — The test itself fails frequently enough across runs to be flagged as a regression. This is a real regression that needs investigation.

2. Insufficient completed runs — Not enough runs completed successfully to perform the statistical test (e.g., only 5 of 10 jobs produced results). This manifests as mass test failures across many unrelated tests. The root cause is whatever prevented the other runs from completing — this could be infrastructure issues, install failures, or an actual product bug causing crashes. Investigate the underlying job runs that did not complete to determine why.

3. Non-deterministic test presence — A test only ran in a small subset of completed jobs, even though the other jobs completed successfully. The failure message will say something like "Passed 1 times, failed 0 times, skipped 0 times: we require at least 6 attempts to have a chance at success". Every test must produce results in every job run; it is a bug if it does not. This is a regression — someone introduced a test that doesn't produce results deterministically. Investigate which test is non-deterministic and why it only runs in some jobs.

When analyzing an aggregated job failure, first determine which failure mode applies before diving into individual test analysis. For mode 2, focus on why runs failed rather than individual test results. For mode 3, investigate why the test only ran in some jobs.

Finding underlying job run URLs:

The aggregated junit XML contains links to every underlying job run. Download it from:

gs://test-platform-results/{bucket-path}/artifacts/release-analysis-aggregator/openshift-release-analysis-aggregator/artifacts/release-analysis-aggregator/{job-name}/{payload-tag}/junit-aggregated.xml

Each <testcase> element has a <system-out> with YAML-formatted data including passes:, failures:, and skips: lists. Each entry has:

• jobrunid: The build ID of the underlying job run
• humanurl: Prow URL for the job run (e.g., https://prow.ci.openshift.org/view/gs/test-platform-results/logs/{job-name}/{jobrunid})
• gcsartifacturl: Direct link to GCS artifacts

Use the humanurl links to investigate individual job run failures with the normal (non-aggregated) analysis steps below.

Step 4.1: Download build-log.txt

gcloud storage cp gs://test-platform-results/{bucket-path}/build-log.txt .work/prow-job-analyze-test-failure/{build_id}/logs/build-log.txt --no-user-output-enabled

Step 4.2: Parse and validate

• Read .work/prow-job-analyze-test-failure/{build_id}/logs/build-log.txt

Branch on whether {test_name} was provided:

Branch A: test_name provided

• Search build-log.txt for the exact test name string
• Gather the stack trace and surrounding context for that specific test
• Store the single failing test's error output for use in Steps 4.4 and 4.9

Branch B: test_name NOT provided (multi-step discovery)

This is the common case for multi-step CI workflows (e.g., HyperShift, bare-metal, OADP jobs) where failures occur in CI steps rather than named unit tests.

1. Discover failed CI steps from JUnit XML

   Use {JOB_NAME} (from Step 3, extracted from .spec.job) for all GCS artifact paths — not {target}. On PR jobs these values differ and {target} will miss the artifacts.

   Search for JUnit XML files under the artifacts directory:

   gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/**/junit*.xml" 2>/dev/null
   

   Download all found JUnit XML files:

   gcloud storage cp "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/**/junit*.xml" \
     .work/prow-job-analyze-test-failure/{build_id}/logs/ --no-user-output-enabled --recursive 2>/dev/null || true
   

   Parse each downloaded XML file and collect every <testcase> element where:

   • A <failure> or <error> child element is present, OR
   • The <testcase> has attribute status="failed"

   For each failed testcase record:

   • step_name: value of classname or name attribute (whichever identifies the CI step)
   • failure_message: text content of the <failure> or <error> element
   • junit_file: path of the XML file it came from

2. Classify each failed step by phase

   The ci-operator JUnit XML (junit_operator.xml) includes phase-level testcases:

   • "Run multi-stage test pre phase" — setup/installation steps
   • "Run multi-stage test test phase" — functional test steps
   • "Run multi-stage test post phase" — gather/cleanup steps

   Use the phase entries to classify each failed step_name:

   • Look at the failed phase-level testcases (those with "pre phase", "test phase", or "post phase" in their name) — these tell you which phase failed
   • Cross-reference the individual step testcases against their phase:
     • Steps in the pre phase → installation/setup steps
     • Steps in the test phase → functional test steps
     • Steps in the post phase → gather/cleanup steps (rarely the root cause)

   If the phase-level testcases are absent from the JUnit XML (some jobs omit them), fall back to the ci-operator-step-graph.json artifact, which lists all steps with their dependencies and timing — use execution order and naming conventions as a secondary signal. When still ambiguous, prefer classifying as a test step so must-gather analysis is not skipped.

3. Route each failed step to the appropriate analysis path

   • pre phase steps (installation) → invoke the ci:analyze-prow-job-install-failure skill for each such step, passing the same Prow job URL. Must-gather is not attempted for pre phase failures: must-gather requires a live apiserver, and when the installation phase fails the cluster is not fully up, so collection will fail. Capture the delegated output and store it as a per-step entry in the shared results collection used by Step 5 and Step 5.5:

     {
       step_name:      {step_name},
       type:           "installation",
       summary:        {one-line summary from ci:analyze-prow-job-install-failure},
       evidence:       {key error / stack trace from delegated analysis},
       recommendation: {recommended action from delegated analysis}
     }
     

   • test phase steps (functional tests) → proceed with the iteration in item 4 below (download step log, artifacts, extract stack trace) and include must-gather analysis (Steps 4.4–4.9). Store findings in the same per-step results collection:

     {
       step_name:      {step_name},
       type:           "test",
       summary:        {one-line failure summary},
       evidence:       {stack trace / error output},
       recommendation: {recommended action}
     }
     

   • post phase steps (gather/cleanup) → treat as informational. Download the step log and note the failure in the report, but do not perform must-gather analysis for post-phase failures alone (they are usually a consequence of earlier failures).

4. Iterate over each test phase step (pre and post phase steps already routed above)

   For each step_name in the test phase, perform the following sub-steps:

   a. Download step-specific log

   gcloud storage cp \
     "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/{step_name}/build-log.txt" \
     .work/prow-job-analyze-test-failure/{build_id}/logs/{step_name}-build-log.txt \
     --no-user-output-enabled 2>/dev/null || true
   

   If the step log is not found at that path, fall back to scanning build-log.txt for lines mentioning {step_name} and collect surrounding context (±50 lines).

   b. Download step-specific artifacts

   gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/{step_name}/" \
     2>/dev/null
   

   Download any relevant artifacts (e.g., *.json, *.yaml, events*.txt) to .work/prow-job-analyze-test-failure/{build_id}/logs/{step_name}/.

   c. Extract stack trace and context

   • Scan the step log for panic traces, FAIL, Error:, fatal, or assertion failures
   • Collect the full stack trace block (from the triggering line to the end of the trace)
   • Note the failure message from the JUnit XML as additional context

   d. Store per-step findings Record for each step:

   • step_name
   • failure_message (from JUnit XML)
   • stack_trace (from step log or build-log.txt context)
   • artifacts (list of downloaded artifact paths)

5. Produce a multi-step failure summary

   After iterating all failed steps, produce an ordered list of findings:

   Failed Steps Discovered:
   1. {step_name_1}: {one-line failure summary}
   2. {step_name_2}: {one-line failure summary}
   ...
   

   This list is used in Step 4.4 (evidence gathering) and Step 5 (report) to drive per-step analysis rather than singular test analysis.

Step 4.3: Examine intervals files for cluster activity during E2E failures

• Search recursively for E2E timeline artifacts (known as "interval files") within the bucket-path:

  gcloud storage ls 'gs://test-platform-results/{bucket-path}/**/e2e-timelines_spyglass_*json'
  

• The files can be nested at unpredictable levels below the bucket-path
• There could be as many as two matching files
• Download all matching interval files (use the full paths from the search results):

  gcloud storage cp gs://test-platform-results/{bucket-path}/**/e2e-timelines_spyglass_*.json .work/prow-job-analyze-test-failure/{build_id}/logs/ --no-user-output-enabled
  

• If the wildcard copy doesn't work, copy each file individually using the full paths from the search results
• Scan interval files for test failure timing:
  • Look for intervals where source = "E2ETest" and message.annotations.status = "Failed"
  • Note the from and to timestamps on this interval - this indicates when the test was running
• Scan interval files for related cluster events:
  • Look for intervals that overlap the timeframe when the failed test was running
  • Filter for intervals with:
    • level = "Error" or level = "Warning"
    • source = "OperatorState"
  • These events may indicate cluster issues that caused or contributed to the test failure

Step 4.3b: Check for Known Symptom Labels

The CI system may attach symptom labels to job runs — machine-detected patterns (e.g., "test failures during high CPU events") stored as JSON artifacts. These are not root causes but provide useful environmental context that may help explain failures when no other cause is found.

1. List the job_labels directory

   gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/job_labels/" 2>/dev/null
   

   • If the directory does not exist or returns an error, skip this step silently

2. Download any JSON symptom files (exclude label-summary.html)

   gcloud storage cp "gs://test-platform-results/{bucket-path}/artifacts/job_labels/*.json" \
     .work/prow-job-analyze-test-failure/{build_id}/logs/job_labels/ --no-user-output-enabled 2>/dev/null || true
   

3. Parse symptom labels — each JSON file describes a detected symptom with a summary and explanation. Collect all symptom summaries for inclusion in the report.

4. Use symptoms as investigative context — symptoms are environmental observations, NOT definitive causes. They should inform your investigation (e.g., if 2 tests failed while high CPU was measured, CPU pressure could explain the failures if no other cause is found) but you must still perform thorough root cause analysis. Include them in the "Known Symptoms Seen" section of the report.

Step 4.4: Gather initial evidence

• Analyze stack traces from build-log.txt
• Analyze related code in the code repository
• Store artifacts from Prow CI job (json/yaml files) related to the failure under .work/prow-job-analyze-test-failure/{build_id}/tmp
• Store logs under .work/prow-job-analyze-test-failure/{build_id}/logs/
• Collect evidence from logs and events and other json/yaml files
• Check for CI step script errors: If the error is a scripting issue in a CI step (e.g., unbound variable, syntax error, missing command, bad exit code from a shell script) rather than a product bug, check for recent commits to that step's script in the openshift/release repository. This is part of evidence gathering — identifying the responsible PR early informs the rest of the analysis. Include the responsible PR in your evidence.

Step 4.4b: Investigate crash-looping or failing containers

Be tenacious. When the build log or test output mentions crash-looping pods, container restarts, or deployments not becoming ready, you MUST trace the failure to its root cause. Never stop at "containers are crash-looping" — find out why.

Pursue all available log sources:

• Must-gather data (Step 4.6–4.7): Contains pod YAMLs with containerStatuses (exitCode, lastState.terminated.reason, restartCount), container logs (current and previous), events, and operator conditions. This is often the richest source for diagnosing crash-looping pods.
• Gather-extra / gather-audit logs: Some jobs run additional gather steps that collect extra diagnostics. List the step artifacts directory for gather steps beyond gather-must-gather (e.g., gather-extra, gather-audit-logs) and download relevant logs.
• Step-level build logs: The build log for the failing test step (downloaded in Step 4.2) often contains error output, stack traces, and timeout messages that reference specific pods or containers.
• Events from interval files (Step 4.3): Operator state transitions and warning events correlated with the failure window.

Follow the dependency chain. Always trace upstream to the originating error rather than stopping at the first symptom.

Step 4.5: Check for Must-Gather Availability

1. Parse optional flags

   • Parse user input for --fast flag
   • If --fast flag present:
     • Skip must-gather detection and analysis entirely
     • Proceed directly to Step 5 — scope is preserved:
       • If test_name was provided → produce singular test report (Branch A)
       • If test_name was omitted → produce multi-step report (Branch B)
     • Do NOT prompt user about must-gather

2. Extract actual test name from prowjob.json

   The artifacts directory uses the test name from prowjob.json, NOT the full URL path.

   # Extract test name from prowjob.json (e.g., "e2e-aws-operator-serial-ote")
   JOB_NAME=$(jq -r '.spec.job' .work/prow-job-analyze-test-failure/{build_id}/logs/prowjob.json)
   
   # Note: For PR jobs, TARGET contains the full PR path like:
   #   pr-logs/pull/openshift_service-ca-operator/306/pull-ci-openshift-service-ca-operator-main-e2e-aws-operator-serial-ote
   # But artifacts are stored under just the test name:
   #   e2e-aws-operator-serial-ote
   

3. Detect must-gather archive (only if --fast not present)

   Use JOB_NAME (not TARGET) for artifact paths.

   HyperShift jobs may have different must-gather patterns:

   Pattern 1: Unified Archive (dump-management-cluster)

   • Single archive with both management and hosted cluster data
   • Used by: hypershift-aws-e2e-external workflow

   Pattern 2: Dual Archives (gather-must-gather + dump)

   • Standard must-gather for management cluster
   • Separate hypershift-dump for additional data (may or may not have hosted cluster)
   • Used by: hypershift-kubevirt-e2e-aws workflow

   Pattern 3: Standard Only (gather-must-gather)

   • Standard OpenShift must-gather only
   • No HyperShift-specific dump

   Detection logic:

   # Check for Pattern 1: Unified archive (dump-management-cluster)
   UNIFIED_DUMP=$(gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/$JOB_NAME/dump-management-cluster/artifacts/artifacts.tar*" 2>/dev/null | head -1 || true)
   
   # Check for Pattern 2/3: Standard must-gather
   STANDARD_MG=$(gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/$JOB_NAME/gather-must-gather/artifacts/must-gather.tar" 2>/dev/null || true)
   
   # Check for Pattern 2: Additional hypershift-dump (multiple possible locations)
   # Use wildcards to match all current and future HyperShift dump patterns:
   # 1. **/artifacts/hypershift-dump.tar (covers dump/, hypershift-mce-dump/, etc.)
   # 2. **/artifacts/**/hostedcluster.tar (covers all E2E test patterns)
   HYPERSHIFT_DUMP=""
   for pattern in \
       "**/artifacts/hypershift-dump.tar" \
       "**/artifacts/**/hostedcluster.tar"; do
       FOUND=$(gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/$JOB_NAME/$pattern" 2>/dev/null | head -1 || true)
       if [ -n "$FOUND" ]; then
           HYPERSHIFT_DUMP="$FOUND"
           break
       fi
   done
   
   # Determine pattern and check for hosted cluster data
   if [ -n "$UNIFIED_DUMP" ]; then
     # Pattern 1: Unified archive
     PATTERN="unified"
   
     # Download temporarily to check for hosted cluster data
     TMP_CHECK="/tmp/check-unified-$$.tar"
     gcloud storage cp "$UNIFIED_DUMP" "$TMP_CHECK" --no-user-output-enabled
   
     # Check if archive contains hostedcluster-* directory
     HAS_HOSTED_CLUSTER=$(tar -tf "$TMP_CHECK" 2>/dev/null | grep -q "hostedcluster-" && echo "true" || echo "false")
     rm -f "$TMP_CHECK"
   
   elif [ -n "$STANDARD_MG" ] && [ -n "$HYPERSHIFT_DUMP" ]; then
     # Pattern 2: Dual archives
     PATTERN="dual"
   
     # Download hypershift-dump temporarily to check for hosted cluster
     TMP_CHECK="/tmp/check-dump-$$.tar"
     gcloud storage cp "$HYPERSHIFT_DUMP" "$TMP_CHECK" --no-user-output-enabled
   
     # Check if hypershift-dump contains hostedcluster-* directory
     HAS_HOSTED_CLUSTER=$(tar -tf "$TMP_CHECK" 2>/dev/null | grep -q "hostedcluster-" && echo "true" || echo "false")
     rm -f "$TMP_CHECK"
   
   elif [ -n "$STANDARD_MG" ]; then
     # Pattern 3: Standard must-gather only
     PATTERN="standard"
     HAS_HOSTED_CLUSTER=false
   
   else
     # No must-gather found
     PATTERN="none"
     HAS_HOSTED_CLUSTER=false
   fi
   

   Possible outcomes:

   • PATTERN="none": No must-gather found → Skip to Step 5 (silent, expected for some jobs)
   • PATTERN="standard": Standard OpenShift must-gather only → Extract single cluster
   • PATTERN="unified" + HAS_HOSTED_CLUSTER=false: HyperShift unified archive with management only
   • PATTERN="unified" + HAS_HOSTED_CLUSTER=true: HyperShift unified archive with both clusters
   • PATTERN="dual" + HAS_HOSTED_CLUSTER=false: Two archives but hosted cluster not in hypershift-dump
   • PATTERN="dual" + HAS_HOSTED_CLUSTER=true: Two archives with hosted cluster in hypershift-dump

   Important Pattern Notes:

   • Pattern 1 (unified): Management at logs/artifacts/output/, hosted at logs/artifacts/output/hostedcluster-{name}/
   • Pattern 2 (dual): Management in standard must-gather, hosted MAY be in hypershift-dump.tar or hostedcluster.tar (not guaranteed)
     • Wildcard patterns match all current and future dump locations
   • Pattern 3 (standard): Management cluster only, no HyperShift-specific data

4. Ask user if they want must-gather analysis

   • Only if must-gather(s) were found and --fast not present
   • Use AskUserQuestion tool:
     • Question: "Must-gather data is available. Include cluster diagnostics in the analysis?"
     • Header: "Must-gather"
     • Options:
       • Label: "Yes - Extract and analyze must-gather (Recommended)" Description: "Provides cluster-level diagnostics that may reveal root causes (pods, operators, nodes, events). Takes additional time to download and analyze."
       • Label: "No - Skip must-gather (faster)" Description: "Only analyze test-level artifacts (build-log, intervals). Faster but may miss cluster-level issues."
   • If user chooses "No", skip to Step 5

Step 4.6: Extract Must-Gather (Conditional)

Only if user chose "Yes" in Step 4.5:

1. Determine extraction strategy

   • If single must-gather detected → extract to must-gather/logs/
   • If dual must-gather detected (HyperShift) → extract both:
     • Management cluster → must-gather-mgmt/logs/
     • Hosted cluster → must-gather-hosted/logs/

2. Check for existing extraction

   For single must-gather:

   • Check if .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/ exists with content

   For dual must-gather (HyperShift):

   • Check if .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/ exists with content
   • Check if .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/ exists with content

   If either exists:

   • Use AskUserQuestion tool:
     • Question: "Must-gather already extracted for this build. Use existing data?"
     • Header: "Reuse"
     • Options:
       • Label: "Use existing" Description: "Reuse previously extracted must-gather data (faster)"
       • Label: "Re-extract" Description: "Download and extract fresh must-gather data"
   • If "Re-extract":
     • rm -rf .work/prow-job-analyze-test-failure/{build_id}/must-gather*/
     • Continue to step 3 (fresh extraction)
   • If "Use existing":
     • Validate content directories exist and are not empty (see validation in step 4.6)
     • If validation fails, fall back to re-extraction
     • If validation succeeds, skip to Step 4.7

3. Create must-gather directories

   Based on PATTERN from Step 4.5.3:

   For Pattern 3 (standard only):

   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather/tmp
   

   For Pattern 1 (unified) or Pattern 2 (dual):

   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp
   

4. Download must-gather archives

   Use JOB_NAME (from Step 4.5.2) for artifact paths, not {target}:

   For Pattern 3 (standard only):

   gcloud storage cp "$STANDARD_MG" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather/tmp/must-gather.tar \
     --no-user-output-enabled
   

   For Pattern 1 (unified):

   # Download unified archive (contains both management and hosted cluster data)
   gcloud storage cp "$UNIFIED_DUMP" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/unified-dump.tar \
     --no-user-output-enabled
   

   For Pattern 2 (dual):

   # Download management cluster must-gather
   gcloud storage cp "$STANDARD_MG" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/must-gather.tar \
     --no-user-output-enabled
   
   # Download hypershift-dump (may or may not contain hosted cluster)
   gcloud storage cp "$HYPERSHIFT_DUMP" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp/hypershift-dump.tar \
     --no-user-output-enabled
   

5. Extract archives

   For Pattern 3 (standard only):

   # Use existing extract_archives.py script for standard must-gather
   python3 plugins/ci/skills/prow-job-extract-must-gather/extract_archives.py \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather/tmp/must-gather.tar \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs
   

   For Pattern 1 (unified):

   # Extract unified archive to temporary location
   TMP_EXTRACT=".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/extracted"
   mkdir -p "$TMP_EXTRACT"
   
   # Handle both .tar and .tar.gz
   if [[ "$UNIFIED_DUMP" == *.tar.gz ]]; then
     tar -xzf .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/unified-dump.tar -C "$TMP_EXTRACT"
   else
     tar -xf .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/unified-dump.tar -C "$TMP_EXTRACT"
   fi
   
   # Find the output directory (may be at logs/artifacts/output or just output)
   OUTPUT_DIR=$(find "$TMP_EXTRACT" -type d -name "output" | head -1)
   
   if [ -z "$OUTPUT_DIR" ]; then
     echo "ERROR: Could not find output directory in unified dump"
     rm -rf "$TMP_EXTRACT"
     # Clear variables to prevent subsequent usage
     HAS_HOSTED_CLUSTER="false"
     unset HOSTED_DIR
     unset OUTPUT_DIR
     # Skip to Step 5 - no must-gather analysis possible
   else
     # Move management cluster data (root level in output/)
     # Exclude hostedcluster-* directories
     for item in "$OUTPUT_DIR"/*; do
       if [ -e "$item" ] && [[ ! "$(basename "$item")" =~ ^hostedcluster- ]]; then
         mv "$item" .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/
       fi
     done
   
     # Move hosted cluster data (hostedcluster-* subdirectory)
     if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
       HOSTED_DIR=$(find "$OUTPUT_DIR" -maxdepth 1 -type d -name "hostedcluster-*" | head -1)
       if [ -n "$HOSTED_DIR" ]; then
         mv "$HOSTED_DIR"/* .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/
         echo "✓ Hosted cluster data extracted from unified archive"
       else
         echo "WARNING: Expected hosted cluster data but hostedcluster-* directory not found"
       fi
     fi
   
     # Cleanup temporary extraction directory
     rm -rf "$TMP_EXTRACT"
   fi
   

   For Pattern 2 (dual):

   # Extract management cluster must-gather (standard format)
   python3 plugins/prow-job/skills/prow-job-extract-must-gather/extract_archives.py \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/must-gather.tar \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs
   
   # Extract hypershift-dump (may contain hosted cluster)
   TMP_EXTRACT=".work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp/extracted"
   mkdir -p "$TMP_EXTRACT"
   tar -xf .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp/hypershift-dump.tar -C "$TMP_EXTRACT"
   
   if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
     # Look for hostedcluster-* directory in dump
     HOSTED_DIR=$(find "$TMP_EXTRACT" -maxdepth 2 -type d -name "hostedcluster-*" | head -1)
     if [ -n "$HOSTED_DIR" ]; then
       mv "$HOSTED_DIR"/* .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/
       echo "✓ Hosted cluster data extracted from hypershift-dump"
     else
       echo "WARNING: HAS_HOSTED_CLUSTER=true but no hostedcluster-* directory found in dump"
       echo "Hypershift-dump likely contains only management cluster data"
       HAS_HOSTED_CLUSTER=false  # Update flag since hosted cluster not actually present
     fi
   else
     echo "INFO: Hypershift-dump does not contain hosted cluster data (management cluster only)"
   fi
   
   # Cleanup temporary extraction directory
   rm -rf "$TMP_EXTRACT"
   

6. Locate and validate content directories

   For Pattern 3 (standard only):

   # Check for content/ directory first (renamed by extraction script)
   if [ -d ".work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/content" ]; then
       MUST_GATHER_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/content"
   else
       # Fall back to finding the directory containing -ci- (e.g., registry-build09-ci-...)
       MUST_GATHER_PATH=$(find .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs -maxdepth 1 -type d -name "*-ci-*" | head -1)
   fi
   
   # Validate MUST_GATHER_PATH is set and directory exists
   if [ -z "$MUST_GATHER_PATH" ] || [ ! -d "$MUST_GATHER_PATH" ]; then
       echo "ERROR: Must-gather content directory not found after extraction"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   elif [ -z "$(ls -A "$MUST_GATHER_PATH" 2>/dev/null)" ]; then
       echo "ERROR: Must-gather content directory is empty"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   else
       echo "✓ Must-gather content located at: $MUST_GATHER_PATH"
       # Continue to Step 4.7 with MUST_GATHER_PATH set
   fi
   

   For Pattern 1 (unified) or Pattern 2 (dual):

   # Management cluster validation
   if [ "$PATTERN" = "unified" ]; then
       # Pattern 1: Data extracted directly to logs/ directory
       MUST_GATHER_MGMT_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs"
   else
       # Pattern 2: Standard must-gather extraction (look for content/ or hash directory)
       if [ -d ".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/content" ]; then
           MUST_GATHER_MGMT_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/content"
       else
           MUST_GATHER_MGMT_PATH=$(find .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs -maxdepth 1 -type d -name "*-ci-*" | head -1)
       fi
   fi
   
   # Validate management cluster path
   if [ -z "$MUST_GATHER_MGMT_PATH" ] || [ ! -d "$MUST_GATHER_MGMT_PATH" ]; then
       echo "ERROR: Management cluster directory not found"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   elif [ -z "$(ls -A "$MUST_GATHER_MGMT_PATH" 2>/dev/null)" ]; then
       echo "ERROR: Management cluster directory is empty"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   else
       echo "✓ Management cluster data located at: $MUST_GATHER_MGMT_PATH"
   fi
   
   # Hosted cluster validation - only if HAS_HOSTED_CLUSTER is true
   if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
       MUST_GATHER_HOSTED_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs"
   
       # Validate hosted cluster path
       if [ ! -d "$MUST_GATHER_HOSTED_PATH" ]; then
           echo "WARNING: Hosted cluster directory not found (expected based on archive detection)"
           MUST_GATHER_HOSTED_PATH=""  # Clear the path
       elif [ -z "$(ls -A "$MUST_GATHER_HOSTED_PATH" 2>/dev/null)" ]; then
           echo "WARNING: Hosted cluster directory is empty"
           MUST_GATHER_HOSTED_PATH=""  # Clear the path
       else
           echo "✓ Hosted cluster data located at: $MUST_GATHER_HOSTED_PATH"
       fi
   else
       echo "✓ Management cluster must-gather located at: $MUST_GATHER_MGMT_PATH"
   fi
   
   # Only validate hosted cluster if HAS_HOSTED_CLUSTER is true
   if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
       if [ -z "$MUST_GATHER_HOSTED_PATH" ] || [ ! -d "$MUST_GATHER_HOSTED_PATH" ]; then
           echo "ERROR: Hosted cluster must-gather content directory not found"
       elif [ -z "$(ls -A "$MUST_GATHER_HOSTED_PATH" 2>/dev/null)" ]; then
           echo "ERROR: Hosted cluster must-gather content directory is empty"
       else
           echo "✓ Hosted cluster must-gather located at: $MUST_GATHER_HOSTED_PATH"
           echo "✓ Hosted cluster namespace: $HOSTED_NAMESPACE"
       fi
   fi
   

Step 4.7: Analyze Must-Gather (Conditional)

Only if Step 4.6 completed successfully:

1. Locate must-gather-analyzer scripts

   The must-gather plugin provides analysis scripts. Locate the scripts directory:

   # Try to find the must-gather-analyzer scripts in common locations
   for SEARCH_PATH in \
       "plugins/must-gather/skills/must-gather-analyzer/scripts" \
       "~/.claude/plugins/cache/*/plugins/must-gather/skills/must-gather-analyzer/scripts" \
       "$(find ~ -type d -path "*/must-gather/skills/must-gather-analyzer/scripts" 2>/dev/null | head -1)"; do
       SCRIPTS_DIR=$(eval echo "$SEARCH_PATH")
       if [ -d "$SCRIPTS_DIR" ] && [ -f "$SCRIPTS_DIR/analyze_clusteroperators.py" ]; then
           break
       fi
       SCRIPTS_DIR=""
   done
   
   if [ -z "$SCRIPTS_DIR" ]; then
       echo "WARNING: Must-gather analysis scripts not found."
       echo "Install the must-gather plugin: /plugin install must-gather@ai-helpers"
       # Continue to Step 5 without cluster analysis
   fi
   

2. Run targeted cluster diagnostics

   Focus on issues relevant to test failures (not full cluster analysis).

   For single must-gather (standard OpenShift):

   # Core diagnostics (only if scripts and path are available)
   if [ -n "$SCRIPTS_DIR" ] && [ -n "$MUST_GATHER_PATH" ]; then
       python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" "$MUST_GATHER_PATH"
       python3 "$SCRIPTS_DIR/analyze_pods.py" "$MUST_GATHER_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_nodes.py" "$MUST_GATHER_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_events.py" "$MUST_GATHER_PATH" --type Warning --count 50
   else
       echo "WARNING: Skipping must-gather analysis (scripts or path not available)"
   fi
   

   For dual must-gather (HyperShift):

   # Management cluster diagnostics (only if scripts and path are available)
   if [ -n "$SCRIPTS_DIR" ] && [ -n "$MUST_GATHER_MGMT_PATH" ]; then
       echo "=== Analyzing Management Cluster ==="
       python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" "$MUST_GATHER_MGMT_PATH"
       python3 "$SCRIPTS_DIR/analyze_pods.py" "$MUST_GATHER_MGMT_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_nodes.py" "$MUST_GATHER_MGMT_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_events.py" "$MUST_GATHER_MGMT_PATH" --type Warning --count 50
   else
       echo "WARNING: Skipping management cluster analysis (scripts or path not available)"
   fi
   
   # Hosted cluster diagnostics (only if scripts and path are available)
   if [ -n "$SCRIPTS_DIR" ] && [ -n "$MUST_GATHER_HOSTED_PATH" ]; then
       echo "=== Analyzing Hosted Cluster (Namespace: $HOSTED_NAMESPACE) ==="
       python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" "$MUST_GATHER_HOSTED_PATH"
       python3 "$SCRIPTS_DIR/analyze_pods.py" "$MUST_GATHER_HOSTED_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_nodes.py" "$MUST_GATHER_HOSTED_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_events.py" "$MUST_GATHER_HOSTED_PATH" --type Warning --count 50
   else
       echo "INFO: Skipping hosted cluster analysis (scripts or path not available)"
   fi
   

3. Run conditional diagnostics based on test context

   # Network diagnostics (if test name suggests network issues)
   if [[ "$JOB_NAME" =~ network|ovn|sdn|connectivity|route|ingress|egress ]]; then
       if [ -n "$MUST_GATHER_PATH" ]; then
           python3 "$SCRIPTS_DIR/analyze_network.py" "$MUST_GATHER_PATH"
       fi
       if [ -n "$MUST_GATHER_MGMT_PATH" ]; then
           echo "=== Management Cluster Network ==="
           python3 "$SCRIPTS_DIR/analyze_network.py" "$MUST_GATHER_MGMT_PATH"
       fi
       if [ -n "$MUST_GATHER_HOSTED_PATH" ]; then
           echo "=== Hosted Cluster Network ==="
           python3 "$SCRIPTS_DIR/analyze_network.py" "$MUST_GATHER_HOSTED_PATH"
       fi
   fi
   
   # etcd diagnostics (if test name suggests control-plane issues)
   if [[ "$JOB_NAME" =~ etcd|apiserver|control-plane|kube-apiserver ]]; then
       if [ -n "$MUST_GATHER_PATH" ]; then
           python3 "$SCRIPTS_DIR/analyze_etcd.py" "$MUST_GATHER_PATH"
       fi
       if [ -n "$MUST_GATHER_MGMT_PATH" ]; then
           echo "=== Management Cluster etcd ==="
           python3 "$SCRIPTS_DIR/analyze_etcd.py" "$MUST_GATHER_MGMT_PATH"
       fi
       if [ -n "$MUST_GATHER_HOSTED_PATH" ]; then
           echo "=== Hosted Cluster etcd ==="
           python3 "$SCRIPTS_DIR/analyze_etcd.py" "$MUST_GATHER_HOSTED_PATH"
       fi
   fi
   

   See plugins/must-gather/skills/must-gather-analyzer/SKILL.md for all available analysis scripts.

4. Capture analysis output

   • Store script output for correlation in Step 4.8
   • Keep management and hosted cluster outputs separate
   • Use in final report in Step 5

Step 4.8: Correlate Cluster Issues with Test Failure

Only if Step 4.7 completed:

1. Temporal correlation

   • From Step 4 (interval files), you identified when the test was running (from/to timestamps)
   • Review cluster operator conditions, pod events, and warning events for timing alignment
   • Identify cluster issues that occurred during or shortly before test failure (±5 minutes)
   • Example: "Test failed at 10:23:45. Network operator became degraded at 10:23:12."

   For HyperShift (dual must-gather):

   • Correlate issues from BOTH management and hosted clusters
   • Note which cluster (management vs hosted) each issue occurred in
   • Example: "Test failed at 10:23:45. Hosted cluster network operator became degraded at 10:23:12."

2. Component correlation

   • Map test failure to cluster components:
     • Namespace correlation: Test runs in specific namespace → check for pod failures in that namespace
       • For HyperShift: Tests typically run in hosted cluster namespace (e.g., clusters-{namespace})
     • Test assertions correlation: Test type suggests affected components
       • Network tests → network operator status, CNI pods, network policies
       • Storage tests → storage operator, CSI pods, PVs/PVCs
       • API tests → kube-apiserver pods, API server operator
     • Stack trace correlation: Error messages in stack trace → related Kubernetes resources
       • "connection refused" → check pod restarts, network issues
       • "timeout" → check node pressure, resource constraints
       • "not found" → check resource deletion events

   For HyperShift (dual must-gather):

   • Management cluster issues typically affect:
     • HostedControlPlane pods (kube-apiserver, etcd, etc. in clusters-{namespace} namespace)
     • HyperShift operator
     • Management cluster nodes hosting control plane
   • Hosted cluster issues typically affect:
     • Worker node pods
     • Cluster operators
     • Application workloads

3. Generate correlated insights

   • Create specific, actionable correlations like:
     • "Test failed at {time}. {Operator} became degraded at {time} with reason: {reason}"
     • "Pod crash-looping in test namespace: {namespace}/{pod-name}"
     • "Node {node-name} reported {condition} at {time}, test pod was scheduled on this node"
     • "Warning event: {event-message} at {time} (during test execution)"

   For HyperShift (dual must-gather):

   • Prefix correlations with cluster type:

     • "[Management Cluster] HostedControlPlane pod restarting: clusters-{namespace}/kube-apiserver-*"
     • "[Hosted Cluster] Network operator degraded with reason: {reason}"

   • Cross-cluster correlations:

     • "Management cluster node pressure → Hosted cluster control plane unavailable"
     • "HyperShift operator error → HostedControlPlane rollout failed"

   • Store these insights for inclusion in Step 4.9 root cause determination

Step 4.9: Determine Root Cause

Synthesize all gathered evidence to determine the most likely root cause for the test failure.

CRITICAL: Be tenacious — trace symptoms back to root cause. Never stop at high-level symptoms like "nodes didn't join", "operator unavailable", or "containers are crash-looping". Trace backwards through the dependency chain (pod statuses → container logs → originating error) until you find the specific, actionable root cause.

1. Analyze all available evidence

   • Stack traces from Step 4.2
   • Test code analysis from Step 4.4
   • Interval file events from Step 4.3
   • Known symptom labels from Step 4.3b (if available — use as supporting context, not as root cause)
   • Cluster diagnostics from Step 4.7 (if available)
   • Correlations from Step 4.8 (if available)

2. Prioritize evidence based on temporal proximity

   • Issues occurring during test execution time (from interval files) are most relevant
   • Cluster issues that occurred shortly before test failure (±5 minutes) are highly relevant
   • Pre-existing cluster issues may be contributing factors but not root causes

3. Generate root cause hypothesis

   • Primary cause: The most direct, immediate cause of test failure
   • Contributing factors: Cluster or environmental issues that enabled the failure
   • Evidence summary: Key evidence supporting the hypothesis

   For HyperShift jobs:

   • Distinguish between management cluster vs hosted cluster root causes
   • Identify cross-cluster dependencies (e.g., management node pressure → hosted control plane unavailable)

4. Formulate actionable recommendations

   • What needs to be fixed (code, configuration, cluster state)
   • Where to look for more information
   • Suggested next steps for debugging or resolution

Step 5: Present Results to User

1. Display structured summary with enhanced formatting

   Branch on whether test_name was provided (mirrors Step 4.2 branching):

   • Branch A (test_name provided): Use the singular {test_name} report shape below.
   • Branch B (test_name NOT provided): Use the multi-step report shape further below, iterating over each failed step discovered in Step 4.2 Branch B and aggregating their findings into a combined report.

   For single must-gather or no must-gather (Branch A — test_name provided):

   # Test Failure Analysis Complete
   
   ## Job Information
   - **Prow Job**: {prowjob-name}
   - **Build ID**: {build_id}
   - **Target**: {target}
   - **Test**: {test_name}
   
   ## Known Symptoms Seen
   *(Only if symptom labels were found in Step 4.3b — omit section entirely if none)*
   - {symptom summary}: {symptom explanation}
   > **Note**: Symptoms are machine-detected environmental observations, not definitive causes. They add context to help explain failures when correlated with other evidence.
   
   ## Test Failure Analysis
   
   ### Error
   {error message from stack trace}
   
   ### Summary
   {failure analysis from stack trace and code}
   
   ### Evidence
   {evidence from build-log.txt and interval files}
   
   ### Additional Evidence
   {additional evidence from logs/events}
   
   ---
   
   ## Cluster Diagnostics
   *(Only if must-gather was analyzed)*
   
   ### Cluster Operators
   {output from analyze_clusteroperators.py}
   
   ### Problematic Pods
   {output from analyze_pods.py --problems-only}
   
   ### Node Issues
   {output from analyze_nodes.py --problems-only}
   
   ### Recent Warning Events
   {output from analyze_events.py}
   
   ### Network Analysis
   *(Only if network-related test)*
   {output from analyze_network.py}
   
   ### etcd Analysis
   *(Only if etcd-related test)*
   {output from analyze_etcd.py}
   
   ---
   
   ## Correlation
   *(Only if must-gather was analyzed)*
   
   ### Timeline
   - **Test started**: {from timestamp}
   - **Test failed**: {to timestamp}
   - **Cluster events during test**:
     - {cluster-event} at {timestamp}
     - {cluster-event} at {timestamp}
   
   ### Affected Components
   - {affected operators/pods/nodes}
   
   ### Root Cause Hypothesis
   {correlated analysis combining test-level and cluster-level evidence}
   
   ---
   
   ## Artifacts
   - **Test artifacts**: `.work/prow-job-analyze-test-failure/{build_id}/logs/`
   - **Must-gather**: `.work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/` *(if extracted)*
   

   For dual must-gather (HyperShift):

   # Test Failure Analysis Complete (HyperShift)
   
   ## Job Information
   - **Prow Job**: {prowjob-name}
   - **Build ID**: {build_id}
   - **Target**: {target}
   - **Test**: {test_name}
   - **Hosted Cluster Namespace**: {HOSTED_NAMESPACE}
   
   ## Known Symptoms Seen
   *(Only if symptom labels were found in Step 4.3b — omit section entirely if none)*
   - {symptom summary}: {symptom explanation}
   > **Note**: Symptoms are machine-detected environmental observations, not definitive causes. They add context to help explain failures when correlated with other evidence.
   
   ## Test Failure Analysis
   
   ### Error
   {error message from stack trace}
   
   ### Summary
   {failure analysis from stack trace and code}
   
   ### Evidence
   {evidence from build-log.txt and interval files}
   
   ### Additional Evidence
   {additional evidence from logs/events}
   
   ---
   
   ## Management Cluster Diagnostics
   
   ### Cluster Operators
   {output from analyze_clusteroperators.py for management cluster}
   
   ### Problematic Pods
   {output from analyze_pods.py --problems-only for management cluster}
   
   ### Node Issues
   {output from analyze_nodes.py --problems-only for management cluster}
   
   ### Recent Warning Events
   {output from analyze_events.py for management cluster}
   
   ### Network Analysis
   *(Only if network-related test)*
   {output from analyze_network.py for management cluster}
   
   ### etcd Analysis
   *(Only if etcd-related test)*
   {output from analyze_etcd.py for management cluster}
   
   ---
   
   ## Hosted Cluster Diagnostics
   
   **Namespace**: `{HOSTED_NAMESPACE}`
   
   ### Cluster Operators
   {output from analyze_clusteroperators.py for hosted cluster}
   
   ### Problematic Pods
   {output from analyze_pods.py --problems-only for hosted cluster}
   
   ### Node Issues
   {output from analyze_nodes.py --problems-only for hosted cluster}
   
   ### Recent Warning Events
   {output from analyze_events.py for hosted cluster}
   
   ### Network Analysis
   *(Only if network-related test)*
   {output from analyze_network.py for hosted cluster}
   
   ### etcd Analysis
   *(Only if etcd-related test)*
   {output from analyze_etcd.py for hosted cluster}
   
   ---
   
   ## Correlation
   
   ### Timeline
   - **Test started**: {from timestamp}
   - **Test failed**: {to timestamp}
   - **Management cluster events during test**:
     - {cluster-event} at {timestamp}
   - **Hosted cluster events during test**:
     - {cluster-event} at {timestamp}
   
   ### Affected Components
   
   **Management Cluster**:
   - {affected operators/pods/nodes}
   
   **Hosted Cluster**:
   - {affected operators/pods/nodes}
   
   ### Root Cause Hypothesis
   {correlated analysis combining:
   - Test-level evidence
   - Management cluster diagnostics
   - Hosted cluster diagnostics
   - Cross-cluster interactions}
   
   ---
   
   ## Artifacts
   - **Test artifacts**: `.work/prow-job-analyze-test-failure/{build_id}/logs/`
   - **Management cluster must-gather**: `.work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/`
   - **Hosted cluster must-gather**: `.work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/`
   

   For Branch B (test_name NOT provided — multi-step failure report):

   Iterate over the ordered list of failed steps discovered in Step 4.2 Branch B and produce one section per step, then aggregate into a combined root cause section.

   # Test Failure Analysis Complete (Multi-Step)
   
   ## Job Information
   - **Prow Job**: {prowjob-name}
   - **Build ID**: {build_id}
   - **Target**: {target}
   - **Failed Steps**: {count of discovered failed steps}
   
   ## Known Symptoms Seen
   *(Only if symptom labels were found in Step 4.3b — omit section entirely if none)*
   - {symptom summary}: {symptom explanation}
   > **Note**: Symptoms are machine-detected environmental observations, not definitive causes.
   
   ---
   
   ## Failed Step Analyses
   
   *(Repeat the following block for each failed step discovered in Step 4.2 Branch B)*
   
   ### Step: {step_name}
   
   #### Error
   {failure_message from JUnit XML for this step}
   
   #### Summary
   {analysis of this step's stack trace and log context}
   
   #### Evidence
   {stack trace extracted from {step_name}-build-log.txt or build-log.txt context}
   
   #### Artifacts Examined
   - `.work/prow-job-analyze-test-failure/{build_id}/logs/{step_name}-build-log.txt`
   - `.work/prow-job-analyze-test-failure/{build_id}/logs/{step_name}/` *(step artifacts if downloaded)*
   
   ---
   
   ## Cluster Diagnostics
   *(Only if must-gather was analyzed — same structure as Branch A; include per-cluster
   sections as appropriate for standard or HyperShift patterns)*
   
   ---
   
   ## Aggregated Root Cause
   
   ### Failed Steps Summary
   | Step | One-line Failure |
   |------|-----------------|
   | {step_name_1} | {summary} |
   | {step_name_2} | {summary} |
   | ... | ... |
   
   ### Timeline
   *(Emit one entry per failed step. If interval data cannot be mapped to a specific
   step, omit the Timeline section entirely rather than showing a misleading single window.)*
   
   - **{step_name_1}**: started {from timestamp} — failed {to timestamp}
     - Cluster events during this window: {cluster-event} at {timestamp}
   - **{step_name_2}**: started {from timestamp} — failed {to timestamp}
     - Cluster events during this window: {cluster-event} at {timestamp}
   - *(repeat for each failed step with mappable interval data)*
   
   ### Root Cause Hypothesis
   {synthesized root cause drawn from all per-step findings (install + test steps),
   per-step interval events, symptom labels, and cluster diagnostics; clearly
   identify whether one step is the root cause or whether multiple independent
   failures occurred}
   
   ### Recommendations
   - {actionable next step 1}
   - {actionable next step 2}
   
   ---
   
   ## Artifacts
   - **Test artifacts**: `.work/prow-job-analyze-test-failure/{build_id}/logs/`
   - **Must-gather**: `.work/prow-job-analyze-test-failure/{build_id}/must-gather*/logs/` *(if extracted)*
   

2. Display completion message

   ✅ Analysis complete!
   📄 Report: .work/prow-job-analyze-test-failure/{build_id}/analysis.md
   
   💡 Tip: The Markdown report can be copied directly into JIRA Description field
   

Error Handling

Handle errors in the same way as "Error handling" in "prow-job-analyze-resource" skill, with these additional must-gather-specific cases:

1. Must-gather not available

   • If gcloud storage ls returns 404 for must-gather.tar, this is expected (not all jobs have must-gather)
   • Silently skip must-gather analysis - do NOT warn the user
   • Continue to Step 5 preserving scope: Branch A (singular test) or Branch B (multi-step) as determined by whether test_name was provided

2. Must-gather extraction fails

   • If download or extraction fails, warn the user but continue with test analysis
   • Display: "WARNING: Must-gather extraction failed: {error}. Continuing without cluster diagnostics."
   • Continue to Step 5 preserving scope: Branch A (singular test) or Branch B (multi-step) as determined by whether test_name was provided

3. Analysis scripts not found

   • If find command returns empty (no scripts found), warn the user
   • Display: "WARNING: Must-gather analysis scripts not installed. Install the must-gather plugin from openshift-eng/ai-helpers for cluster diagnostics."
   • Continue to Step 5 preserving scope: Branch A (singular test) or Branch B (multi-step) as determined by whether test_name was provided

4. Partial analysis script failures

   • If one script fails (non-zero exit code), continue with other scripts
   • Capture and report which analyses succeeded/failed
   • Display failed analyses as: "WARNING: {script-name} analysis failed: {error}"
   • Include successful analyses in final report

5. Empty analysis results

   • If a script runs successfully but produces no output or no issues found
   • Display: "{Analysis-type}: No issues detected"
   • This is informational, not an error

Performance Considerations

Follow the instructions in "Performance Considerations" in "prow-job-analyze-resource" skill