stolostron/learn-release

Learn Submariner Release Process

Teach users about the Submariner release process. Use $ARGUMENTS to determine what to explain.

$ARGUMENTS

---

What to Teach

| Argument | Content | | -------- | ------- | | (none) | Show this menu with examples | | overview | The big picture below | | step N | Explain one step (1-20, including 3b, 5b, 10b, 13b, 16b, 18b) | | all | Walk through all steps briefly |

The Big Picture

Submariner releases 9 container images through Konflux to Red Hat's registry. The process has 4 phases:

1. Setup (Y-stream only): Create release branch, configure Konflux pipelines
2. Build: Fix policy violations, scan for CVEs, cut upstream release, update bundle
3. Stage: Create release, add notes, publish to stage registry, QE tests
4. Prod: After QE approval, publish to production registry

Y-stream (0.21→0.22): New minor version. Run all steps starting from Step 1. Z-stream (0.21.1→0.21.2): Patch release. Skip to Step 4 (branch already exists).

Gates: Setup (Y-stream) → Builds + EC → CVE triage → Release notes → Stage: Stage release → FBC update + builds + EC → FBC stage releases → QE approval → Prod: Prod release (same snapshot) → FBC prod releases

Key Concepts

Build:

• Component: Konflux build unit. Each produces one container image. Submariner has 9 components across 5 repos.
• Hermetic: Isolated builds with pre-fetched dependencies (Go mods, RPMs). Required by EC for reproducibility.
• SBOM: Software Bill of Materials. Lists all dependencies. Required by EC for security/compliance.
• Multi-arch: Builds for x86_64, aarch64, ppc64le, s390x. Requires Customer Portal activation key for RPM access.

Release artifacts:

• Snapshot: Immutable component references (image SHAs). Event types: push (merge) or pull_request.
• Bundle: Container with operator metadata (CSV, CRDs). References 7 component images via relatedImages.
• FBC: File-Based Catalog. Operator index for OLM. See FBC section below.

FBC (File-Based Catalog):

• Purpose: Makes Submariner installable via OLM. Publishes to Red Hat's operator index (appears in OperatorHub).
• Template: catalog-template.yaml is source of truth. make build-catalogs generates 6 catalog-4-XX/ directories.
• Bundles: Version entries (e.g., submariner.v0.22.0) containing bundle image SHA and relatedImages (7 components).
• Channels: Update paths (e.g., stable-0.22). Users subscribe to a channel and get upgrades within it.
• Version pruning: drop-versions.json maps OCP versions to minimum Submariner versions (e.g., OCP 4.20 drops anything before 0.20).
• Image lifecycle: Bundles use temporary quay.io URLs (~90 day TTL). Step 20 updates to registry.redhat.io.

Konflux resources (K8s CRDs):

• Application: Groups components. submariner-0-X has 9 components; 6 submariner-fbc-4-XX apps each have 1 catalog.
• Snapshot: Immutable component references (image SHAs). Created after builds complete. Query with oc get snapshots.
• ReleasePlan: Links application to RPA. Lives in tenant namespace (submariner-tenant). Referenced by Release CRs.
• RPA: ReleasePlanAdmission. Defines release pipeline, EC policy, registry config. Lives in managed namespace (rhtap-releng-tenant).
• Release CR: Triggers publishing. References snapshot + releasePlan. Contains releaseNotes for advisories.
• EC Policy: Enterprise Contract policy. Checks labels, hermetic builds, CVEs, signatures, provenance. Violations block releases.
• activation-key Secret: Enables RPM prefetch in hermetic builds. Contains Customer Portal org ID and activation key.

Reference:

• Stage vs Prod: Stage publishes to registry.stage.redhat.io; prod to registry.redhat.io.
• Version formats: 0.21 (branch), 0-21 (Konflux names), v0.21.2 (Dockerfile labels), 0.21.2 (commits/PRs).
• ACM mapping: Submariner 0.X → ACM 2.(X-7). Example: 0.21 → 2.14, 0.22 → 2.15.
• Advisory types: RHSA (security/CVEs), RHBA (bug fixes), RHEA (enhancements). Type determines release notes format.

Steps

| Step | What happens | Y/Z | | ---- | ------------ | --- | | 1 | Create release-0.Y branches across all upstream repos | Y | | 2 | Add Konflux components, ReleasePlans, and RPAs in konflux-release-data | Y | | 3 | Customize bot-generated Tekton configs, set version labels | Y | | 3b | Update bundle SHAs from component builds, set up bundle pipeline | Y | | 4 | Fix Enterprise Contract violations in component and FBC repos | Y/Z | | 5 | Scan and fix CVEs: iterative fix→rebuild→rescan across components and libraries | Y/Z | | 5b | Bump Dockerfile version labels for the new patch version | Z | | 6 | Create git tags and publish images to quay.io/submariner | Y/Z | | 7 | Update bundle CSV with final component SHAs from snapshot | Y/Z | | 8 | Create stage Release CR YAML (no notes yet) | Y/Z | | 9 | Query Jira for CVEs (automatic) and issues (user selects), build releaseNotes | Y/Z | | 10 | Apply stage release to cluster via make apply | Y/Z | | 10b | Check Released=True, debug failures, retry if infra issue | Y/Z | | 11 | Update FBC catalogs with bundle SHA from stage registry | Y/Z | | 12 | Create 6 FBC stage release YAMLs (one per OCP 4.16-4.21) | Y/Z | | 13 | Apply all 6 FBC stage releases to cluster | Y/Z | | 13b | Verify all 6 FBC pipelines succeeded | Y/Z | | 14 | Create Jira ticket with stage catalog URLs for QE | Y/Z | | 15 | Copy stage YAML to prod, change releasePlan to prod | Y/Z | | 16 | Apply prod release to cluster | Y/Z | | 16b | Verify prod pipeline succeeded | Y/Z | | 17 | Copy 6 FBC stage YAMLs to prod, change releasePlans | Y/Z | | 18 | Apply all 6 FBC prod releases to cluster | Y/Z | | 18b | Verify all 6 FBC prod pipelines succeeded | Y/Z | | 19 | Share prod index URLs with QE - release complete | Y/Z | | 20 | Update FBC templates to use registry.redhat.io URLs | Y/Z |

Step Details

| Step | Details | | ---- | ------- | | 1 | Use releases repo tooling to create release-0.Y branches across 9 upstream repos. | | 2 | Add overlays (app, 9 components, ReleasePlans) and RPAs in konflux-release-data. ArgoCD syncs; triggers bot PRs. | | 3 | Customize Tekton configs: hermetic builds (Go mods, RPM lockfiles), multi-arch, SBOM. Version labels. 8 components, 5 repos. | | 3b | Two parts: (1) update bundle CSV with component SHAs from snapshot, (2) set up bundle Tekton pipeline. Components must build first. | | 4 | Enterprise Contract validates Red Hat release policies. Fix violations in component repos (9 images) and FBC repo (6 catalogs). | | 5 | Grype scans Go (7 repos), clair scans images. Fix→rebuild→rescan loop. Go stdlib CVEs fixed in Shipyard (base image for others). | | 5b | Bump version labels in 9 Dockerfiles across 5 repos. Bundle has 3 labels (csv-version, release, version). Rebuild triggers. | | 6 | Run releases repo tooling to create git tags and publish images to quay.io/submariner. Official upstream release. | | 7 | Update bundle CSV relatedImages with SHAs from latest passing Konflux snapshot. Must use registry.redhat.io URLs for EC. | | 8 | Create Release CR YAML: copy previous, update name/snapshot. Save to releases/0.X/stage/. Don't add notes yet. | | 9 | Query Jira: CVEs automatic, user selects other issues. RHSA/RHBA/RHEA based on content. Exclude submariner-addon. | | 10 | Run make apply to create Release CR on cluster. Pipeline publishes 9 images to registry.stage.redhat.io. | | 10b | Check Released condition. If failed: check ManagedPipelineProcessed, get log URL, determine retry vs fix. Increment suffix. | | 11 | Update FBC catalogs in submariner-operator-fbc repo with bundle SHA from stage registry. Wait ~15-30 min for rebuilds. | | 12 | Find passing FBC snapshots (push events only). Verify bundle SHA matches across all 6 catalogs. Create 6 Release YAMLs. | | 13 | Apply 6 FBC releases with make apply. Each publishes catalog to stage index for its OCP version. | | 13b | Check all 6 Released conditions. Same debug process as 10b. All must succeed before QE handoff. | | 14 | Extract catalog URLs from snapshots. Create Jira ticket for QE with 6 URLs. Wait for QE approval before prod. | | 15 | Copy stage YAML to prod directory. Change name (stage→prod) and releasePlan (stage-0-X→prod-0-X). Same snapshot/notes. | | 16 | Apply prod release. Pipeline publishes to registry.redhat.io (production). Same 9 images as stage. | | 16b | Verify prod pipeline succeeded. Same debug process as 10b. | | 17 | Copy 6 FBC stage YAMLs to prod directories. Change names and releasePlans. Same snapshots - catalog URLs work for both. | | 18 | Apply 6 FBC prod releases. Publishes catalogs to production indices (registry.redhat.io/redhat/redhat-operator-index). | | 18b | Verify all 6 succeeded. Release is now live in production OperatorHub. | | 19 | Extract index URLs from release status. Notify QE. Submariner 0.X.Y production release complete. | | 20 | Optional cleanup: update FBC templates to use registry.redhat.io URLs. Prevents breakage when quay.io images expire. |

Repos

Each step's workflow is in .agents/workflows/<step-name>.md. When it says "follow docs in X repo", read that repo's workflow docs.

Note: Branch in parentheses (devel for submariner-io repos, main for others).

| Repo | Local | Docs | Purpose | | ---- | ----- | ---- | ------- | | This repo | ~/konflux/submariner-release-management | .agents/workflows/ (main) | Release orchestration | | submariner-io/releases | ~/go/src/submariner-io/releases | README.md (devel) | Branch creation, tags | | submariner-io/submariner-operator | ~/go/src/submariner-io/submariner-operator | .agents/workflows/ (devel) | Operator + bundle | | submariner-io/submariner | ~/go/src/submariner-io/submariner | .agents/workflows/ (devel) | Gateway, globalnet, route-agent | | submariner-io/lighthouse | ~/go/src/submariner-io/lighthouse | .agents/workflows/ (devel) | Agent, coredns | | submariner-io/shipyard | ~/go/src/submariner-io/shipyard | .agents/workflows/ (devel) | Nettest | | submariner-io/subctl | ~/go/src/submariner-io/subctl | .agents/workflows/ (devel) | Subctl CLI | | stolostron/submariner-operator-fbc | ~/konflux/submariner-operator-fbc | .agents/workflows/ (main) | FBC catalogs (6 OCP) | | konflux-release-data (GitLab) | ~/konflux/konflux-release-data | tenants-config/.../CLAUDE.md (main) | Konflux tenant config | | konflux-ci/docs | ~/konflux/konflux-ci/docs | modules/ (main) | Konflux platform docs | | rhtap-ec-policy | ~/konflux/konflux-ci/rhtap-ec-policy | data/ (main) | EC policy definitions | | users-docs (GitLab) | ~/konflux/users-docs | docs/modules/ (main) | Konflux user guides |