Build dedicated release-tools container image for op1st Tekton release pipeline #156

New issue

Closed

opened 2026-05-25 13:58:38 +00:00 by goern · 1 comment

goern commented

2026-05-25 13:58:38 +00:00

(Migrated from codeberg.org)

Problem

Three live op1st Tekton release attempts (v2.24.0, v2.24.1, v2.24.2) failed at distinct steps because tools (syft, goreleaser, cosign, node, mcpb) are installed at Step runtime in stock public images with mismatched assumptions:

v2.24.0 — Tekton Step containers share workspaces but not /usr/local/bin/. Tools installed in install-* Steps disappeared before the release Step started → goreleaser: command not found (exit 127).
v2.24.1 — GOCACHE / GOMODCACHE set to paths inside the workspace mount made git status dirty → GoReleaser refused to publish.
v2.24.2 — ghcr.io/sigstore/cosign/cosign:vX is distroless (no shell), so Tekton script: blocks fail with fork/exec /tekton/scripts/script-0-XXXX: no such file or directory. Release shipped 13/14 assets (no .sig).

Each failure burned a release tag. The pattern is structural: every Task improvises its own toolchain.

Proposed change

Build a dedicated release-tools OCI image with all release tooling baked in, pinned, signed, and SBOM-attached. The three consuming Tekton Tasks switch to referencing this image and drop the runtime installs (separate follow-up change).

Shape

Base: Project Hummingbird — registry.access.redhat.com/hi/go:<ver>-builder. Red Hat hardened image with bash + dnf available for layering additional tools.
Bundled: Go toolchain, syft, goreleaser, cosign, Node 22, npm, jq, curl, ca-certificates. All pinned via build ARGs sourced from VERSIONS.md.
Isolation: every introduced path lives under image/release-tools/ or .tekton/release-tools/. Lift to a separate repo = git mv of those two trees + 5 enumerated operator actions.
CEL gates: PR build fires only when files under image/release-tools/ or .tekton/release-tools/ change. Tag publish fires only on refs/tags/release-tools/v*. Won't collide with v* forgejo-mcp tags.
Signing: cosign sign-by-digest first, then promote human-readable tag (closes TOCTOU advertisement window). Image pipeline is fail-closed; release artifact pipeline stays fail-open (asymmetry documented in ADR).
Supply chain: npm package pinned via committed package-lock.json + npm ci --ignore-scripts + SHA256 in VERSIONS.md. No live npm registry pull at build.
SBOM: CycloneDX via syft, attached as OCI referrer.
Versioning: image-API semver (MAJOR = base swap / removed tool / broken CLI; MINOR = added tool / Hummingbird bump / Go minor; PATCH = tool patches). Avoids Renovate treadmill.

Why a separate issue and not just the PR

PR #155 is the OpenSpec design + adversarial review only. This issue exists so the design + rationale + adversarial findings are discoverable without reading 4 markdown files. It's the user-facing summary; PR #155 is the artifact.

Linked PRs and beads

PR: #155 — OpenSpec change release-tools-image (proposal-only; no Containerfile yet)
Bead: forgejo-mcp-1b4 (P0)
Adversarial review: openspec/changes/release-tools-image/review.md — 14 critiques, 9 patched in-tree, 5 future-work beads (forgejo-mcp-aun, -3h2, -j52, -46j) filed blocked on forgejo-mcp-1b4
Related:
- PR #150 — original Tekton release pipeline that surfaced these failures
- PR #151 — soft-disable Forgejo Actions release auto-trigger
- PR #152 — point cosign verify docs at op1st normative public key
- PR #153, #154 — Path B step-collapse + GOCACHE workarounds (will be undone once image lands)

Open questions for maintainers

Final registry path: quay.io/operate-first/release-tools, ghcr.io/operate-first/..., or codeberg.org/operate-first/...? Needs op1st-emea-b4mad coordination.
PaC controller in op1st-pipelines ≥ 0.22.0 (required for files.any CEL)? Verify before merging the implementation PR.
A8 versioning policy referee call (Go-bump-as-MINOR) — revisit after first 6 months of Renovate data if MINOR bumps cause unexpected consumer breakage.

Acceptance

PR #155 reviewed and merged (proposal lands)
Implementation PR opens with Containerfile + build.sh + two pipelines under image/release-tools/ and .tekton/release-tools/
release-tools/v1.0.0 tag pushed, publish pipeline produces a signed, SBOM-attached image at the agreed registry path
cosign verify --key <op1st-pub> <image-ref> succeeds
Follow-up change release-pipeline-use-release-tools-image rewrites the three consuming Tekton Tasks to reference the image
First clean end-to-end Tekton release counts toward cutover criteria (forgejo-mcp-gdz)

## Problem Three live op1st Tekton release attempts (`v2.24.0`, `v2.24.1`, `v2.24.2`) failed at distinct steps because tools (syft, goreleaser, cosign, node, mcpb) are installed at Step runtime in stock public images with mismatched assumptions: 1. **`v2.24.0`** — Tekton Step containers share workspaces but not `/usr/local/bin/`. Tools installed in `install-*` Steps disappeared before the `release` Step started → `goreleaser: command not found` (exit 127). 2. **`v2.24.1`** — `GOCACHE` / `GOMODCACHE` set to paths inside the workspace mount made `git status` dirty → GoReleaser refused to publish. 3. **`v2.24.2`** — `ghcr.io/sigstore/cosign/cosign:vX` is distroless (no shell), so Tekton `script:` blocks fail with `fork/exec /tekton/scripts/script-0-XXXX: no such file or directory`. Release shipped 13/14 assets (no `.sig`). Each failure burned a release tag. The pattern is structural: every Task improvises its own toolchain. ## Proposed change Build a dedicated **`release-tools`** OCI image with all release tooling baked in, pinned, signed, and SBOM-attached. The three consuming Tekton Tasks switch to referencing this image and drop the runtime installs (separate follow-up change). ### Shape - **Base**: [Project Hummingbird](https://hummingbird-project.io/) — `registry.access.redhat.com/hi/go:<ver>-builder`. Red Hat hardened image with bash + dnf available for layering additional tools. - **Bundled**: Go toolchain, syft, goreleaser, cosign, Node 22, npm, jq, curl, ca-certificates. All pinned via build ARGs sourced from `VERSIONS.md`. - **Isolation**: every introduced path lives under `image/release-tools/` or `.tekton/release-tools/`. Lift to a separate repo = `git mv` of those two trees + 5 enumerated operator actions. - **CEL gates**: PR build fires only when files under `image/release-tools/` or `.tekton/release-tools/` change. Tag publish fires only on `refs/tags/release-tools/v*`. Won't collide with `v*` forgejo-mcp tags. - **Signing**: cosign sign-by-digest first, then promote human-readable tag (closes TOCTOU advertisement window). Image pipeline is fail-closed; release artifact pipeline stays fail-open (asymmetry documented in ADR). - **Supply chain**: npm package pinned via committed `package-lock.json` + `npm ci --ignore-scripts` + SHA256 in `VERSIONS.md`. No live npm registry pull at build. - **SBOM**: CycloneDX via syft, attached as OCI referrer. - **Versioning**: image-API semver (MAJOR = base swap / removed tool / broken CLI; MINOR = added tool / Hummingbird bump / Go minor; PATCH = tool patches). Avoids Renovate treadmill. ## Why a separate issue and not just the PR PR #155 is the OpenSpec design + adversarial review only. This issue exists so the design + rationale + adversarial findings are discoverable without reading 4 markdown files. It's the user-facing summary; PR #155 is the artifact. ## Linked PRs and beads - **PR**: [#155](https://codeberg.org/goern/forgejo-mcp/pulls/155) — OpenSpec change `release-tools-image` (proposal-only; no Containerfile yet) - **Bead**: `forgejo-mcp-1b4` (P0) - **Adversarial review**: `openspec/changes/release-tools-image/review.md` — 14 critiques, 9 patched in-tree, 5 future-work beads (`forgejo-mcp-aun`, `-3h2`, `-j52`, `-46j`) filed blocked on `forgejo-mcp-1b4` - **Related**: - PR #150 — original Tekton release pipeline that surfaced these failures - PR #151 — soft-disable Forgejo Actions release auto-trigger - PR #152 — point cosign verify docs at op1st normative public key - PR #153, #154 — Path B step-collapse + GOCACHE workarounds (will be undone once image lands) ## Open questions for maintainers - Final registry path: `quay.io/operate-first/release-tools`, `ghcr.io/operate-first/...`, or `codeberg.org/operate-first/...`? Needs op1st-emea-b4mad coordination. - PaC controller in `op1st-pipelines` ≥ 0.22.0 (required for `files.any` CEL)? Verify before merging the implementation PR. - A8 versioning policy referee call (Go-bump-as-MINOR) — revisit after first 6 months of Renovate data if MINOR bumps cause unexpected consumer breakage. ## Acceptance - PR #155 reviewed and merged (proposal lands) - Implementation PR opens with Containerfile + build.sh + two pipelines under `image/release-tools/` and `.tekton/release-tools/` - `release-tools/v1.0.0` tag pushed, publish pipeline produces a signed, SBOM-attached image at the agreed registry path - `cosign verify --key <op1st-pub> <image-ref>` succeeds - Follow-up change `release-pipeline-use-release-tools-image` rewrites the three consuming Tekton Tasks to reference the image - First clean end-to-end Tekton release counts toward cutover criteria (`forgejo-mcp-gdz`)

goern commented

2026-05-25 16:56:03 +00:00

(Migrated from codeberg.org)

Implementation PR opened: #157 (both iterations — image source + Tekton pipelines — in a single PR). Local verifier (bash build.sh && bash verify.sh + yaml/pre-commit/openspec) all green. Pending operator coordination on registry path before first release-tools/vX.Y.Z tag push.

Implementation PR opened: #157 (both iterations — image source + Tekton pipelines — in a single PR). Local verifier (`bash build.sh && bash verify.sh` + yaml/pre-commit/openspec) all green. Pending operator coordination on registry path before first `release-tools/vX.Y.Z` tag push.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

agentic-forges/forgejo-mcp#156

No description provided.

Rows
Columns