[CHORE]: Helm Chart Test Harness & Red Hat chart-verifier #223
Description
🧭 Epic
Title: Helm Chart Test Harness & Certification
Goal: Ship the mcp-stack chart with a self-contained test suite (Helm test hooks + BATS acceptance tests) and an automated chart-verifier step so every release is cert-ready for Artifact Hub / Red Hat OpenShift.
Why now: We already lint on CI, but we still miss functional checks (helm test) and formal certification (chart-verifier). Adding both prevents broken releases, accelerates partner onboarding, and unlocks listing on the OpenShift chart repo.
See also: https://github.com/redhat-certification/chart-verifier
🧭 Type of Feature
- Reliability / Release hardening
🙋♂️ User Story 1 — Cluster-level smoke test
As a: Chart consumer
I want: make chart-test-kind to spin up a kind cluster, install the chart, run built-in Helm test hooks, then tear everything down
So that: I can validate the chart locally in < 2 minutes before submitting a PR.
✅ Acceptance Criteria
Scenario: One-liner chart test
Given Docker is running and kind is installed
When I execute `make chart-test-kind`
Then kind spins up a cluster called "mcp-kind"
And Helm installs mcp-stack with default values
And `helm test mcp-stack --logs` returns exit code 0
And the kind cluster is deleted afterwards🙋♂️ User Story 2 — Red Hat certification gate
As a: Release engineer
I want: make chart-verify to run chart-verifier (profile = redhat by default) and fail on any negative check
So that: every CI run produces a signed YAML report ready for partner submission.
✅ Acceptance Criteria
- CLI or Docker-wrapper returns 0 and writes
reports/mcp-stack-<chartVersion>.yaml. - Report shows pass for every mandatory check in the redhat profile.
- GitHub Actions artifact uploads the report on successful runs.
🙋♂️ User Story 3 — Continuous integration
As a: CI maintainer
I want: the lint job to call make charts (meta-target) which depends on chart-lint chart-test-kind chart-verify
So that: any Helm-related regression fails fast in the same pipeline as code & container checks.
🗺️ High-Level Implementation Notes
| Area / Component | Change | |
|---|---|---|
charts/mcp-stack/tests/ |
• Add a Connection test Pod (curl /health until 200) • Add a DB-migration completion test • Mark both with helm.sh/hook: test |
|
charts/mcp-stack/test/acceptance/ |
BATS tests 01_install.bats, 02_ingress.bats, 03_cleanup.bats |
|
Makefile (repo root) |
```make .PHONY: chart-lint chart-test chart-test-kind chart-verify charts chart-lint: ; helm lint charts/mcp-stack chart-test: ; helm test $$(helm ls -q |
grep mcp-stack) chart-test-kind: ; ./scripts/test_kind.sh chart-verify: ; chart-verifier verify charts/mcp-stack --profile redhat --output reports charts: chart-lint chart-test-kind chart-verify ``` |
scripts/test_kind.sh |
1) create kind cluster 2) helm upgrade --install 3) helm test 4) delete cluster |
|
| GitHub Actions | Add a matrix job runs-on: ubuntu-latest steps: make charts |
|
| Chart files | Update values.yaml examples so the Helm tests work without extra secrets |
|
| Docs | Extend chart README: “Running the test suite & certification” |
📦 New Make Targets
| Target | Purpose / Notes |
|---|---|
make values-schema |
(exists—just ensure it lands in charts/mcp-stack/values.schema.json) |
make chart-lint |
helm lint charts/mcp-stack |
make chart-test |
Run helm test against a current release (assumes cluster already exists) |
make chart-test-kind |
End-to-end test in a throw-away kind cluster |
make chart-verify |
chart-verifier verify charts/mcp-stack --profile redhat (Docker if USE_DOCKER=true) |
make charts |
Meta-target: chart-lint chart-test-kind chart-verify |
🔄 Alternatives Considered
| Option | Pros | Cons |
|---|---|---|
| Minikube-based CI tests | Mirrors prod clusters | Costly, slow, needs cloud creds |
Only helm lint + template diff |
Fast, already in CI | Doesn’t catch runtime or OpenShift specifics |
| Skip chart-verifier, rely on manual submission | Less tooling | Delays releases, easy to |