|
| 1 | +--- |
| 2 | +hip: "0025" |
| 3 | +title: "Better Support for Resource Creation Sequencing" |
| 4 | +authors: [ "Joe Beck <[email protected]>", "Evans Mungai <[email protected]>" ] |
| 5 | +created: "2025-05-21" |
| 6 | +type: "feature" |
| 7 | +status: "draft" |
| 8 | +--- |
| 9 | + |
| 10 | +## Abstract |
| 11 | + |
| 12 | +This HIP proposes a new feature set in Chart v3 to provide [Application Distributors](https://github.com/helm/community/blob/main/user-profiles.md#2-application-distributor)—a key Helm user profile—with a first-class mechanism for defining the deployment order of chart resources and subcharts. By default, Helm applies all rendered manifests simultaneously. This HIP introduces the ability for Helm to deploy resources in ordered batches and evaluate their readiness before proceeding. |
| 13 | + |
| 14 | +At a high level, this HIP proposes the following |
| 15 | +- Ability for chart authors to specify how to sequence deployment of **resources within a single chart** |
| 16 | +- Ability for chart authors to specify how to sequence **subcharts within a parent chart** |
| 17 | +- Ability for Helm operators and tool developers to enable sequencing behaviour using `--wait=ordered` CLI flag and `WaitStrategy=ordered` SDK parameter respectively |
| 18 | + |
| 19 | +The HIP only targets resources deployed in the Helm install phase. Resources deployed as hooks are not sequenced using changes proposed here. Any sequencing of hooks will still rely on using `"helm.sh/hook-weight"` annotations. Annotations added to resources in hooks will be ignored. |
| 20 | + |
| 21 | +## Motivation |
| 22 | + |
| 23 | +The driving motivator here is to allow application distributors to control what order resources are bundled and sent to the K8s API server, referred to as resource sequencing for the rest of this HIP. |
| 24 | + |
| 25 | +Today, to accomplish resource sequencing, there are currently two main options: using Helm hooks, or building the sequencing logic into the application itself (e.g., using startup code or init containers). The existing hooks and weights can be tedious to build and maintain for application distributors, and built-in app sequencing can unnecessarily increase complexity of a Helm application that needs to be maintained by application distributors. Helm, as a package manager, should provide built-in mechanisms for sequencing resource deployment, reducing reliance on complex hooks or in-application logic. This will significantly improve the application distributor's experience. |
| 26 | + |
| 27 | +Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed and this featureset would ideally address this. |
| 28 | + |
| 29 | +## Rationale |
| 30 | + |
| 31 | +The proposed design prioritizes simplicity and ease of use for both Helm developers and chart authors. It leverages familiar YAML patterns and Helm conventions, avoiding heavyweight solutions while offering powerful orchestration capabilities. |
| 32 | + |
| 33 | +## Specification |
| 34 | + |
| 35 | +At a high level, allow Chart Developers to assign named dependencies to both their Helm templated resources and Helm chart dependencies that Helm then uses to generate a deployment sequence at installation time. |
| 36 | + |
| 37 | +For Helm CLI, the `--wait=ordered` flag will enable sequencing where resources are applied in groups. SDK users will also be able to enable sequencing by setting a `WaitStrategy` field. By default, resources are all applied at once which is the same behaviour in Chart v2. |
| 38 | + |
| 39 | +Each release will store information of whether sequencing was used or not. This information is used when performing uninstalls and rollbacks. |
| 40 | + |
| 41 | +### Sequencing Execution Flow |
| 42 | + |
| 43 | +When sequencing is enabled, Helm installs resources in a structured order across both subcharts and resource-groups: |
| 44 | + |
| 45 | +**1. Subchart Ordering** |
| 46 | + |
| 47 | +Helm builds a dependency graph from definitions in `Chart.yaml`: |
| 48 | + |
| 49 | +* `helm.sh/depends-on/subcharts` key in `annotations` field |
| 50 | +* `depends-on` key on `dependencies` list entries |
| 51 | + |
| 52 | +Subcharts are installed in dependency order. Each subchart must be fully deployed and ready before its dependents begin. |
| 53 | + |
| 54 | +**2. Resource-Group Sequencing (Per Chart)** |
| 55 | + |
| 56 | +Within each chart (parent and subcharts), Helm builds a resource-group graph using: |
| 57 | + |
| 58 | +* `helm.sh/resource-group` |
| 59 | +* `helm.sh/depends-on/resource-groups` |
| 60 | + |
| 61 | +Resources in each group are deployed together, and Helm waits for all to be ready before continuing to the next group. |
| 62 | + |
| 63 | +**3. Unsequenced Resources** |
| 64 | + |
| 65 | +Resources that: |
| 66 | + |
| 67 | +* lack annotations, |
| 68 | +* depend on non-existent groups, or |
| 69 | +* belong to isolated groups |
| 70 | + |
| 71 | +will be deployed after all properly sequenced groups have been processed. |
| 72 | + |
| 73 | +If a resource includes sequencing annotations but falls into this unsequenced category due to misconfiguration (e.g., referencing missing groups), Helm will emit a warning to alert the user of the potential issue. |
| 74 | + |
| 75 | +*Additions to templates* |
| 76 | + |
| 77 | +The following annotations would be added to enable this. |
| 78 | +- `helm.sh/resource-group`: Annotation to declare a resource-group that a given resource belongs to. Any number of resources can belong to a group. A resource can only belong to one group. |
| 79 | +- `helm.sh/depends-on/resource-groups`: Annotation to declare resource-groups that must exist and in a ready state before this resource can be deployed. The order in which they are listed does not affect deployment sequencing. |
| 80 | + |
| 81 | +These annotations are only used for sequencing resources within the same chart. They do not influence or interact with resources across charts or subcharts. |
| 82 | + |
| 83 | +*Additions to Chart.yaml* |
| 84 | +* `helm.sh/depends-on/subcharts`: An annotation added to `Chart.yaml` to specify chart dependencies—identified by their `name` or `alias`—that must be fully deployed and in a ready state before the current chart resources can be installed. A dependent chart is considered ready only when all of its resources, including any defined sequencing, have been successfully deployed and marked ready. The order in which dependencies are listed has no effect on execution. |
| 85 | +- `depends-on`: A new field added to `Chart.yaml` `dependencies` fields that is meant to declare a list of subcharts, by `name` or `alias`, that need to be ready before the subchart in question get installed. This will be used to create a dependency graph for subcharts. |
| 86 | + |
| 87 | +The installation process would group resources in the same group and send them to the K8s API Server in one bundle, and once all resources are "ready", the next group would be installed. A resource-group would not be considered "ready" and the next group installed until all resources in that group are considered "ready". Readiness is described in a later section. A similar process would apply for upgrades. Uninstalls would function on the same resource-group order, but in reverse, where a resource-group is not uninstalled until all resource-groups that depend on it are first uninstalled. Upgrades would follow the same order as installation. |
| 88 | + |
| 89 | +#### Template examples: |
| 90 | +```yaml |
| 91 | +# resource 1 |
| 92 | +metadata: |
| 93 | + name: db-service |
| 94 | + annotations: |
| 95 | + helm.sh/resource-group: database |
| 96 | +--- |
| 97 | +# resource 2 |
| 98 | +metadata: |
| 99 | + name: my-app |
| 100 | + annotations: |
| 101 | + helm.sh/resource-group: app |
| 102 | + helm.sh/depends-on/resource-groups: ["database", "queue"] |
| 103 | +--- |
| 104 | +# resource 3 |
| 105 | +metadata: |
| 106 | + name: queue-processor |
| 107 | + annotations: |
| 108 | + helm.sh/resource-group: queue |
| 109 | + helm.sh/depends-on/resource-groups: ["another-group"] |
| 110 | + |
| 111 | +``` |
| 112 | +In this example, Helm would be responsible for resolving the annotations on these three resources and deploy all resources in the following order. Resources in `database` and `queue` resource-groups would be deployed at the same time. They would need to be ready before attempting to deploy `app` resource-group: |
| 113 | + |
| 114 | +``` |
| 115 | +"database" group: [db-service] |
| 116 | + || |
| 117 | +"queue" group: || [queue-processor] |
| 118 | + || || |
| 119 | + || || |
| 120 | + \/ \/ |
| 121 | + \\ // |
| 122 | + \\ // |
| 123 | + \/ \/ |
| 124 | +"app" group: [my-app] |
| 125 | +``` |
| 126 | + |
| 127 | +#### Chart dependencies example |
| 128 | + |
| 129 | +To control the order in which subcharts are installed, upgraded, or uninstalled, chart authors must use the `helm.sh/depends-on/subcharts` annotation or the `depends-on` field in the `Chart.yaml`. These declarations enable Helm to determine the correct sequencing of subchart operations, as illustrated below. |
| 130 | + |
| 131 | +```yaml |
| 132 | +name: foo |
| 133 | +annotations: |
| 134 | + helm.sh/depends-on/subcharts: ["bar", "rabbitmq"] |
| 135 | +dependencies: |
| 136 | + - name: nginx |
| 137 | + version: "18.3.1" |
| 138 | + repository: "oci://registry-1.docker.io/bitnamicharts" |
| 139 | + - name: rabbitmq |
| 140 | + version: "9.3.1" |
| 141 | + repository: "oci://registry-1.docker.io/bitnamicharts" |
| 142 | + - name: bar # This is a subchart packaged with "foo" in charts dir. It's not pulled from a remote location |
| 143 | + version: "0.1.0" |
| 144 | + depends-on: ["nginx", "rabbitmq"] |
| 145 | + condition: bar.enabled |
| 146 | +``` |
| 147 | +
|
| 148 | +``` |
| 149 | + [nginx] [rabbitmq] |
| 150 | + || || |
| 151 | + || || |
| 152 | + \/ \/ |
| 153 | + \\ // |
| 154 | + \/ \/ |
| 155 | + [bar] |
| 156 | + || |
| 157 | + || |
| 158 | + \/ |
| 159 | + [foo] |
| 160 | +``` |
| 161 | + |
| 162 | +In this example, Helm will first install and wait for all resources of `nginx` and `rabbitmq` dependencies to be "ready" before attempting to install `bar` resources. Once all resources of `bar` are "ready" then and only then will `foo` chart resources be installed. `foo` would require `rabbitmq` to be ready but since the subchart resources would have been installed before `bar`, this requirement would have been fulfilled. |
| 163 | + |
| 164 | +This approach of building a directed acyclic graph (DAG) is prone to circular dependencies. During the templating phase, Helm will have logic to detect, and report any circular dependencies found in the chart templates. Helm will also provide a command to print the DAG for development and troubleshooting purposes. |
| 165 | + |
| 166 | +### Readiness |
| 167 | + |
| 168 | +To enforce sequencing, Helm determines whether resources are “ready” before deploying dependent resources. By default, Helm uses [`kstatus`](https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition) library to assess readiness based on the resource’s type and `.status` field. |
| 169 | + |
| 170 | +Chart authors can optionally override this behavior using the following annotations: |
| 171 | + |
| 172 | +* `helm.sh/readiness-success`: A list of custom success conditions. If any are true, the resource is marked **ready**. |
| 173 | +* `helm.sh/readiness-failure`: A list of custom failure conditions. If any are true, the resource is marked **failed**, which takes precedence over any success check. |
| 174 | + |
| 175 | +Both `helm.sh/readiness-success` and `helm.sh/readiness-failure` must both be provided to override the default readiness logic. If only one is present, Helm will fall back to `kstatus` and emit a warning. Helm will also fail linting when only one of the two is defined, to prevent ambiguous readiness evaluation. |
| 176 | + |
| 177 | +#### JsonPath syntax |
| 178 | + |
| 179 | +The `readiness-success` and `readiness-failure` annotations accept lists of expressions with the format: |
| 180 | + |
| 181 | +``` |
| 182 | +{<jsonpath_query>} <logical_operator> <value> |
| 183 | +``` |
| 184 | + |
| 185 | +Where: |
| 186 | + |
| 187 | +* `<jsonpath_query>` is a [Kubernetes JSONPath](https://kubernetes.io/docs/reference/kubectl/jsonpath/) query scoped to `.status`. |
| 188 | +* `<logical_operator>` supports: `==`, `!=`, `<`, `<=`, `>`, `>=`. |
| 189 | +* `<value>` is the expected literal for comparison. The value should be a scalor (string, number, boolean). Object comparisons will not be supported. |
| 190 | + |
| 191 | +##### Example |
| 192 | + |
| 193 | +```yaml |
| 194 | +kind: Job |
| 195 | +metadata: |
| 196 | + name: db-init |
| 197 | + annotations: |
| 198 | + helm.sh/readiness-success: ["{.succeeded} == 1", "{.succeeded} == 2"] |
| 199 | + helm.sh/readiness-failure: ["{.failed} >= 1"] |
| 200 | +status: |
| 201 | + succeeded: 1 |
| 202 | +``` |
| 203 | +
|
| 204 | +In this case, Helm will consider the resource ready because `.status.succeeded == 1`. If `.status.failed >= 1` had been true, the Job would instead be marked as failed. |
| 205 | + |
| 206 | +A resources readiness is checked if there is a resource that depends on it as per the sequencing DAG, otherwise the checks are ignored. |
| 207 | + |
| 208 | +Helm will wait up to a default of **1 minute** for a resource to either succeed or fail. If the resource does not reach a success or failure state within this period, the operation will time out, causing the chart install or upgrade to fail. This timeout can be customized using the `--readiness-timeout` CLI flag or the `ReadinessTimeout` field in the SDK. However, the specified readiness timeout must not exceed the overall `--timeout` value, which defines the maximum duration allowed for the entire chart installation or upgrade process. |
| 209 | + |
| 210 | +### Sequencing order |
| 211 | + |
| 212 | +Resources with sequencing annotations in a chart would be deployed first followed by resources without. If the chart has a `helm.sh/depends-on/subcharts` annotation in the `Chart.yaml`, all resources of the defined subcharts would be deployed before deploying the main chart. If any sequencing annotations are defined in the subchart resources, Helm will enforce ordering of resources within. Sequencing of resources in a chart are sandboxed within the chart. Sequencing annotations will not affect resources in other charts. |
| 213 | + |
| 214 | +- Installs: Helm will install resources in the order defined by the DAG. If any of the readiness checks fail or timeout, the entire install would fail and the release marked as failed. If `--atomic`, or its SDK equivalent is used, a rollback to the last successful install would take place. |
| 215 | +- Uninstalls: Helm would uninstall resources in the reverse order they were installed, as per the sequencing order. The logic to delete each resource will not change. |
| 216 | +- Rollbacks: Helm will check from the release object whether the revision being rolled back to, was installed in a sequenced manner. If it was, Helm will respect and enforce this order when installing resources from that revision. When deleting unneeded resources of the revision being rolled back from, the reverse order is followed just like uninstalls. |
| 217 | +- `helm template` would print all resources in the order they would be deployed. Groups of resources in a resource-group would be delimited using a `## START resource-group: <chart>/<subchart> <group-name>` comment indicating the beginning of each resource-group and `END resource-group: <chart>/<subchart> <group-name>`. |
| 218 | + |
| 219 | +```yaml |
| 220 | +## START resource-group: foo group1 |
| 221 | +# resource 1 |
| 222 | +metadata: |
| 223 | + name: foo |
| 224 | + annotations: |
| 225 | + helm.sh/resource-group: group1 |
| 226 | +--- |
| 227 | +# resource 2 |
| 228 | +metadata: |
| 229 | + name: bar |
| 230 | + annotations: |
| 231 | + helm.sh/resource-group: group1 |
| 232 | +## END resource-group: foo group1 |
| 233 | +--- |
| 234 | +## START resource-group: foo/bar group2 |
| 235 | +# resource 3 |
| 236 | +metadata: |
| 237 | + name: fizz |
| 238 | + annotations: |
| 239 | + helm.sh/resource-group: group2 |
| 240 | + helm.sh/depends-on/resource-groups: ["group1"] |
| 241 | +## END resource-group: foo/bar group2 |
| 242 | +``` |
| 243 | + |
| 244 | +## Backwards compatibility |
| 245 | + |
| 246 | +Helm will continue to install/upgrade/uninstall/rollback all resources and dependencies at one go for all charts using `Charts v2` and below. |
| 247 | + |
| 248 | +## Security implications |
| 249 | + |
| 250 | +None. |
| 251 | + |
| 252 | +## How to teach this |
| 253 | + |
| 254 | +- Document how sequencing works in the official helm documentation website. Include ordering of Kubernetes resources that Helm enforces when applying resources to the cluster. Examples will be added to best demonstrate how this feature works. |
| 255 | +- Document how this feature works for SDK users. |
| 256 | + |
| 257 | +## Reference implementation |
| 258 | + |
| 259 | +N/A |
| 260 | + |
| 261 | +## Rejected ideas |
| 262 | + |
| 263 | +1. A weight based system, similar to Helm hooks |
| 264 | + - Static numbering of the order is more challenging to develop and maintain |
| 265 | + - Modifying the order can lead to cascading changes. |
| 266 | + - Dynamically named system solves these problems for the application distributors. |
| 267 | + |
| 268 | +## Open issues |
| 269 | + |
| 270 | +## Prior raised issues |
| 271 | + |
| 272 | +- https://github.com/helm/helm/pull/12541 |
| 273 | +- https://github.com/helm/helm/pull/9534 |
| 274 | +- https://github.com/helm/helm/issues/8439 |
| 275 | +- https://github.com/helm/community/pull/230 |
0 commit comments