Skip to main content

Utilize Server Side Apply for installs/upgrades

HIPTitleAuthor(s)CreatedTypeStatusHelm Version
23Utilize Server Side Apply for installs/upgradesGeorge Jenkins gvjenkins@gmail.com2023-07-07featureaccepted4

Abstract​

This HIP proposes Helm support Kubernetes' Server-Side Apply (SSA) for object updates/management.

Motivation​

Helm 3 introduced the three-way strategic merge and patch process, enabling Helm to update objects which have been modified by other processes. Kubernetes now offers a similar server-side merge process, that yields several advantages over the client-side apply (CSA) methods that Helm and kubectl (for example) have traditionally utilized.

Kubernetes has evolved this client-side merge process to be managed server-side, yielding several advantages over the client-side apply (CSA) methods that Helm and kubectl (for example) have traditionally utilized.

The following describes the advantages of SSA from Kubernetes perspective, where SSA generally is considered to be superior:

https://kubernetes.io/blog/2022/10/20/advanced-server-side-apply/

(See KEP 555 and the Kubernetes documentation for server-side apply's full motivation and detailed description)

Helm supporting to SSA allows Helm users (chart operators) to realize the benefits of Kubernetes server-side field management for object updates. And enabling SSA by default (similar to kubectl) will avoid users having to specifically enable SSA to realize its benefits.

Finally, from a code maintenance benefit, SSA support may eventually allow Helm to eventually drop support for the strategic-merge patch CSA implementation/code.

Rationale​

Kubernetes' has sigificant movement towards SSA as preferred method for object updates/management. Helm should adopt SSA in continuance with this trend, as it is unlikely client-side methods will continue to be improved upon. Nor will client side methods likely ever be able to provide the functional benefits of SSA.

With Helm 4, it was decided to opt into utilizing SSA by default. Allowing chart operators to gain the benefit of SSA without needing to know about the server-side vs. client-side mechanisms. If during Helm 4 release candidate field usage, SSA as default is deemed to be unresolvably problematic, Helm 4 will revert to utilizing CSA by default.

A CLI flag will allow users to override SSA opt-in and revert to utilizing CSA. Should a chart operator need for a particular circumstance.

A chart release mechanism will allow Helm to track and follow SSA enablement for a given release. Tracking SSA enablement will allow Helm to automatically follow SSA enablement for a given release, and in particular revert to CSA (or SSA) during rollbacks if that prior release utilized CSA (or SSA).

Additionally it was desirable to mimic kubectl's CLI interface/flags. As a way for Helm to be consistent with an existing kubectl functionality that Kubernetes users may be familar with.

Specification​

General​

Helm will add a new flag --server-side=false|true|auto to the install, upgrade and rollback commands (modeled after the like kubectl apply flag). And add a corresponding field to the respective install/upgrade/rollback SDK API objects.

Helm will default to true for installs, enabling SSA. And auto for upgrade and rollbacks, where Helm will enable SSA based on the whether SSA was utilized for the prior release (see below). When enabled, Helm will submit object to the Kubernetes API server via SSA patches.

If SSA or CSA is utilized for a release, this will be stored as a field in the release's metadata. If the field is unset when reading a prior release, this implies that the release was created with an old version of Helm utilizing CSA.

type Release struct {
   ApplyMethod *string  `json:"apply_method,omitempty"` // "ssa" | "csa"
}

If a user deploys a chart with a new version of Helm with SSA enabled, then upgrades the chart with an older version of Helm. It is expected that the older Helm will be able to upgrade the chart with CSA. And the new release's metadata will omit the indication of SSA usage (and so further upgrades/rollbacks will continue to use CSA by default).

In a future version of Helm 4 CLI, it might be desirable to omit a warning if a user is not using SSA (in order to prompt the user to enabling SSA for further releases).

Custom resource objects​

Helm will utilize SSA for custom resource objects, which it previously didn't merge. The API server will [merge custom resources][ssa-crds based on CRD metadata.

Conflicts and forcing​

It is possible that when Helm upgrades (or rollbacks) a chart, there will be a field conflict with another field manager. In this case, Helm will report the error for the conflict field(s) to the user.

Helm will also add a new flag --force-conflicts=false|true (default false) to the install, upgrade and rollback commands (modeled after the life kubectl apply flag) And corresponding API objects field. When true, Helm will enabling overriding the field conflicts, setting field(s) to the charts values. And also make Helm the field manager for the conflicting field(s). The --force-conflicts flag documentation should explicity mention the downsides of conflict forcing ie. the chart update is overriding object values that another process is expecting to own/manage.

For more details, see SSA documentation Overwrite value, become sole manager

(To note, helm upgrade|rollback commands already includes a --force flag, which tells Helm to replace objects during upgrade. However, given this different semantics of --force, a new flag is required)

An additional special case includes rollbacks on failed upgrade (the --atomic flag). It is possible for Helm to attempt to rollback to a version which now has a conflict. For which the rollback will fail. Since rollbacks are "best effort" and can fail for other reasons as well, it seems best to treat this new rollback failure-case similarly (report to the user via logging and error). If the user specified --force-conflicts to the upgrade, the corresponding rollback should similarly force conflicts (this is the behavior of --force). If the rollback is to a release version which previously didn't have SSA enabled, the rollback install should also disable SSA.

Special​

  • --dry-run=client won't accurately simulate an install with SSA enabled. This is already often true even with the current merge strategy, but especially true of Server Side Apply. To accurately simulate an install with SSA enabled, instead use --dry-run=server.
  • If the user attempts to use SSA with a cluster which does not support it (unlikely: SSA went GA in Kubernetes v1.22), Helm should error
  • Client-side API object validation isn't applicable when SSA is enabled (helm install|upgrade --disable-openapi-validation)
  • Hooks could be deployed with SSA. But as hook objects are not updated (create/delete only), they are not really affected
  • Stored release manifests should not change (they store the object as Helm intends)

Implementation​

Helm will send object's manifests to the Kubernetes API server via SSA patches. Similar to kubectl apply. Utilizing the field manager name "helm":

https://github.com/kubernetes/kubectl/blob/197123726db24c61aa0f78d1f0ba6e91a2ec2f35/pkg/cmd/apply/apply.go#L248 https://github.com/kubernetes/kubectl/blob/197123726db24c61aa0f78d1f0ba6e91a2ec2f35/pkg/cmd/apply/apply.go#L564-L602

Backwards compatibility​

SSA should be generally compatible functionality-wise with CSA. The expectation is that SSA should generally produce equivalent Kubernetes objects as CSA.

There four main behavioural differences for Helm switching to SSA:

  1. The differences between object's manifests managed with SSA vs the existing three-way strategic merge process. SSA will enable:
    • Arrays will be merged
    • Removal of unset fields
    • Default values
  2. Issues with a user enabling/disabling SSA over the lifetime of a release e.g. environment differences or older Helm clients which don't understand SSA
  3. SSA will apply to custom resources
  4. Field management ownership conflicts

For 1., chart-operators may opt-out out of SSA. For 2., the user can control by ensuring SSA is consistently used within their environment. For 3., it is generally considered a limitation of Helm not updating custom resource objects, for which SSA is expected to be an improvement. For 4., the --force-conflicts flag exists.

Additionally as noted, SSA isn't compatible with very old Kubernetes versions (1.22 GA / 1.17 Beta)

Security implications​

There should not be any additional security concerns. Server-side apply and client-side apply should be equivalent security-wise.

How to teach this​

Documentation for how Helm does object upgrades, and the advantages of SSA. Also documentation for the --server-side=false|true|auto and --force-conflicts=false|true flags, and the defaults for install/upgrades/rollbacks. And the corresponding documentation for the field(s) SDK API objects.

Reference implementation​

Kubectl:

Prototype (hack) for Helm:

Rejected ideas​

Chart field manager support​

This HIP defers including field manager support in charts. ie. Helm could enable applying multiple manifests for the same object with distinct field manager names. Either from the same chart, or different charts/releases.

It is unclear how Helm users would interact with such a feature e.g. the chart developer might want/need to specify field manager names per output manifests in a chart. And helm template might somehow need to allow describing the object's field manager name in its output. And whether it would be needed to control per-manifest conflict forcing.

Once more information is gathered on how chart developers might utilze explicit field manager support, a proposal introducing field manage support for charts could be written.

Field ownership transfer​

Helm won't support ownership transfer of fields. Similar to the kubectl example: https://kubernetes.io/docs/reference/using-api/server-side-apply/#transferring-ownership.

Dissimilar to kubectl, Helm distinguishes between chart developers and chart operators. Chart developers may not consider, or may not even know, which fields may be overwritten (managed by) by another process in a user's Kubernetes cluster.

Ownership transfer would likley first require charts have field manager support. As well as additional support for per-object conflicts resolution.

Open issues​

References​