Helm 4 Development Process
| HIP | Title | Author(s) | Created | Type | Status |
|---|---|---|---|---|---|
| 0012 | Helm 4 Development Process | Matt Butcher matt.butcher@fermyon.com, George Jenkins gvjenkins@gmail.com, Matt Farina matt.farina@suse.com | 2021-04-02 | process | accepted |
Abstract
Helm 4 is planned as the next major version release of Helm. Continuing to evolve Helm in its role as the Kubernetes package manager.
This HIP outlines the requirements and process for building Helm 4. It describes the scope of Helm 4 changes, with a focus on maintaining continuity within the Helm ecosystem.
This HIP also aims for an open and productive process that enables Helm 4 to build upon the success of Helm 3. To continue advancing Helm and its ecosystem for the benefit of the Helm community.
Motivation
Helm 3 has been greatly successful, and Helm has built a significant ecosystem around the Helm client (CLI), charts, chart repositories, SDK, plugins, etc. As Kubernetes continues to grow and change, Helm needs to adapt while continuing to add new features and functionality to support its mission as Kubernetes' package manager.
However, Helm has accumulated debt. One significant issue with Helm 3 is that making many (good) changes is overly difficult due to Helm's HIP-0004 backwards compatibility guarantees –- behavioral or API incompatible changes are not allowed during regular minor/patch version releases. Helm 4 as a major release allows the project to make breaking API and other changes, which will enable it to reduce technical debt and leverage newer capabilities of Kubernetes.
However, Helm 4 must maintain continuity in the Helm ecosystem. It is crucial that Helm 4 be largely functionally/feature compatible with Helm 3. Significant functional breakage will prevent users from easily migrating to Helm 4 and, at best, delay Helm 4's uptake and incur costs for Helm's users. And at worst, could splinter the Helm ecosystem.
The project also aims to timebox the Helm 4 release process. Given the limited community resources available to enact changes, it is considered more important to produce a Helm 4 release that can deliver significant benefits in a moderate timeframe than a prolonged process that takes too long to provide meaningful results.
Rationale
During the Helm 3 development process, Helm core maintainers received criticism that the process was not transparent enough. This proposal aims to outline a formal process for a successful Helm 4 release to address those criticisms.
Further criticism included a lack of clarity regarding who was in charge and how decisions were being made, as well as uncertainty about the criteria used for determining when Helm 3 was complete.
This document is intended to address these criticisms by presenting a clear plan for the timely delivery of a valuable Helm 4 release to the community. It describes the process in detail, providing both a roadmap for us to follow and a point of reference for community members to use when contributing changes.
The process is based on the following ideas:
- Requiring changes to be documented, either as reduced "Helm 4 proposals" (H4HIP) or standard HIPs. Allowing both Helm community members and maintainers to review them. This oversight ensures that changes are in line with this HIP, regarding their functional and architectural implementation. Additionally, documented changes enable contributors to independently implement agreed-upon proposals in coordination with the release engineer.
- Requiring a timeboxed development effort intends to promote focus on the most valuable changes that the community and maintainers can deliver within a reasonable timeframe. It also aims to prevent an "open-ended" development cycle that might drag on or fizzle out. Ideally, the majority of changes will be proposed and agreed upon during the planning phase of the development cycle. However, amendments to the initial scope are permitted as long as they can fit within the agreed-upon timeframe. Explicitly allowing for scope amendments ensures that any changes are well thought out. For changes that don't make it into Helm 4's initial development cycle, they can included in Helm 4.x feature releases (subject to minor release compatibility guarantees). And for further changes, there will be a Helm 5 release of course!
Specification
The process / timelines
Helm 4 development will consist of three main phases:
- Planning
- Feature development
- Release preparation
The timeline will be approximately:
- Marking this HIP as "accepted"
- Naming of a Release Engineer: Nov. 2024
- Kick-off: KubeCon/CloudNativeCon North America 2024 (Nov. 2024, Salt Lake City)
- Earliest that engineering can begin: Nov. 2024
- Feature freeze for release preparation: By Aug. 2025
- Expected release of Helm v4.0.0: By the week before KubeCon/CloudNativeCon North America 2025 (expected Nov. 2025)
Helm v4.1.0 release will be 3 months later (expected Jan. 2026 for the above timeline). And from then on Helm 4 minor releases will follow the same release cadence as Helm v3, shadowing Kubernetes by approximately 1 month.
Helm 3 will reach end of life 6-8 months after Helm 4 release (July 2026 estimated)
Change suitability
The Helm ecosystem is large, including not just the Helm CLI but also the SDK, Charts, and Chart repositories, plugins. Helm also has several roles or personas for different user categories, from "Application Operators" to "Helm Developers." The Helm project must be careful in how it delivers changes without causing significant adverse impact to Helm's users or the overall Helm ecosystem.
Therefore, while Helm 4 does not need to be semantically backwards compatible with Helm 3, it must not cause significant breakage for users who adopt Helm 4. Clear migration options and paths should be documented and presented for Helm 3 users to update to Helm 4.
Change proposals
To ensure that Helm 4 changes align with this HIP, Helm core maintainers will review proposals. These proposals can take the form of lighter-weight Helm 4-specific proposals "H4HIP"s, or standard HIPs suitable for Helm 4 implementation. Standard Helm HIPs that are marked suitable for Helm 4 will also be accepted.
Helm 4 proposals (H4HIPs)
A H4HIP is simply a reduced version of a standard Helm HIP. They follow the same format as a HIP, but the author can choose to skip sections they do not feel applicable. Intended to minimize the friction required for implementing small, functional changes in Helm 4 that do not warrant a full HIP process. And to allow categorization of changes specific to the Helm 4 process and changes. As well as to avoid feature development from Github issues.
Compatibility
Even though Helm 4 is a major version release not bound by HIP-0004 compatibility guarantees, it must remain largely feature/functionality compatible with Helm 3. Due to Helm's maturity and the extensive ecosystem built around its CLI, charts, SDK, etc.
Where non-compatible changes are introduced, they should have migration documentation for equivalent functionality. Functionality removal should prefer deprecation over immediate removal. Behavioral changes that negatively impact a significant number of users without reasonable migration options cannot be allowed.
Specific requirements for compatibility include the following:
Compatibility with existing charts and releases
While the tooling can evolve, Helm must maintain compatibility with existing charts and releases. A Helm 4 that cannot operate on Helm 3 charts (and vice versa) effectively becomes a different tool, which likely would diverge the Helm ecosystem.
Specifically, charts deployable with Helm 3 should be deployable by Helm 4. Similarly, any existing chart (release) managed by Helm 3 must be upgradable by Helm 4. However, Helm 3 may not be able to manage releases created using later Helm 4.x series (which may introduce incompatible changes).
Exceptions for already deprecated functionality, such as requirements.yaml, may be made.
Compatibility with existing CLI workflows
The Helm CLI is deeply integrated into thousands of release pipelines and automation systems, so while it can evolve, it must remain loosely compatible with existing users' workflows.
A judgment call will be required by the release engineer and maintainers regarding how much a given change might impact user workflows. Possible mitigation strategies for those changes should also be considered.
As a baseline, most "Application Operator" users of Helm 3 CLI should experience no disruption or minimal adjustments to their workflows during the upgrade to Helm 4. Changes should be clearly documented in a migration guide.
More advanced users, such as "Application Developers," might be subject to more significant changes at the discretion of the release engineer and maintainers.
The Release Engineer
A single individual will oversee the planning and development phases of Helm 4. This person will be the Release Engineer. For Helm v4 the Release Engineer will be Matt Farina. The person will have the following responsibilities:
- Lead the kick-off meeting
- Define the process for including features
- Determine the timelines for development
- Make the determination of when the project has indeed reached a milestone
- Name and oversee releases
- After the release, the release engineer will determine what the "intent" of a Helm behavior was (which informs determining whether an incoming breaking change is a feature or a bug fix)
Determining Who Will Be Release Engineer
To become the release engineer for the Helm project, an individual must be a core maintainer on https://github.com/helm/helm project. The engineer must be an active core maintainer. Self-nominations are encouraged, and potential nominees must agree to their nomination. The Helm project maintainers will then decide which nominated maintainer should serve as the release engineer.
If the release engineer is unable to fulfill their duties, the Helm project maintainers may select a replacement.
Planning
This phase begins when the release engineer is appointed. The goal of the planning phase is to have a specific period to prioritize writing and reviewing proposals. And enable time to focus on buidling, prioritizing and reviewing the major changes for Helm 4.
The release engineer shall schedule a kick-off meeting, inviting all Helm maintainers from any Helm org project to attend. The release engineer may also decide who else to include and whether the meeting will be public. During the kick-off meeting, participants are encouraged to propose feature ideas and changes, with preference given to those already documented as HIPs.
This process continues until the release engineer deems it sufficient, but for a minimum of two months, to allow ample community input.
All proposed features and breaking changes must be described in a standard HIP or Helm 4 proposal (H4HIP). This approach offers several advantages:
- All changes are documented
- Changes undergo scrutiny before PR creation
- A clear record of alternative considerations exists
Minor non-functional changes for "bug fixes" or cleanup, as determined by core maintainers, do not require formal proposals.
The outcome of the planning cycle is a well-documented list of approved HIPs detailing Helm 4's major changes. This list will serve as the foundation for working on Helm 4.
Items which are readily agreeable for Helm 4 (as determined by the release engineer and core maintainers) may begin implementation during the planning phase. Cleanup and non-functional changes may proceed while in the planning phase.
Feature Development
During this phase, agreed-upon features and changes from the planning stage will be implemented. Breaking changes are permitted, as well as experimental features that may be removed before final release. The approval process for PRs remains as previously defined within the project, but any PR not aligned with the current plan must receive approval from the Release Engineer.
Additional features can be propsed during the feature development phase, at the release engineers discretion.
This phase concludes with Alpha releases, where new features can still be incorporated and breaking changes may be implemented at the release engineer's discretion.
Release Preparation
During this final phase, no new features will be added except for those deemed critical by the release engineer. The focus shifts towards bug fixes, documentation improvement, security audits and addresses, and preparation of the ecosystem, such as chart modifications. All Beta and Release Candidates (RC) are produced during this stage.
Before the stable release, there will be a 4-6 week testing period for public feedback. The duration is determined based on the testing progress and the trust in the code leading up to the release candidate window.
This phase concludes when Helm v4.0.0 is released, but it must not be shorter than a three-month duration to ensure adequate time for ecosystem testing and feedback.
Halting Helm 4 Development
During the course of Helm 4 development, it may become clear that there is not enough justification for continuing development. Reasons could include insufficient feature HIPs or lack of sponsorship and support from Helm project maintainers.
If this occurs, the Helm maintainers can hold a "circuit breaker" vote to temporarily halt Helm 4 development until a more suitable time arises.
The procedure for stopping Helm 4 development requires a simple majority vote among the Helm project maintainers.
Maintaining Helm 3
This section outlines how we will support Helm 3 throughout the Helm 4 development process. The Release Engineer retains the freedom to modify this approach for the sake of providing the best experience to Helm users.
During the "Feature Development" phase of Helm 4, Helm 3 will continue to receive security, bug, and minor feature patches.
During the Alpha, Beta, and RC phases of Helm 4, no feature patches will be applied to Helm 3 to avoid introducing Helm 4 features prematurely while Helm 4's features are locked.
Helm 3 will receive bug fixes for 6-8 months following the release of Helm 4 and security fixes for one year from the day Helm 4 is made available.
Backwards compatibility
Nothing in this document contradicts our existing decision-making process. Specifically, HIP-0004 pertains only to minor and patch releases, not major version releases. This does not affect the PR review process or the HIP process, as both are expected to remain unchanged.
How to teach this
The plans for Helm 4 will to be announced during KubeCon/CloudNativeCon North America 2024 at both the Helm Maintainers talk and Helm 'contribfest'.
The content of this HIP will be discussed publicly in the regular Helm Dev meetings, which are recorded.
Throughout all stages of Helm 4 development, the Release Engineer is committed to providing updates on progress during no fewer than two Helm public dev calls per month.
Rejected ideas
Helm 3 was primarily developed using an informal version of the proposed process, but we were criticized for lacking documentation and transparency. This proposal presents an alternative to an "informal process," which might elicit similar criticism.
Multiple release engineers were considered, but deemed inappropriate for two reasons:
- The need for a singular authority in charge of architectural integrity and resolution
- The lack of sufficient Helm core maintainers to justify a committee
We entertained the idea of allowing a non-core maintainer to serve as release engineer, but rejected it due to these concerns:
- A release engineer requires comprehensive knowledge of Helm's design, intents, and implementation, which is best ensured through maintainer status
- A non-core maintainer is not bound by the core maintainer process, potentially introducing tension and exceptionalism
We pondered enabling any Helm maintainer from any project to serve as the release engineer for Helm, but determined that the in-depth technical knowledge of the Helm core codebase is essential for this role's successful execution. We regard the release engineer position as more akin to an architect than a project manager.
We considered restricting the release engineer role to org maintainers, but decided against it since their focus is on organizational (not technical) aspects of the project. Release engineering is beyond their charter.
We debated abandoning the requirement for HIPs or written proposals for features and changes during the Helm 4 process, as we had done so in the past. However, we also faced pushback from the community regarding major changes being implemented without proper discussion. Furthermore, there is a lack of documentation detailing past decisions made in Helm 2, with only a few draft documents available in Helm 3.
Therefore, HIPs and their lighter-weigtht couterparts "Helm 4 proposals" (H4HIPs) will serve as a platform for public discourse and provide a recorded decision-making process. While some may argue this adds unnecessary overhead, we believe the advantages far outweigh the burden.
We considered eliminating the mandatory minimum durations for planning and release preparation periods. However, careful consideration is required during a major version transition on a project as impactful as Helm's. It is in the community's best interest for developers to proceed at a deliberate pace with clearly defined timeframes.