What type of changes should be allowed and applied to stable releases?

[nashif]

Per the document at https://github.com/zephyrproject-rtos/zephyr/wiki/Zephyr-Change-Control-Board-%28CCB%29:

Changes can be submitted to a stable and a released branch in 2 ways:

- Directly submitted to the branch as a pull request with the Backport label
- Pull requests to main with the special label backport-vx.xx-branch (can be used with any branch) will automatically be back-ported to the specified branch (via the label) and a corresponding PR will created against that branch.

The following are the candidates for back-porting to stable branches and specifically the LTS branch:

- Security Bug Fixes
- Functional Bug Fixes
- Usability Bug Fixes and improvements
- Documentation and Test Improvements
- Standalone and drop-in hardware support without any change to existing code.
- Important

All changes to stable releases SHALL have a bug open documenting the issue being fixed and the severity of the bug.

Everything has to go through the CCB. Security fixes and bugfixes are non-contentious (though they still need go be approved by the CCB). Anything else needs approval from the CCB upon request from a particular developer, and will require justification and discussion.

The above seems to be vague and not enough and we need a better definition of what falls under a change to a stable release and the process needed to apply such changes.

Please provide your input below..

[stephanosio]

I am sure one thing that everyone can agree on is that automation is not the way to go; everything that gets backported must be reviewed case-by-case by the CCB/release managers so as to not introduce yet another bug and also to not introduce any breaking changes, and we need to be as conservative as possible in general for that purpose.

Also it is worth noting that the release branch (e.g. v2.7-branch) is a "trunk" branch for the next dot/patch release and we do not, by any means, guarantee that this branch will always be stable/free of breakages during development -- that is why we have the patch releases (e.g. v2.7.x) that are tested and known to be stable in the first place.

[cfriedt]

As mentioned here
#43335

Of course, all backport candidate PRs should continue to be audited, but the process of creating them can be fully automated.

[cfriedt]

I would say that low priority bugs should definitely not have automated backports. High (and medium) priority bugs should most likely have automated backports.

They should still be reviewed by the CCB, of course - nothing should ever be merged automatically.

We talk about scaling but without some automation, scaling will be impossible. More and more bugfixes will simply slip through the cracks.

[stephanosio]

I think it would be useful to introduce a new label called "Backport Candidate" or "Backport Review," which can be added to the PRs that collaborators or maintainers consider worth backporting.

The PRs that are labelled as such can be reviewed during the release meeting and, if approved for backporting, the release manager can add the automated backport request label(s) (i.e. backport v2.7-branch, backport v3.0-branch, ...) for the affected versions.

[ycsin]

I think it would be useful to introduce a new label called "Backport Candidate" or "Backport Review," which can be added to the PRs that collaborators or maintainers consider worth backporting.

I'd suggest introducing another label for PR that shouldn't be backported, so it tells everyone that the maintainer has already considered this, instead of forgetting to apply the backport label

[nashif]

some references:

• https://wiki.ubuntu.com/StableReleaseUpdates

not that we are in any way similar to ubuntu, but some guidelines make sense to Zephyr, for example:

Stable release updates are automatically recommended to a very large number of users, and so it is critically important to treat them with great caution. Therefore, when updates are proposed, they must be accompanied by a strong rationale and present a low risk of regressions.

We never assume that any change, no matter how obvious, is completely free of regression risk.

In line with this, the requirements for stable updates are not necessarily the same as those in the development release. When preparing future releases, one of our goals is to construct the most elegant and maintainable system possible, and this often involves fundamental improvements to the system's architecture, rearranging packages to avoid bundled copies of other software so that we only have to maintain it in one place, and so on. However, once we have completed a release, the priority is normally to minimise risk caused by changes not explicitly required to fix qualifying bugs, and this tends to be well-correlated with minimising the size of those changes. As such, the same bug may need to be fixed in different ways in stable and development releases.

Stable release updates will, in general, only be issued in order to fix high-impact bugs. Examples of such bugs include:

• Bugs which may, under realistic circumstances, directly cause a security vulnerability. These are done by the security team and are documented at SecurityTeam/UpdateProcedures.
• Bugs which represent severe regressions from the previous release of Ubuntu. This includes packages which are totally unusable, like being uninstallable or crashing on startup.
• Bugs which may, under realistic circumstances, directly cause a loss of user data

etc...

[ycsin]

Standalone and drop-in hardware support without any change to existing code.

Does this means that PR of new drivers can be backported? i.e. #40477

And "Usability Bug Fixes and improvements" seems a bit vague, especially "improvements", is there any example for this?

[nashif]

this is exactly what we are trying to clarify here.