Release maintenance

From Koha Test Wiki MediaWiki Postgres

Note: this guidelines are work in progress, and release maintainers have the right to change this to whatever they prefer. For now this will serve as a documentation place for the tasks involved in release maintenance.

The Release Maintainer role

Release maintainers do the task of keeping the stable Koha releases in line with patches provided for bugfixes. People are elected for this task after they have volunteered for it, and there is consensus on the development community for them to take on the job.

RMaints are assigned one or more stable branches to maintain. Crazy people like Chris Cormack, Chris Nighswonger and Fridolin Somers have maintained more than one release at a time. It's a crazy Chris thing®.

Schedule

Every month there is a stable maintainance release. The relevant dates are:

• 15th: String Freeze
• 22nd: Release

String Freeze: During String Freeze patches that change strings should not be pushed! Please keep in mind that a changed string will mean that the string appears untranslated after updating, which can be especially toublesome for OPAC strings. Most users rely on the packages to provide translations and are not aware how they could fix such problems locally.

• Get in touch with the translation manager on time to have them update the po files on translate.koha-community.org
• Announce String Freeze on the koha-translate mailing list and user list (koha@lists.katipo.co.nz)

Release: After each release, get in touch with your fellow release maintainers and the packaging manager in order to agree about a date for string freeze, a date for when your branches will all be ready for packaging and a date for the release. This is thought to make the life of our packaging manager easier as they can build all branches at once. Also it means that the packages will be ready at release.

Initial setup

Request push access for git

You will need to be given privileged access to the community git server and it's security counterpart.

To enable this, you will need to get access to https://git.koha-community.org/Koha-community/ (ask an account on IRC or koha-devel mailing-list) and link your public ssh key to your gitea account.

Create a remote for pushing

You could set up a remote for pushing:

  git remote add upstream git@git.koha-community.org:Koha-community/Koha.git

and then push using this:

  git push upstream 18.11.x

Git pre-push hook

There is a pre-push hook documented in the 'Tips & Tricks' page, this can be beneficial as it adds a series of final checks before allowing you to push your local updates to the public repository.

Satisfied users testimonials:

• «I'm not worried about pushing to the wrong branch anymore!»
• «The pre-push hook detected a forgotten SCSS compilation since many patches»
• «I would have pushed so many commits without a signoff without this hook»
• «Helps to not mix things up when releasing two branches»

Release tools

This section lists the set of tools a release maintainer needs to be able to make releases.

The release-tools repo

The Koha dev community has a nice set of tools to make the job easier. This set of scripts is on a git repo that can be cloned using

$ git clone https://gitlab.com/koha-community/release-tools.git

GPG keys

In order to sign the git version tag you need to have valid gpg signatures. Generate them issuing:

$ gpg --gen-key

Keys can be generated using 4096 or 2048 bits. It took me a while and lots of USB HDD activity to get enough entropy but I managed to generate it with 4096 bits.

You may need to configure your git repo, see here.

You can add the key quickly to a repo (or all repos) with

$ git config --global user.signingkey [gpg-key-id]

Request the URL of the form that prepares the translations for the release

It is used in this step of the release: Release_maintenance#Push_updated_translation_files

The RM or any RMaint can provide the URL to you.

Request access to the community website

It will be used to announce the new releases.

For getting access on services in general, see this page to find who to ask: https://wiki.koha-community.org/wiki/Website_Administration

Request access for ssh access for download.koha-community.org

Used to upload the tarballs.

Handover between two cycles

The old and new release maintainer of a given branch should contact each other to share this information:

• The position of the current "RMaint cursor" i.e. the last bug considered for backporting.
• The eventual list of bugs that due to the last string freeze have not been yet backported.
• Any other significant relevant info like gotchas of the branch (in conflicts, in random automated tests failures, etc).

Daily maintenance workflow

Release Maintainers usually watch the activity on the master git branch. If they maintain a release other than the latest stable release, they should be looking at higher-numbered versions git logs (for example, if you are maintaining the 3.10.x stable branch, then you should be probably watching 3.12.x's, and keep in touch with its maintainer).

MRenvoize uses a quick and dirty script to add bugs from master into hist todoist list.

You should also watch Bug 17108 - Automatic debian/control updates for patches specific to your version as they may well not hit master ;)

Once you spot a specific commit or commit series on your queue, the first thing is to look at bugzilla and check if it is a bugfix or an enhancement and decide whether to back port.

Deciding on what to back port

The general principle is to maintain stability and don't break things for supported releases!

Other notes:

• Enhancements: Generally only backport minor enhancements, leave a message on the bug to say it can be done on demand if required.

• Seek clarification where required: Check with the rest of the devs on IRC or mailing the author of the patches for clarification.

• Changes with database updates: Be particularly cautious about backporting a change that includes a database update. All database updates 'should' be idempotent, meaning they can be run repeatedly without negative consequences. It's always worth double checking this as errors have been known to creep through occasionally. For example, a DB update that adds a new table will result in a warning, at least, during upgrade, and should be avoided unless it is absolutely required in order to resolve a security bug. When it doubt, please discuss with the RM and fellow Release Maintainers first.

When not back porting

Mark the bug as "Resolved - Fixed."

It's also nice to leave a comment if relevant along the lines "Not backporting, not relevant to VERSION" or "Not backporting, new feature" etc

Processing a commit or commit series

Once you have decided the patch should be backported to the stable branch you are maintaining, update your local git repository (just in case) and create a working branch. You should define a naming schema to avoid confusion now you have commit rights! As 18.11.x release maintainer I chose 18.11bug_<bugnumber>. Do what you want :-D

  git checkout upstream/18.11.x -b 18.11bug_<bugnumber>

You should then look for the commit hash of the patch(es). And then cherry-pick them like this:

git cherry-pick -x 1a8db0ba7d62a70cd6b49ee705a5db9b11ad7672

If there's some conflict with cherry-picking, git will tell you. You will have two options

• Solve the conflict
• Try to solve the conflict and give up

If solving the conflict is not straightforward, comment on the bug like this: 'This does not apply cleanly to 18.11.x please submit one formatted for 18.11.x'.

If you solved the conflict you finish by

git add <touched files>
git commit

A cherry-pick message is added to the commit, and if there were conflicts, on which files.

Note: If the patchset includes any database updates, you will need to correct the version number bump to match your branch and running number.

Note: If the patchset includes any database updates, backporting from version 21.11.x to 21.05.x will require to move the DB changes to updatedatabase.pl.

Note: If the patchset includes changes to database table structures you will likely need to rebuild the DBIx::Class schema files. See Release_management#Update_the_schema_files

Note: If the patchset includes a database atomic update, like if it's only for your branch, see Database_updates#Pushing_to_a_branch

Note: If the patchset includes CSS changes, you will likely need to rebuild the CSS files from their source SCSS Working_with_SCSS_in_the_OPAC_and_staff_client

You should run some tests. For example, if the patch modifies several templates, you should at least run:

perl xt/tt_valid.t
perl xt/author/valid-templates.t

Do some testing, and keep in mind you should be confident about the QA team.

Once everything is solved, and testing proves everything is fine with the cherry-picked commits, you should sign on them:

git commit --amend -s

And then merge the changes with your maintenance branch

git checkout 18.11.x
git merge 18.11bug_<bugnumber> --ff-only

Pushing upstream

Once you've collected some bugs in your local branch, you need to remember to push it up to the community.

 git push upstream 18.11.x

Assuming you have configured the git pre-push hook, using this branch should run a final pre-push check on your work before allowing you to push your changes up to the wider world.

Update Bugzilla

This is very important: Let people know you pushed the patches to your stable branch by changing the status to Pushed to stable/oldstable/oldoldstable and we also now have a custom field (Version(s) released in:) in Bugzilla which you should add the specific release version to for which the bug will be released (This field takes the form of a comma-delimited list of versions, for example, `19.05.00, 18.11.03, 18.05.11`).

Note: You could use the `koha-push` script from the release-tools repository to do the bugzilla update. It will do a final check on the bug status, update the status, add text to 'Version(s) released in' and add a comment to the bug for you.

Monitoring Jenkins

We use Jenkins, a CI tool, to test each branch after new code is pushed to it. You should have been added to the email contacts for failing jobs and as such should get notified of any test breakages shortly after pushing any changes.

You can view the progress of the build tests on the Koha community Jenkins site. For the 18.11 branch, this takes about 30 minutes for each build.

Releasing

Push updated translation files

Keep in touch with the translation manager to get the strings on translate.koha-community.org updated in time for String Freeze on the 15th of a month.

Right before release, you can use a form provided by the translation manager, to automatically create a new branch on the TM's git repository with the latest version of the po files. Note: As the form is not protected at the moment, you can get the link to it from one of the previous Release Maintainers. Also, when inputting the version number, the format is 19.05.12, not v19.05.12.

You will receive an email with all necessary information about the created branch and can then merge it into yours:

$ git fetch remotename
$ git merge remotename/branch

Remember to sign the relevant patches.

You must check the po files for errors, you can do:

sudo apt-get install translate-toolkit
junitmsgfmt misc/translator/po/*
# search for 'fatal error' in the output

An error can be fixed, for example, by marking the string as 'fuzzy' in the po file and re-running the check.

Errors (not warnings) can lead to major translation breakage not just on the invalid string. e.g. only half the files get copied into the translated directories. Or a broken page due to a missing variable causing program code to end up in the page.

You will also need to check for possible XSS attempts:

 The POs must *not* contain:
 % git grep '<script' **/*.po
 % git grep PERL **/*.po

You should then commit those changes:

$ git add misc/translator/po
$ git commit -m 'Fix translations for Koha YY.MM.XX'
$ git commit --amend -s

Bump version number

In Koha.pm increment the following line:

 $VERSION = "YY.MM.XX.000";

Remember that between releases there might be minor database version upgrades. They would have been done the same way, using the latest minor number to reflect those version changes. For example 17.05.02.001, 17.05.02.002 and so on.

If Koha >= 21.11 :

Create a new db_rev file in /installer/data/mysql/db_revs with name "211110000.pl" (for 21.11.10-000)

use Modern::Perl;

return {
    bug_number => undef,
    description => 'Koha 21.11.10 release',
    up => sub {},
}

Else : Add this database upgrade to installer/data/mysql/updatedatabase.pl:

$DBversion = 'YY.MM.XX.000';
if( CheckVersion( $DBversion ) ) {
    NewVersion( $DBversion, "", "Koha YY.MM.XX release" );
}

You should then commit those changes:

$ git add Koha.pm installer/data/mysql/updatedatabase.pl
$ git commit -m 'Increment version for YY.MM.XX release'
$ git commit --amend -s

You can now push the latest changes. Or continue with the next step.

Generate the release notes

NOTE: Make sure to pull the latest version of the release tools before generating the notes

Release notes are automatically generated by the koha-release script from the release-tools repo. If you are releasing version 17.05.03 you should run this on your clean 17.05.x branch:

$ ~/<path_to>/release-tools/bin/koha-release v17.05.02..HEAD notes
# The HTML version is generated from the Markdown one.
# So if editions need to be made, they can be done to the Markdown version only and the HTML can be regenerated.
$ ~/<path_to>/release-tools/bin/koha-release v17.05.02..HEAD html misc/release_notes/release_notes_17_05_03.html

it will create the following files:

- misc/release_notes/release_notes_17_05_03.md
- misc/release_notes/release_notes_17_05_03.html

Check for inconsistency

It is important that you edit that file and fix any inconsistency you find. Usually

- First release: The team might be wrong if the config file has the wrong master version. (release-tools/etc/user.yaml)
- Security release: security patches aren't listed if etc/user.yaml doesn't contain the Bugzilla credentials. As the tickets are private.
- The branch name might be wrong at the end of the file.
    It will likely be wrong when doing a security release (e.g. fix it with 20.05.x instead of 20.05.x-security)
- If there are no features/enhancements/bugfixes the current script may generates empty sections for them.
- The system requirements and recommendations should point to the wiki page.

Just make sure everything is sound. Once you are done, you should commit the release notes:

$ git add misc/release_notes
$ git commit -m 'Update release notes for YY.MM.XX release'
$ git commit --amend -s

It's not a bad idea at this point to push your branch upstream one last time and watching all the tests pass before tagging it for release.

Tag the release

Tagging will prompt for your GPG passphrase. It is done issuing:

$ git tag -a v17.05.03 -s -m 'Koha release 17.05.03'

Note the two digits. You can now push the new tag doing:

$ git push upstream v17.05.03

Roll the release

This step creates the release tarball. It needs the release-tools repo. You have to run

$ mkdir releases
$ ~/path-to/release-tools/roll_release.sh 17.05.03 local_release_branch

↑↑↑ watch out for the local_release_branch in case of a security release when you might use another local branch than usually.

It will create the following files under the releases dir:

• koha-17.05.03.tar.gz
• koha-17.05.03.tar.gz.MD5
• koha-17.05.03.tar.gz.MD5.asc
• koha-17.05.03.tar.gz.sig

You then have to upload the generated files for distribution:

$ scp releases/koha-17.05.03.tar.gz* download.koha-community.org:/home/uploads/files/

Then log into the server.

Older versions of the same release should be moved into the old_releases directory.

If you create latest stable release, recreate symlink koha-latest.tar.gz.

Public announcement

People should know there's a new Koha release. We announce it by email and posting it on the Koha Community site.

Post on the community site

The current community site is a Wordpress site. You should have a valid credentials (https://koha-community.org/wp-admin) and create a new Post. It should be titled 'Koha YY.MM.XX released'. We recommend using the Text editor instead of the Visual one. You can copy the HTML source code of the generated release notes into the Text editor with very few changes necessary:

Just change the first paragraphs of the generated release notes a bit, for example:

The Koha community is proud to announce the release of version 17.05.03.
This is a maintenance release and contains many bugfixes and enhancements.
  
As always you can download the release from:
https://download.koha-community.org

This is my first release as the 17.05.XX release maintainer.
Thank you very much to everyone involved in this release.

Please continue reading for the details this release.

<!--more-->

Don't forget More entry (<!--more-->), that will split the announcement so it looks fine in the front page.

Choose the right categories: Announcements, release and <your branch>.

Email announcement

Email announcements are sent to the user list koha@lists.katipo.co.nz with a subject that should read like 'Koha 17.05.03 released'.

Note: keep in mind that the release announcement will be more than 40k and will be subject to moderation. So, be patient :-D

And that's it :-D.

Coordinated security releases

Sometimes it will be necessary to do a coordinated release of all maintained versions at the same time because of security related fixes. It can be a security-only urgent release or a non-urgent release following the usual schedule. The non-security workflow must still be followed but with some additions and changes:

• RM initiates a security release by emailing the RMaints and the Packaging Manager and provide them with the necessary information, like a list of bugs to push.
• RMaints push the security patches from Bugzilla to the protected security repository. The non-security patches are still pushed to the normal branches.

git remote add security git@git.koha-community.org:Koha-community/security.git
git push security HEAD:new/security-release-foo

• On the day of the release, in case of a non-urgent release, the normal branch must be merged into the security branch.
• For a urgent release, the security branch must contain only the security patches. The regular patches might have string changes which will be untranslated. So to avoid that, the security branch must created from the last release, not from the public branch that might have additional patches.
• Urgent release: don't do the translation steps. Depending on month's date, the translation platform will be synced with the public branch. Since we base the security branch on the last release, if the public branch has string changes then it will cause untranslated strings for the security release.
• String changes in security releases can't be translated as the current workflow requires the patches to be in the public branch. And some time (a week) for the translators to have a chance to translate.
• RMaints generate the release notes as usual. The Bugzilla credentials in release-tools/etc/user.yaml allow the release tools to access the security bugs.
  • Urgent release: fix the bug count wording in the release notes: https://gitlab.com/koha-community/release-tools/-/issues/30
• RMaints push the result (after applying the regular workflow) to the security branch and the tag to the security repository:

$ git push security v17.05.03

• RMaints roll tarballs and upload them to their non-public home directory on the download server:

$ scp releases/koha-17.05.03.tar.gz* download.koha-community.org:/home/<youruser>/

• RMaints create drafts of the release notes on koha-community.org - unpublished, but ready for publishing.

• Once an RMaint has finished those steps, they will inform the others.
• The Packaging Manager will build the packages from the security repository.
• The RMaint of the most recent release (RMaint1) coordinates the last steps with the Packaging Manager. RMaint1 then moves the tarballs into the public directory.
• Non urgent release: RMaints push the content of the security branch to the public branch. No force push should be required as a merge was done earlier.
• Urgent release: RMaints merge the security branch into the public branch. And then push to the public branch. That way, no force push needed and the usual workflow can continue and the security patches are in the public branch.
• RMaints also push the tag to the public repository.
• RMaints publish the drafts on the website.
• RMaint1 and sends an email announcing the security release to the mailing list.
• The RMaint1 changes the bug status from security project to Koha project after pushing to master.
• RMaints change the bug status and add comments after that.
• Urgent release: RMaints go through the already backported but unreleased bugs and bump the "Version(s) released in:". Because the out of schedule security release caused the version of the next regular release to not be the same as planned.
• Urgent release: RMaints go through their updatedatabase.pl file or db_revs directory to increment the version (the 3rd part of the number) of the pending/unreleased DBRevs. Those were on the public branch and not included in the security release. So their planned version has been replaced by the security release. Which mean that installs that upgraded to it won't have the DBRevs executed for those pending patches when they will be released next release.

• Urgent release: next regular release will be impacted: Release notes generation will have some things messed up. Namely the security patches will be listed again. Because to get the backported patches before the urgent release. We will set the beginning of the commit range from February's release. So it will get the security patches again. Fix: manually remove them from the release notes. Along with the security fix section and fix the overall number of patches/fixes.

Backporting a patch with a new perl dependency

In manual or automated tests you will get an error about the missing dependency.

1. Install manually the missing dependency and retest
2. Ensure that the patchset adds the dependency to ./cpanfile (PerlDependencies.pm for 19.11.X)
   1. If not, do a followup patch if you can or ask for help
3. Push
4. Disregard the CI that will fail
5. Wait for the koha-common package for your branch to be rebuilt, (rebuild happens every 60 mins)
   1. Monitor this: http://debian.koha-community.org/koha-staging/pool/main/k/koha/?C=M;O=D
6. When it's done, wait for koha-testing-docker images to be rebuilt
   1. Monitor this: https://gitlab.com/koha-community/koha-testing-docker/-/pipelines
7. Rerun the CI, if you can't launch manual builds ask someone who have the rights

Tips and Tools

• One can use a slightly adapted version of the release manager pre-push hook to check your work before pushing to your maintenance branch: hook

• I use the follow git post-commit hook as a simple way to track whether a fix I've backported has introduced a new author that should be added to the history.txt file and about page.

#!/bin/bash

AUTHOR="$(git log -1 --pretty=format:'%an' HEAD)"
FILE="/home/martin/kohaclone/docs/history.txt"

if grep -q "${AUTHOR}" ${FILE}; then
 tput setaf 2; echo "Existing Author: \"${AUTHOR}\""; tput sgr0;
 exit;
else
 tput setaf 1; echo "New Author: \"${AUTHOR}\""; tput sgr0;
fi;

exit;

PGP/GPG and checksums clarifications

koha-$VERSION.tar.gz.sig

It allows to verify that koha-$VERSION.tar.gz has been signed by the RMaint's public key. Which needs to be added to the GPG keyring from a trusted source.

gpg --verify koha-$VERSION.tar.gz.sig
↓↓↓
Good signature from $NAME_OF_RMAINT

koha-$VERSION.tar.gz.MD5

It allows to check the integrity of koha-$VERSION.tar.gz against the checksum in koha-$VERSION.tar.gz.MD5 It doesn't guaranty that the file come from the RMaint and has not been compromised. It's just for checking that the download hasn't corrupted the file.

md5sum -c koha-$VERSION.tar.gz.MD5

koha-$VERSION.tar.gz.MD5.asc

It contains the same checksum as above but it's wrapped in a section and is followed by it's signature. Example:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

36002d5acf2a1ca5957d5f03cdfaac2d  koha-20.11.07.tar.gz
-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQRT86JTIbcYFBCjtfABQKyjt9hwuwUCYNrpxAAKCRABQKyjt9hw
u8t/APwLhxg5Zh7qbOC8Dgko2+CxTepCPq4daThHzY80cEJjBwEA7v3xW9WOT5Cc
tgRa9lpkUtin0baSZa7URcmgcxUceAQ=
=Kxx0
-----END PGP SIGNATURE-----

Verify:

gpg --verify koha-$VERSION.tar.gz.MD5.asc
↓↓↓
Good signature from $NAME_OF_RMAINT
gpg: WARNING: not a detached signature; file 'koha-$VERSION.tar.gz.MD5' was NOT verified!

The warning tells us that the .MD5 file isn't verified. Indeed, because only it's copy in the .MD5.asc file has been. So you would need to check that the .MD5 file content matches the one from the content section of the .MD5.asc file.

Release notes generation using the dedicated docker image

Get/Update the image

docker pull registry.gitlab.com/koha-community/release-tools

Run it to get a shell inside

docker run \                                                                                                                                                                                                                       
-v /PATH_TO_YOUR/release-tools/etc/user.yaml:/release-tools/etc/user.yaml \
-v /PATH_TO_YOUR/koha-code:/koha \
-v /OPTIONAL/PATH_TO_YOUR/persistent-files/bash_history-release-tools-root:/root/.bash_history \
-ti registry.gitlab.com/koha-community/release-tools \
bash

Generate the release notes

Example with releasing v20.05.15

export PERL5LIB=/koha
cd release-tools/
git pull https://gitlab.com/koha-community/release-tools master
cd /koha/
 
../release-tools/bin/koha-release v20.05.14..HEAD notes
chown 1000:100 misc/release_notes/*
# see normal workflow for the details
# fix the release notes belonging to root. UID and GID might need to be something else than 1000 and 100 to match your host system
 
../release-tools/bin/koha-release v20.05.14..HEAD html misc/release_notes/release_notes_20_05_15.html
chown 1000:100 misc/release_notes/*