Committers' Guide

Apache Isis documentation is broken out into a number of user, reference and "supporting procedures" guides.

The user guides available are:

The reference guides are:

The remaining guides are:

This guide provides guidance for Apache Isis' own committers.

If a diff file has been provided, then it can easily be applied using eGit’s wizards (Eclipse’s Git integration)…

If a patch file has been provided, then it can be applied using command line tools.

The overall process is as follows:

This screencast also shows the process.

The script uses 'jq' to parse JSON. To install:

The syntax is:

where:

The listing below shows the steps taken by the script:

The build now commences. Once done, the script continues:

The screenshot belows shows the history we end up with:

This shows the fork being merged into the temporary branch ("ISIS-1162_pr-31"), then some further tidy-up, and finally the merging of the temporary branch into mainline.

Note that there is no rebasing in this model. This is intentional: when the merged branch is pushed, github will automatically close the original pull request.

Before releasing core, ensure there is consensus on the dev mailing list that this is the right time for a release. The discussion should include confirming the version number to be used, and to confirm content.

These discussions should also confirm the version number of the module being released. This should be in line with our semantic versioning policy.

Make sure you have a JIRA ticket open against which to perform all commits. In most cases a JIRA ticket will have been created at the beginning of the previous release cycle.

We use environment variables to parameterize as many of the steps as possible. For example:

Obviously, alter $ISISREL as required, and bump $ISISRC for re-releasing following an unsuccessful release.

Set the HEAD of your local git repo to the commit to be released. This will usually be the tip of the origin’s master branch. Then, create a release branch for the version number being released; eg:

The release is performed on a branch; if we are successful, this branch will be merged back into master.

Check that there are no snapshot dependencies:

The only mention of SNAPSHOT should be in the revision property of the root core/pom.xml file.

Previously we used the maven-release-plugin which took responsibility for bumping the <version> from -SNAPSHOT to release, and then onto the next -SNAPSHOT. However, this plugin does not seem to be compatible with the support for ${revision} property introduced in Maven 3.5.0 in order to support CI/CD use cases; at least we couldn’t make it work.

So, instead, we perform the edit of ${revision} in the three pom.xml files (for core and the two archetypes):

Change all of these to the release version, using:

so that they should read:

After release, there is a similar step at the end to reset back to 1.0.0-SNAPSHOT.

First, we release core. Switch to the appropriate directory:

The Apache Isis archetypes are reverse engineered from example applications. Once reverse engineered, the source is checked into git (replacing any earlier version of the archetype) and released.

There are currently two archetypes, simpleapp and helloworld. In this section we release simpleapp, in the following section we release helloworld.

Switch to the directory containing the simpleapp example:

We now repeat the archetype release procedure, this time for the helloworld example app.

Start by switching to the directory containing the helloworld example:

The mvn deploy commands will have uploaded release artifacts for both core and the simpleapp archetype into a newly created staging repository on the ASF Nexus repository server.

Log onto repository.apache.org (using your ASF LDAP account):

And then check that the release has been staged (select staging repositories from left-hand side):

If nothing appears in a staging repo you should stop here and work out why.

Assuming that the repo has been populated, make a note of its repo id; this is needed for the voting thread. In the screenshot above the id is org.apache.isis-008.

After checking that the staging repository contains the artifacts that you expect you should close the staging repository. This will make it available so that people can check the release.

Press the Close button and complete the dialog:

Nexus should start the process of closing the repository.

All being well, the close should (eventually) complete successfully (keep hitting refresh):

The Nexus repository manager will also email you with confirmation of a successful close.

If Nexus has problems with the key signature, however, then the close will be aborted:

Use gpg --keyserver hkp://pgp.mit.edu --recv-keys nnnnnnnn to confirm that the key is available.

At the beginning of the release process we bumped the revision property to the release version, ie ${ISISREL}. With the release now deployed we now need to reset the revision back down to the base snapshot, ie 1.0.0-SNAPSHOT.

In the root of the Apache Isis repo, we do this for all three pom.xmls with:

Push the release branch to origin:

and also push tags for both core and the archetype:

Once the artifacts have been uploaded, you can call a vote.

In all cases, votes last for 72 hours and require a +3 (binding) vote from members.

Whenever a release manager announces a vote on a release (as per the release process) on the dev mailing list, it is the responsibility of the project’s PMC to cast their vote on the release. Anyone else can also vote, but only members of the Apache Isis PMC’s vote are binding.

Per this ASF documentation, the legal requirements for an ASF release are:

Note that the binaries are not an ASF release, they merely exist on the Maven central repo as a convenience. That said, you might also want to verify the release by pulling the binaries from the Maven staging repository. Details of how to do this are also documented below.

You will also require the following commands/tools:

The easiest way to verify the source artifacts is to use a script that automates the steps. Run these commands (they should also be listed in the VOTE email):

Just copy and paste these commands to your terminal, and run.

The verify-isis-release.sh script performs the following:

Assuming this completes successfully, you can then test the two applications generated from the simpleapp and helloworld archetypes:

You can if you wish perform some additional optional checks, listed in the sections below. Alternatively, you can cast your vote.

You can verify the binary releases by configuring your local Maven install to point to the Maven staging repository (or repositories) and then using them, eg to run the HelloWorld archetype or the SimpleApp archetype and running the resultant app.

Configuring your local Maven install amounts to updating the ~/.m2/settings.xml file:

where the repository URL is as provided in the VOTE email. If there is more than one repository (as is sometimes the case if multiple components have been released), then repeat the <repository> section for each.

Once the vote has completed, the staging repositories will be removed and so you should deactive the profile (comment out the <activeProfile> element). If you forget to deactive the profile, there should be no adverse effects; Maven will just spend unnecessary cycles attempting to hit a non-existent repo.

The Apache Creadur project exists to provide a set of tools to ensure compliance with Apache’s licensing standards.

The main release auditing tool, Apache RAT is used in the release process.

Creadur’s remaining tools - Tentacles and Whisker - are to support the verification process.

For example, Tentacles generates a report called archives.html. This lists all of the top-level binaires, their LICENSE and NOTICE files and any LICENSE and NOTICE files of any binaries they may contain.

Validation of the output at this point is all still manual. Things to check include:

In this report, each binary will have three links listed after its name '(licenses, notices, contents)'

When you have made the above checks (and any other checks you think may be relevant), cast your vote by replying to the email thread on the mailing list.

If you are casting -1, please provide details of the problem(s) you have found.

Post the results to the dev@isis.a.o mailing list:

using the body (alter last line as appropriate):

Replace the -RCn tag with another without the qualifier.

You can do this using the scripts/promoterctag.sh script; for example:

This script pushes the tag under refs/tags/rel. As per Apache policy (communicated on 10th Jan 2016 to Apache PMCs), this path is 'protected' and is unmodifiable (guaranteeing the provenance that the ASF needs for releases).

Then, continue onto the next section for the steps to promote and announce the release.

From the ASF Nexus repository, select the staging repository and select 'release' from the top menu.

This moves the release artifacts into an Apache releases repository; from there they will be automatically moved to the Maven repository.

As described in the Apache documentation, each Apache TLP has a release/TLP-name directory in the distribution Subversion repository at https://dist.apache.org/repos/dist. Once a release vote passes, the release manager should svn add the artifacts (plus signature and hash files) into this location. The release is then automatically pushed to http://www.apache.org/dist/ by svnpubsub. Only the most recent release of each supported release line should be contained here, old versions should be deleted.

Each project is responsible for the structure of its directory. The directory structure of Apache Isis reflects the directory structure in our git source code repo:

If necessary, checkout this directory structure:

Next, add the new release into the appropriate directory, and delete any previous release. The upd.sh script (also downloadable from this gist) can be used to automate this:

The script downloads the artifacts from the Nexus release repository, adds the artifacts to subversion and deletes the previous version.

Double check that the files are correct; there is sometimes a small delay in the files becoming available in the release repository. It should be sufficient to check just the md5 or .asc files that these look valid (aren’t HTML 404 error pages):

Assuming all is good, commit the changes:

If the files are invalid, then revert using svn revert . --recursive and try again in a little while.

This is currently a placeholder.

TODO:

The idea is to use isis-webdocker to create a new image, then upload.

Prereqs are:

... because isis-webdocker/pom.xml references docker-hub.

In the main isis repo (ie containing the asciidoc source):

Now we need to publish the website:

Now, in the isis-site repo itself:

At this point the files in the root (directly under content) will still be out of date; the publish process simply preserves whatever is in content/versions/current. Also, the SNAPSHOT will be missing (above it was renamed to current). And, we also need to make sure that any future publishing of snapshots has the correct version in the navbar.

Therefore:

If everything looks ok, then push the changes to make live.

Announce the release to users mailing list.

For example, for a release of Apache Isis Core, use the following subject:

And use the following body (summarizing the main points as required):

Log onto the Apache blog and create a new post. Copy-n-paste the above mailing list announcement should suffice.

Because we release from a branch, the changes made in the branch (changes to pom.xml made by the maven-release-plugin, or any manual edits) should be merged back from the release branch back into the master branch:

With the release complete, now is a good time to bump versions of dependencies (so that there is a full release cycle to identify any possible issues).

You will probably want to create a new JIRA ticket for these updates (or if minor then use the "catch-all" JIRA ticket raised earlier for the next release).

This is also a good time to make source code has been cleaned up and formatted according to the Apache Isis and ASF conventions. Use this Eclipse template and this import order.

Finally, push the changes up to origin:

The (non-ASF) Incode Platform should also be released, as per their release guide.

Post the results to the dev@isis.a.o mailing list.

For example, use the following subject for a vote on Apache Isis Core:

using the body (alter last line as appropriate):

Tidy up remote branches in the git repo:

Drop staging repositories:

Finally, rewind the release branch to prior to the previous release candidate, and continue from there.

Before you start, make sure you’ve defined the snapshots repo in your local ~/.m2/settings.xml file:

where xxxxxxx and yyyyyyy are your Apache LDAP username and password. For more information, see these ASF docs.

{note It is also possible to configure to use .ssh secure keys, and thereby avoid hardcoding your Apache LDAP password into your .m2/settings.xml file. A description of how to do this can be found, for example, here. }

Before deploying the snapshot, perform a quick sanity check.

First, delete all Isis artifacts from your local Maven repo:

Next, check that the framework builds ok:

Confirm that the versions of the Isis artifacts now cached in your local repository are correct (both those pulled down from Maven central repo, as well as those of the component built locally).

Deploy the framework using:

This will deploy all the modules that make up a release.

To confirm that they are present, browse to Apache’s Nexus repository manager and search for "isis".

Create a remote fork/clone of the Apache Isis repository (eg using github or bitbucket or similar), and ensure that your local fork specifies this remote.

Also, set up a CI server against your fork/clone, to build against any branches called origin/interim/*. To build it should use the command:

that is, skipping the example simpleapp and archetype; only core framework is built

The CI server should then also publish the resultant artifacts to a local Maven repository. For example, Jenkins provides post-build plugins to perform such a task.

Ensure that the framework builds ok using the same command that your CI server is set up to execute (see section above).

Deploy the framework using:

where xxx is the most recent release of Isis (to act as the base), and yyyyy is the name of the remote.

For example,

This will result in a new branch and tagged being pushed to the remote, with an appropriate version (details below).

As noted in the prereqs (above), the CI server configured should then detect the new branch (or tag, if you prefer), build the framework (skipping application and architecture, as in the prerequisites) and then publish the resultant artifacts to a private Maven repo.

The generated site is published by copying into the content/ directory of the isis-site git repo. You therefore need to check this out this repo.

The procedure assumes that two git repos (for isis itself on github, and for isis-site) are checked out into the same parent directory, eg:

One oddity; ASF’s script to upload the documentation to the web server checks out from the 'asf-site' branch, NOT from the 'master' branch. Therefore, in the isis.git repo site:

To preview changes (ignoring PDFs), run:

This will update the documentation and start a (Python) webserver so that the documents can be viewed.

If you want to also view the PDFs, use:

There’s generally little point in doing this, however.

The content directory for the published website contains not only the "current version" of the docs, but also has an archive of older versions of the document from earlier releases:

The earlier versions of the documentation are (obviously) not recreated when the documentation is published, only the current version is updated.

To publish the current version of the documents, switch to the adocs/documentation directory of the main isis-git.repo. Then run:

This does the following:

In other words, the effect of running publish.sh is in essence to overwrite content/versions/SNAPSHOT and to overwrite index.html, but leave everything else as it is.

Download and install GnuPG (gpg), version 1.4.10 or higher.

Then, edit ~/.gnupg/gpg.conf (on Windows, the file to edit is C:\Users\xxx\AppData\Roaming\gnupg\gpg.conf) so that the default is to generate a strong key:

The Apache Software Foundation requires that keys are signed with a key (or subkey) based on RSA 4096 bits. To do this:

Specify RSA key:

Specify key length as 4096 bits:

Specify key as non-expiring:

Enter your name, email and comment:

Real name: Xxx Xxxxxxxxx Email address: xxx@apache.org Comment: CODE SIGNING KEY You selected this USER-ID: "Xxx Xxxxxxxxx (CODE SIGNING KEY) xxx@apache.org"

Change (N)ame, ©omment, (E)mail or (O)kay/(Q)uit? O

You need a Passphrase to protect your secret key. Enter passphrase:

Provide a passphrase to secure your key.

GPG will goes on to generate your key:

The public key with id nnnnnnnn should now be stored in ~/.gnupg/pubring.pgp (on Windows 7, this is in c:/Users/xxx/AppData/Roaming/gnupg/pubring.pgp).

To confirm the key has been generated, use:

The key Id is the one true way to identify the key, and is also the last 8 digits of the fingerprint. The corresponding secret key for id nnnnnnnn is stored in ~/.gnupg/secring.pgp (on Windows 7, this is in c:/Users/xxx/AppData/Roaming/gnupg/secring.pgp).

It’s also worth confirming the key has the correct preference of algorithms (reflecting the initial configuration we did earlier). For this, enter the gpg shell for your new key:

where nnnnnnnn is your key id. Now, use the 'showpref' subcommand to list details:

The Digest line should list SHA-512 first and SHA-1 last. If it doesn’t, use the "setpref" command:

Finally, remember to take a backup of your key and the keyring (ie, backup the .gnupg directory and its contents).

It’s recommended to use a subkey with an expiry date to sign releases, rather than your main, non-expiring key. If a subkey is present, then gpg will use it for signing in preference to the main key.

To create a subkey Enter the gpg shell using (the identifier of) your main key:

Type 'addkey' to create a subkey, and enter your passphrase for the main key:

Select (6) to choose an RSA key for encryption:

Specify that the key is valid for 1 year:

Quit the gpg shell; you now have a subkey.

It’s good practice to generate a number of revocation certificates so that the key can be revoked if it happens to be compromised. See the gpg page on the Apache wiki for more background on this topic.

First, generate a "no reason specified" key:

Select 0.

Provide a description:

Confirm this is ok.

The file revoke-nnnnnnnn-0.asc should be created: Then, backup this file.

Now repeat the process to create two further revocation certificates:

Specify reason as "1 = Key has been compromised"

and:

Specify reason as "3 = Key is no longer used"

Backup these files also.

It is also necessary to publish your key. There are several places where this should be done. In most cases, you’ll need the "armored" " (ie ASCII) representation of your key. This can be generated using:

where nnnnnnnn is the id of your public key.

You’ll also need the fingerprint of your key. This can be generated using:

The output from this command includes a line beginning "Key fingerprint", followed by a (space delimited) 40 character hexadecimal fingerprint. The last 8 characters should be the same as the key id (nnnnnnnn).

It is strongly advised that the contributor attend a key signing party at an Apache event, in order that other Apache committers/members can in person verify their identity against the key. The process for this is described here and here.

The Maven release plugin will automatically sign the release, however it is necessary to update the ~/.m2/settings.xml file with your GPG acronym passphrase in order that it can use your secret key. This is defined under a profile so that it is activated only when we perform a release (as defined by [org,apache:apache] parent POM.

Therefore, make the following edits:

In addition, to allow the release plugin to tag SVN changes, you must either add in your LDAP username/password or configure .ssh:

Apache Isis releases are built using Java 7, enforced using the maven toolchains plugin. Ensure that Java 7 is installed and the toolchains plugin is configured, as described in the contributors' guide.

The most important configuration you require is to set up public/private key pair. This is used by the maven-release-plugin to sign the code artifacts. See the page on key generation for more details.

In order to prepare the release, you’ll (need to) have a ~/.gnupg directory with the relevant files (gpg.conf, pubring.gpg, secring.gpg etc), and have gpg on your operating system PATH.

During the release process the maven-deploy-plugin uploads the generated artifacts to a staging repo on the Apache repository manager. This requires your Apache LDAP credentials to be specified in your ~/.m2/settings.xml file:

where xxxxxxx and yyyyyyy are your Apache LDAP username and password. For more information, see these ASF docs.

Also, set up keyphrase for gpg; this avoids being prompted during release:

These notes recommend how contributors should work with git. To understand these notes, the only real concepts that you need to grok are:

For more background reading, see:

And, of course, there is loads of good advice on stackoverflow.com

To complete the procedures documented here, you’ll need ssh access to minotaur.apache.org. For example:

When prompted, provide passphrase for private key.

Currently we don’t distinguish between committers and PMC members: every committer is automatically invited to also be a member of the PMC.

Further notes on the steps for new PMC members can be found here.