Release and deploy procedure

This document attempts to outline the steps that are needed to cut a release and then get it deployed on the production environment. It is accurate as of the 11th of April 2017.

Cutting the tag

Hilary

On release0 do:

• cd /opt
• rm -r oae
• puppet agent --no-daemonize --onetime --verbose
  • Which will install the master branch of Hilary.
• service hilary stop
  • Hilary must not be running through the rest of this process.
• cd /opt/oae
• grunt test
  • Ensure that there are no failures. If they are you should be able to explain thoroughly why that is the case.
• git reset --hard HEAD
  • Which will clear out items like log files, target, previous releases, ..
• bin/release -s -r origin -v 10.1.1
  • This creates the release and it will add a commit that:
    • bumps the version in package.json
    • runs npm shrinkwrap
  • At this point you will have to provide your GitHub username and password, twice.
  • It will create a tag that points to the new commit.
  • It will add a commit that takes the shrinkwrap file back out so the latest dependencies can be pulled in during development.
  • It will push the commits and tag to GitHub.

Once everything is pushed to github, TravisCI will do its thing. See TravisCI section below for continuation on Hilary.

3akai-ux

The release process for 3akai-ux is very similar, on release0:

• cd /opt
• rm -r 3akai-ux
• puppet agent --no-daemonize --onetime --verbose
  • Which will install the master branch of 3akai-ux.
• service hilary stop
  • Hilary must not be running through the rest of this process.
• cd /opt/3akai-ux
• npm install
• bin/release -s -r origin -v 10.1.1
  • It will add a commit that:
    • bumps the version in package.json
  • At this point you will have to provide your GitHub username and password, twice.
  • It will create a tag that points to the new commit.
  • It will push the commit and tag to GitHub.

Once everything is pushed to github, TravisCI will do its thing. See TravisCI section below for continuation on 3akai-ux.

Travis CI

For both Hilary and 3akai-ux, travis will:

• Run the unit and integration tests (if there are known intermittent failures in the tests, you might want to replace the test script with /bin/true)
• If successful:
  • (UI only:) Minify all static assets
  • Create a tarball that includes the code and all (node) dependencies
  • Upload the tarball to Amazon S3 to the oae-releases bucket. The generated tarball will include:
    • The tag name (e.g., 10.1.1)
    • The node version on which the tests were run (e.g., 0.10.30)

Verifying releases on QA

On QA's puppet master node we configure a single QA machine (say qa0) to test run the release by configuring environments/qa/hiera/qa0.json with:

"app_install_method": "archive",
"app_install_config": {
    "version_major_minor": "10.1",
    "version_patch": "1",
    "version_nodejs": "0.10.30"
},
"app_ui_path": "/opt/3akai-ux/optimized",

"ux_root_dir": "/opt/3akai-ux",
"ux_install_method": "archive",
"ux_install_config": {
    "version_major_minor": "10.1",
    "version_patch": "1"
}

Then on the qa server run redeploy.sh (/opt/scripts/redeploy.sh) or do the following:

service hilary stop
service nginx stop
cd /opt
rm -rf 3akai-ux
rm -rf oae
puppet agent --no-daemonize --onetime --verbose

Test. Now test it again.

Rolling releases out on production

1. Upgrade Guide notes.

Read the Upgrade Guide notes for the new release and make sure you know if there's anything that needs to happen before or after you roll out the release. If there are no notes, it might be worth pinging the development team whether they've forgotten to put them in the wiki.

If the release procedure requires a script running on a Hilary instance that must not be running at the time, then not only will you need to stop Hilary service hilary stop but you should also stop the puppet agent service puppet stop otherwise it may start Hilary up again.

2. Update puppet-hilary

Update the version numbers in puppet-hilary's production environment. Push your changes to the oaeproject GitHub repository. Do NOT pull it on the production puppet master node. The deploy script will do this for you.

3. Perform roll-out

Performing the actual roll-out can be done by:

• ssh to bastion node as root.
• cd oae-fabric
• fab cluster.oae.upgrade

This will:

• Try upgrading a single node that isn't user-facing (pp0)
• If all goes well it will:
  • Upgrade all other non-user-facing app nodes (activity, pp)
  • Upgrade half of the REST serving nodes (app0, app1)
• If all goes well it will:
  • Upgrade web0
• If all goes well it will:
  • Upgrade the remaining REST serving nodes (app2, app3)

Other thoughts

It's usually a good idea to tail the logs on the syslog node (in another terminal) and tail the logs to see what is going on tail-hilary -f | filter-bunyan

In case something goes wrong and you can't see an immediate fix for it, try and revert to the previous release. This can be done by restoring the previous version numbers in puppet-hilary and repeating the release procedure.

Update the Release Upgrade Guide in case there were any instructions under the "Next Release" section.