Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update prod deploy docs #188

Merged
merged 2 commits into from
Aug 14, 2024
Merged

Update prod deploy docs #188

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9098aa6
remove old prod servers and comment out 04 by default
jxjj Aug 13, 2024
572c5d7
update prod deploy docs - new hosts, ansible tower
jxjj Aug 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions config/deploy/production.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,9 @@
role :app, %w[[email protected]]
role :web, %w[[email protected]]
role :db, %w[[email protected]]
role :app, %w[[email protected]]
role :web, %w[[email protected]]
role :db, %w[[email protected]]
# role :app, %w([email protected])
# role :web, %w([email protected])
# role :db, %w([email protected])
# role :app, %w[[email protected]]
# role :web, %w[[email protected]]
# role :db, %w[[email protected]]

# Configuration
# =============
Expand Down
123 changes: 45 additions & 78 deletions deploy_to_production.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ Deploying `z` to production involves a few more steps since there are multiple s
## 👥 THE PLAYERS

- <https://z.umn.edu> points to a load balancer. The service provides an SSL cert and balances traffic between:
- `cla-z-prd.oit.umn.edu`: Production server 1.
- `cla-z-prd-2.oit.umn.edu`: Production server 2.
- `cla-z-prd-web-03.oit.umn.edu`: Production server 1. (Note: `...-web-01` and `...-web-02` are older servers, no longer in service)
- `cla-z-prd-web-04.oit.umn.edu`: Production server 2.
- Your local machine will be used to verify that each production server deployment was successful.

Just like `dev` and `staging` deployments, we'll use Ansible for updating the platform and Capistrano for deploying the app.
Expand All @@ -27,78 +27,57 @@ Just like `dev` and `staging` deployments, we'll use Ansible for updating the pl

#### PREP

1. Connect to VPN
2. Start tailing log files on production servers with `tail -f /swadm/web/z/current/log/lograge_production.log`
3. Open Ansible Playbook locally
4. Open z.umn.edu repo locally
5. Open `/etc/hosts` locally. Add entries for `cla-z-prd.oit.umn.edu` and `cla-z-prd-2.oit.umn.edu`. {: #config-etc-hosts }
1. Connect to VPN
2. Start tailing log files on production servers with `tail -f /swadm/web/z/current/log/lograge_production.log`
3. Open Ansible Playbook locally
4. Open z.umn.edu repo locally
5. Open `/etc/hosts` locally. Add entries for `cla-z-prd-web-03.oit.umn.edu` and `cla-z-prd-web-04.oit.umn.edu`. {: #config-etc-hosts }

```
# /etc/hosts
```
# /etc/hosts

# Z DEPLOYMENT
# Uncomment to force z.umn.edu to resolve to a particular
# production server. Used when testing that a production
# deploy is successful.
# Z DEPLOYMENT
# Uncomment to force z.umn.edu to resolve to a particular
# production server. Used when testing that a production
# deploy is successful.

# cla-z-prd
# 128.101.122.117 z.umn.edu
# cla-z-prd-web-03.oit.umn.edu
# 134.84.24.45 z.umn.edu

# cla-z-prd-2
# 128.101.122.224 z.umn.edu
```
# cla-z-prd-web-04.oit.umn.edu
# 134.84.24.47 z.umn.edu
```

> **🙋‍♀️ Wait?! Why do we need to do this?**
>
> If you type `cla-z-prd.oit.umn.edu` into your browser and try to sign in, it'll redirect you to `z.umn.edu`. Configuring `/etc/hosts` will make sure any redirects to `z.umn.edu` also resolve to `cla-z-prd.oit.umn.edu`.
> **🙋‍♀️ Wait?! Why do we need to do this?**
>
> If you type `cla-z-prd-web-03.oit.umn.edu` into your browser and try to sign in, it'll redirect you to `z.umn.edu`. Configuring `/etc/hosts` will make sure any redirects to `z.umn.edu` also resolve to `cla-z-prd-web-03.oit.umn.edu`.

### DEPLOYING TO EACH PRODUCTION SERVER (`cla-z-prd.oit.umn.edu`)
### DEPLOYING TO EACH PRODUCTION SERVER (`cla-z-prd-web-03.oit.umn.edu`)

Complete this for each server you're deploying to. In the example below, we'll use `cla-z-prd.oit.umn.edu` as our first target.
Complete this for each server you're deploying to. In the example below, we'll use `cla-z-prd-web-03.oit.umn.edu` as our first target.

1. Take `cla-z-prd-oit.umn.edu` out of the load balancing group. This is done with the help of a [LATIS System Engineer](https://neighborhood.cla.umn.edu/latis/people/latis-staff-list).
1. Take `cla-z-prd-web-03.oit.umn.edu` out of the load balancing group. This is done with the help of a [LATIS System Engineer](https://neighborhood.cla.umn.edu/latis/people/latis-staff-list).

2. Wait for connections to drain by monitoring the logfiles on `cla-z-prd-oit.umn.edu`. Once there's no traffic, proceed. (Note: There may be a small amount of bot (?) traffic directly to an individual host like `cla-z-prd`.).
2. Wait for connections to drain by monitoring the logfiles on `cla-z-prd-web-03.oit.umn.edu`. Once there's no traffic, proceed. (Note: There may be a small amount of bot (?) traffic directly to an individual host like `cla-z-prd`.).

3. Set your local computer to resolve `z.umn.edu` to the ip address of the `cla-z-prd-oit.umn.edu`. On a mac, this means editing `/etc/hosts` with:
3. Set your local computer to resolve `z.umn.edu` to the ip address of the `cla-z-prd-web-03.oit.umn.edu`. On a mac, this means editing `/etc/hosts` with:

```
# cla-z-prd
128.101.122.117 z.umn.edu
# cla-z-prd-web-03.oit.umn.edu
# 134.84.24.45 z.umn.edu
```

Verify that `z.umn.edu` resolves locally to the correct IP with `ping z.umn.edu`.
Verify that `z.umn.edu` resolves locally to the correct IP with `ping z.umn.edu`.

#### USE ANSIBLE TO CONVERGE HOST CHANGES

Ansible is not needed for every Z deployment, but will be required when doing things like bumping a ruby version.
Ansible is not needed for every Z deployment, but will be required when doing things like bumping a ruby or node version. See [Ansible playbook](https://github.umn.edu/latis-sw/ansible_playbooks).

If Ansible-ing, open the [ansible playbook](https://github.umn.edu/latis-sw/ansible_playbooks) locally, then:

1. Edit [z.yml](https://github.umn.edu/latis-sw/ansible_playbooks/blob/main/z.yml) so that `hosts` value references the correct hostname, `cla-z-prd`. See: [inventory.yml](https://github.umn.edu/latis-sw/ansible_playbooks/blob/main/inventory.yml) and [host_vars](https://github.umn.edu/latis-sw/ansible_playbooks/blob/main/host_vars/cla-z-prd.yml) for host options.

```yml
---
- hosts: cla-z-prd
vars:
ruby_version: "2.7.3"
pre_tasks:
...
```

2. Login to lastpass.

```sh
lpass login <[email protected]>
```

Ansible will use the Z Rails Application keys stored in lastpass.

3. Coverge the host:

```sh
ansible-playbook -i inventory.yml z.yml
```
1. Log in to [Ansible Tower](https://tower.oit.umn.edu/).
2. Go to Resources > Templates.
3. Choose `Z`, and launch
4. You'll be prompted for a _Limit_ – the hosts you want to scope the deploy to. Enter the host name as defined in Ansible, e.g. `cla-z-prd-3`, not `cla-z-prd-web-03.oit.umn.edu`.
5. Proceed

#### CAPISTRANO

Expand All @@ -110,12 +89,12 @@ Use Capistrano to deploy Rails:
# role-based syntax
...

role :app, %w([email protected])
role :web, %w([email protected])
role :db, %w([email protected])
# role :app, %w(swadm@cla-z-prd-2.oit.umn.edu)
# role :web, %w(swadm@cla-z-prd-2.oit.umn.edu)
# role :db, %w(swadm@cla-z-prd-2.oit.umn.edu)
role :app, %w[swadm@cla-z-prd-web-03.oit.umn.edu]
role :web, %w[swadm@cla-z-prd-web-03.oit.umn.edu]
role :db, %w[swadm@cla-z-prd-web-03.oit.umn.edu]
# role :app, %w[swadm@cla-z-prd-web-04.oit.umn.edu]
# role :web, %w[swadm@cla-z-prd-web-04.oit.umn.edu]
# role :db, %w[swadm@cla-z-prd-web-04.oit.umn.edu]
...
```

Expand All @@ -138,26 +117,14 @@ Use Capistrano to deploy Rails:

#### NEXT STEPS

1. Add `cla-z-prd.oit.umn.edu` back to load balanced group.
1. Add `cla-z-prd-web-03.oit.umn.edu` back to load balanced group.
2. Monitor traffic for unexpected errors once it's receiving traffic again.
3. If all good, proceed to next server in group: `cla-z-prd-2.oit.umn.edu` and repeat the steps above.
3. If all good, proceed to next server in group: `cla-z-prd-web-04.oit.umn.edu` and repeat the steps above.

### POST DEPLOY

After successful deployment:

- [x] Log in to Team Dynamix, and mark the CAB ticket as completed.
- [x] Merge the release branch into main and develop using `git flow release finish <version>`
- [x] Push merged branches and tags to git:

```
git checkout develop
git push

git checkout main
git push

git push --tags
```

- [x] Check github for [latest release](https://github.com/UMN-LATIS/z/releases)
- [x] Merge the release branch into main (and develop, if needed).
- [x] Create a Release on github, tagged with the latest version.
Loading