procedures: update post-powercut doc (#45)

Author: wizzdom

docs/procedures/post-powercut.md

@@ -2,24 +2,33 @@
 id: post-powercut
 aliases:
   - Post-powercut Todo List
-tags: []
+tags:
+  - powercut
+  - todo
 created: 2023-12-05T01:36:11
-modified: 2024-01-31T08:23:37
-title: Post-Powercut Todo List
+modified: 2024-09-27T13:35:28
+title: post-powercut
 ---
 
 # Post-powercut Todo List
 
 A list of things that should be done/checked immediately after a power cut:
 
-- Check KVM, hit ctrl+D on minerva to make sure it boots.
-- Check KVM, hit F1 on sprout to make sure it boots
-- Check KVM, sometimes you need to press F1 on carbon for it to boot
-- Stop Exim on the mail server (Morpheus) until minerva (NFS) is online.
-- If LDAP is down, you'll need to use the ALOM to do the next step.
-- Check that ldapclient started (svcs -xv). If it didn't, run svcadm clear ldap/client to make it start. This usually happens because murphy comes back before morpheus does, and the LDAP client won't start due to lack of an LDAP server.
-- Apache on [hardcase](../hardware/nix/hardcase.md) sometimes tries to start before networking is finished starting. To fix it, disable/re-enable it a few times. This usually makes it turn on.
-- [paphos](../hardware/paphos.md) is old and sometimes its time will become out of sync. To make sure its time is accurate, run:
+- Ensure the [`aperture`](../hardware/aperture/index.md) servers have the correct IP addresses:
+	- `eno1` should have the internal IP address (`10.10.0.0/24`) - this should be reserved by DHCP on [`mordor`](../hardware/network/mordor.md)
+	- `eno2` should have *no IP address*
+	- `br0` should have the external IP address (`136.206.16.0/24`) - this should also be reserved by DHCP on [`mordor`](../hardware/network/mordor.md)
+- If the [`bastion-vm`](../services/bastion-vm.md) fails to start, check:
+	- `/storage` is mounted `rw` on each [`aperture`](docs/hardware/aperture/index.md) server
+	- `br0` is present and configured on each [`aperture`](docs/hardware/aperture/index.md) server
+	- `vm-resources.service.consul` is running and `http://vm-resources.service.consul:8000/bastion/bastion-vm-latest.qcow2` is accessible
+	- if the `latest` symlink points to a corrupted image, `ln -sf` it to an earlier one
+- All the [`nixos`](docs/procedures/nixos.md) boxes rely on [`DNS`](docs/services/bind.md) for [`ldap`](docs/services/ldap.md) and [`nfs`](docs/services/nfs.md):
+   - Make sure bind is running on [`paphos`](docs/hardware/paphos.md)
+   - mount `/storage`
+   - `systemctl restart` `httpd`, `php-fpm-rbusers-*` and `ldap`
+- Apache on [`hardcase`](../hardware/nix/hardcase.md) sometimes tries to start before networking is finished starting. To fix it, disable/re-enable it a few times. This usually makes it turn on.
+- [`paphos`](../hardware/paphos.md) is old and sometimes its time will become out of sync. To make sure its time is accurate, run:
 
 ```bash
 sudo service ntp restart

docs/services/user-vms.md

@@ -38,7 +38,7 @@ sudo ip link set dev br0 up
 We'll be adding a physical interface to this bridge to allow it to communicate with the external ([UDM](../hardware/network/mordor.md)) network.
 
 ```bash
-sudo ip link set eno1 master br0
+sudo ip link set eno2 master br0
 ```
 
 You'll need to assign an IP address to the bridge interface. This will be used as the default address for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this is to create a DHCP static lease on the [UDM](../hardware/network/mordor.md) for the bridge interface MAC address.