Merge pull request #8 from GOCDB/hartree

Replace old failover references with new failover

Author: gregcorbett

README.md

@@ -5,59 +5,59 @@ Author: David Meredith + JK
 
 This repo contains the service and cron scripts used to run a failover gocdb instance, includes the following dirs:
 * autoEngageFailover/
-  * Contians a Service script (```gocdb-autofailover.sh```) and child scripts that monitors the main production instance. If a prolonged outage is detected, the GOCDB top DNS alias 'goc.egi.eu' is swtiched from the production instance to the failover instance. This switch can also be performed manually when needed. 
+  * Contians a Service script (```gocdb-autofailover.sh```) and child scripts that monitors the main production instance. If a prolonged outage is detected, the GOCDB top DNS alias 'goc.egi.eu' is swtiched from the production instance to the failover instance. This switch can also be performed manually when needed.
 * importDBdmpFile/
-  * Contains a script that should be invoked by cron hourly (```1_runDbUpdate.sh```) to fetch and install a .dmp of the production DB into the local failover DB. This runs separtely from the autoEngageFailover process. 
+  * Contains a script that should be invoked by cron hourly (```1_runDbUpdate.sh```) to fetch and install a .dmp of the production DB into the local failover DB. This runs separtely from the autoEngageFailover process.
 * nsupdate_goc/
-  * Scripts for switching the DNS to/from the production/failover instance. 
+  * Scripts for switching the DNS to/from the production/failover instance.
 * archiveDmpDownload/
-  * Contains a script to download/archive dmp files in a separate process 
+  * Contains a script to download/archive dmp files in a separate process
 
 # Packages
-* The following scripts needs to be installed and configuired for your installation: 
+* The following scripts needs to be installed and configuired for your installation:
 ```
 /root/
   autoEngageFailover/         # Scripts to mon the production instance and engage failover
       |_ gocdb-autofailover.sh# MAIN SERVICE SCRIPT to mon production instance
       |_ engageFailover.sh    #   Child script, run if prolonged outage is detected
-      
+
   importDBdmpFile/            # Scripts fetch/install a .dmp of the prod data
-      |_ 1_runDbUpdate.sh     # MAIN SCRIPT that can be called from cron, invokes child scripts below 
+      |_ 1_runDbUpdate.sh     # MAIN SCRIPT that can be called from cron, invokes child scripts below
       |_ ora11gEnvVars.sh     #   Setup oracle env
-      |_ getDump.sh           #   Fetch a .dmp of the production data 
+      |_ getDump.sh           #   Fetch a .dmp of the production data
       |_ dropGocdbUser.sh     #   Drops the current DB schema
       |_ loadData.sh          #   Load the last successfully fetched DB dmp into the RDBMS
       |_ gatherStats.sh       #   Oracle gathers stats to re-index
       |_ pass_file_exemplar.txt   #   Sample pwd file for DB (rename to pass_file)
 
   nsupdate_goc/              # Scripts for switching the DNS to the failover
       |_ goc_failover.sh     #   Points DNS to failover instance
-      |_ goc_production.sh   #   Points DNS to production instance 
+      |_ goc_production.sh   #   Points DNS to production instance
 
   archiveDmpDownload/        # Contains script to download/archive dmp files in a separate process e.g from cron.daily
-      |_ archiveDump.sh      # Main script that dowloads dmp and saves in a sub-dir 
-      |_ archive/            # Contains archive/dmp files 
+      |_ archiveDump.sh      # Main script that dowloads dmp and saves in a sub-dir
+      |_ archive/            # Contains archive/dmp files
 ```
 
-## /root/autoEngageFailover/ 
+## /root/autoEngageFailover/
 Start in this dir. Dir contains the 'gocdb-autofailover.sh'
 service script which should be installed as a service in
 '/etc/init.d/gocdb-autofailover'. This service invokes
 'engageFailover.sh' which monitors the production instance
 with a ping-check. If a continued outage is detected;
 the script starts the failover procedure which includes the
-following: 
-* the gocdb admins are emailed, 
+following:
+* the gocdb admins are emailed,
 * the age of the last successfully imported dmp file is
-  checked to see that it is current, 
+  checked to see that it is current,
 * the hourly cron that fetches the dmp file is stopped (see
-  importDBdmpFile below), 
+  importDBdmpFile below),
 * <strike>symbolic links to the server cert/key are updated so they
-  point to the 'goc.egi.eu' cert/key</strike> (note, no longer needed as cert contains dual SAN) 
+  point to the 'goc.egi.eu' cert/key</strike> (note, no longer needed as cert contains dual SAN)
 * the dnscripts are invoked to change the dns (see
   nsupdate_goc below).  
 
-## /root/importDBdmpFile/ 
+## /root/importDBdmpFile/
 Contains scripts that fetches the .dmp file and install this
 dmp file into the local Oracle XE instance. The master script
 is '1_runDbUpdate.sh' which needs to be invoked from an hourly
@@ -70,29 +70,29 @@ cron:
 /root/importDBdmpFile/1_runDbUpdate.sh
 ```
 
-You will also need to: 
+You will also need to:
 * generate a public/private key pair using `ssh-keygen` and ensure the public
 key is present on the host with the database dmp file.
 * populate `importDBdmpFile/failover_TEMPLATE.sh` with
 appropriate values and copy it to `/etc/gocdb/failover.sh`
- 
+
 ## /root/nsupdate_goc/
 Contains the nsupdate keys and nsupdate scripts for switching
 the 'goc.egi.eu' top level DNS alias to point to either the
-production instance or the failover. 
+production instance or the failover.
 
 
 ## /root/archiveDmpDownload/
 Contains a script that downloads the dmp file and stores the file in the archive/ sub-dir.
-The script also deletes archived files that are older than 'x' days. 
-This script can be called in a separate process, e.g. from cron.daily to build a 
-set of backups. 
+The script also deletes archived files that are older than 'x' days.
+This script can be called in a separate process, e.g. from cron.daily to build a
+set of backups.
 
 
-#Failover Instructions 
+#Failover Instructions
 * Choose from options 1) 2) 3)
 
-## To start/stop the auto failover service 
+## To start/stop the auto failover service
 This will continuously monitor the production
 instance and engage the failover automatically during prolonged outages
 
@@ -105,24 +105,24 @@ chkconfig --list | grep gocdb-auto
 /sbin/service gocdb-autofailover status
 
 ```
-  
-Directly (not as a service): 
+
+Directly (not as a service):
 
 ```bash
 cd /root/autoEngageFailover
 ./gocdb-autofailover.sh {start|stop|restart}
 
 ```
 
-## To manually engage the failover immediately 
+## To manually engage the failover immediately
 E.g. for known/scheduled outages, run the following
 passing 'now' as the first command-line argument:
 
-Stop the service: 
+Stop the service:
 ```
 service gocdb-autofailover stop
 ```
-Or to stop if running manually: 
+Or to stop if running manually:
 ```
 cd /root/autoEngageFailover
 ./gocdb-autofailover.sh stop
@@ -136,16 +136,16 @@ Engage the failover now:
 You will need to manually revert the steps executed by the
 failover so the dns points back to the production instance
 and restore/restart the failover process. This includes:   
-* <strike>restore the symlinks to the goc.dl.ac.uk server cert and key
-  (see details below)</strike> (no longer needed as cert contains dual SAN) 
+* <strike>restore the symlinks to the gocdb.hartree.stfc.ac.uk server cert and key
+  (see details below)</strike> (no longer needed as cert contains dual SAN)
 * restore the hourly cron to fetch the dmp of the DB
 * run nsupdate procedure to repoint 'goc.egi.eu' back to
   'gocdb-base.esc.rl.ac.uk'
-  MUST read /root/nsupdate_goc/nsupdateReadme.txt. 
+  MUST read /root/nsupdate_goc/nsupdateReadme.txt.
 * restart the failover service
 
 ####Restore Walkthrough
-At end of downtime (production instance ready to be restored) first re-point DNS: 
+At end of downtime (production instance ready to be restored) first re-point DNS:
 
 ```bash
 echo We first switch dns to point to production instance
@@ -154,7 +154,7 @@ cd /root/nsupdate_goc
 
 ```
 
-Now wait for DNS to settle, this takes approx **2hrs** and during this time the goc.egi.eu domain will 
+Now wait for DNS to settle, this takes approx **2hrs** and during this time the goc.egi.eu domain will
 swtich between the failover instance and the production instance. You should monitor this using nsupdate:  
 
 ```bash
@@ -167,7 +167,7 @@ nslookup goc.egi.eu
 	Address: 130.246.143.160
 ```
 
-After DNS has become stable the production instance will now be serving requests. 
+After DNS has become stable the production instance will now be serving requests.
 Only after this ~2hr period should we re-start failover service:
 
 ```bash
@@ -177,14 +177,14 @@ rm /root/autoEngageFailover/engage.lock
 mv cronRunDbUpdate.sh /etc/cron.hourly   
 
 # Below server cert change no longer needed as cert contains dual SAN
-# This means a server restart is no longer needed. 
-#echo Change server certificate and key back for goc.dl.ac.uk
-#ln -sf /etc/pki/tls/private/goc.dl.ac.uk.key.pem /etc/pki/tls/private/hostkey.pem
-#ln -sf /etc/grid-security/goc.dl.ac.uk.cert.pem /etc/grid-security/hostcert.pem
+# This means a server restart is no longer needed.
+#echo Change server certificate and key back for gocdb.hartree.stfc.ac.uk
+#ln -sf /etc/pki/tls/private/gocdb.hartree.stfc.ac.uk.key.pem /etc/pki/tls/private/hostkey.pem
+#ln -sf /etc/grid-security/gocdb.hartree.stfc.ac.uk.cert.pem /etc/grid-security/hostcert.pem
 #service httpd restart
 #service gocdb-autofailover start
 #service gocdb-autofailover status
-#  gocdb-autofailover is running... 
+#  gocdb-autofailover is running...
 ```
 
 Now check the '/root/autoEngageFailover/pingCheckLog.txt' and