Comment out swap server cert and swap dns script calls

Author: davidmeredith

README.md

@@ -3,9 +3,15 @@ Author: David Meredith + JK
 
 [this ascii file is coded in "markdown" and is best viewed in a markdown enabled browser, see https://en.wikipedia.org/wiki/Markdown for more details]
 
-This repo contains the service and cron scripts used to run a failover gocdb instance, includes:  
-* A Cron script (```1_runDbUpdate.sh```) to fetch and install a .dmp of the production DB into the failover DB. This runs separtely from the autoEngageFailover process. 
-* A Service script (```gocdb-autofailover.sh```) that monitors the main production instance. If a prolonged outage is detected, the GOCDB top DNS alias 'goc.egi.eu' is swtiched from the production instance to the failover instance. This switch can also be performed manually when needed. 
+This repo contains the service and cron scripts used to run a failover gocdb instance, includes the following dirs:
+* autoEngageFailover/
+  * Contians a Service script (```gocdb-autofailover.sh```) and child scripts that monitors the main production instance. If a prolonged outage is detected, the GOCDB top DNS alias 'goc.egi.eu' is swtiched from the production instance to the failover instance. This switch can also be performed manually when needed. 
+* importDBdmpFile/
+  * Contains a sript that should be invoked by cron hourly (```1_runDbUpdate.sh```) to download and install a .dmp of the production DB into the local failover DB. This runs separtely from the autoEngageFailover process. 
+* nsupdate_goc/
+  * Scripts for switching the DNS to/from the production/failover instance. 
+* archiveDmpDownload/
+  * Contains a script to download/archive dmp files in a separate process 
 
 # Packages
 * The following scripts needs to be installed and configuired for your installation: 
@@ -15,8 +21,8 @@ This repo contains the service and cron scripts used to run a failover gocdb ins
       |_ gocdb-autofailover.sh# MAIN SERVICE SCRIPT to mon production instance
       |_ engageFailover.sh    #   Child script, run if prolonged outage is detected
       
-  importDBdmpFile/            # Cron scripts download/install a .dmp of the prod data
-      |_ 1_runDbUpdate.sh     # MAIN CRON SCRIPT, invokes scripts below 
+  importDBdmpFile/            # Scripts download/install a .dmp of the prod data
+      |_ 1_runDbUpdate.sh     # MAIN SCRIPT that can be called from cron, invokes child scripts below 
       |_ ora11gEnvVars.sh     #   Setup oracle env
       |_ getDump.sh           #   Download a .dmp of the production data 
       |_ dropGocdbUser.sh     #   Drops the current DB schema
@@ -28,6 +34,10 @@ This repo contains the service and cron scripts used to run a failover gocdb ins
   nsupdate_goc/              # Scripts for switching the DNS to the failover
       |_ goc_failover.sh     #   Points DNS to failover instance
       |_ goc_production.sh   #   Points DNS to production instance 
+
+  archiveDmpDownload/        # Contains script to download/archive dmp files in a separate process e.g from cron.daily
+      |_ archiveDump.sh      # Main script that dowloads dmp and saves in a sub-dir 
+      |_ archive/            # Contains archive/dmp files 
 ```
 
 ## /root/autoEngageFailover/ 
@@ -43,8 +53,8 @@ following:
   checked to see that it is current, 
 * the hourly cron that downloads the dmp file is stopped (see
   importDBdmpFile below), 
-* symbolic links to the server cert/key are updated so they
-  point to the 'goc.egi.eu' cert/key 
+* <strike>symbolic links to the server cert/key are updated so they
+  point to the 'goc.egi.eu' cert/key</strike> (note, no longer needed as cert contains dual SAN) 
 * the dnscripts are invoked to change the dns (see
   nsupdate_goc below).  
 
@@ -72,6 +82,13 @@ the 'goc.egi.eu' top level DNS alias to point to either the
 production instance or the failover. 
 
 
+## /root/archiveDmpDownload/
+Contains a script that downloads the dmp file and stores the file in the archive/ sub-dir.
+The script also deletes archived files that are older than 'x' days. 
+This script can be called in a separate process, e.g. from cron.daily to build a 
+set of backups. 
+
+
 #Failover Instructions 
 * Choose from options 1) 2) 3)
 
@@ -81,17 +98,20 @@ instance and engage the failover automatically during prolonged outages
 
 Run as a service:
 
-```
+```bash
 chkconfig --list | grep gocdb-auto
 /sbin/service gocdb-autofailover stop
 /sbin/service gocdb-autofailover start
 /sbin/service gocdb-autofailover status
+
 ```
 
 Directly (not as a service): 
-```
+
+```bash
 cd /root/autoEngageFailover
 ./gocdb-autofailover.sh {start|stop|restart}
+
 ```
 
 ## To manually engage the failover immediately 
@@ -116,8 +136,8 @@ Engage the failover now:
 You will need to manually revert the steps executed by the
 failover so the dns points back to the production instance
 and restore/restart the failover process. This includes:   
-* restore the symlinks to the goc.dl.ac.uk server cert and key
-  (see details below) 
+* <strike>restore the symlinks to the goc.dl.ac.uk server cert and key
+  (see details below)</strike> (no longer needed as cert contains dual SAN) 
 * restore the hourly cron to download the dmp of the DB
 * run nsupdate procedure to repoint 'goc.egi.eu' back to
   'gocdb-base.esc.rl.ac.uk'
@@ -126,16 +146,18 @@ and restore/restart the failover process. This includes:
 
 ####Restore Walkthrough
 At end of downtime (production instance ready to be restored) first re-point DNS: 
-```
+
+```bash
 echo We first switch dns to point to production instance
 cd /root/nsupdate_goc
 ./goc_production.sh
+
 ```
 
 Now wait for DNS to settle, this takes approx **2hrs** and during this time the goc.egi.eu domain will 
 swtich between the failover instance and the production instance. You should monitor this using nsupdate:  
 
-```
+```bash
 nslookup goc.egi.eu
 # check this returns the following output referring to
 # next.gocdb.eu
@@ -148,21 +170,21 @@ nslookup goc.egi.eu
 After DNS has become stable the production instance will now be serving requests. 
 Only after this ~2hr period should we re-start failover service:
 
-```
+```bash
 echo First go check production instance and confirm it is up
 echo running ok and that dns is stable
 rm /root/autoEngageFailover/engage.lock
 mv cronRunDbUpdate.sh /etc/cron.hourly   
 
-echo Change server certificate and key back for goc.dl.ac.uk
-ln -sf /etc/pki/tls/private/goc.dl.ac.uk.key.pem /etc/pki/tls/private/hostkey.pem
-
-ln -sf /etc/grid-security/goc.dl.ac.uk.cert.pem /etc/grid-security/hostcert.pem
-
-service httpd restart
-service gocdb-autofailover start
-service gocdb-autofailover status
-  gocdb-autofailover is running... 
+# Below server cert change no longer needed as cert contains dual SAN
+# This means a server restart is no longer needed. 
+#echo Change server certificate and key back for goc.dl.ac.uk
+#ln -sf /etc/pki/tls/private/goc.dl.ac.uk.key.pem /etc/pki/tls/private/hostkey.pem
+#ln -sf /etc/grid-security/goc.dl.ac.uk.cert.pem /etc/grid-security/hostcert.pem
+#service httpd restart
+#service gocdb-autofailover start
+#service gocdb-autofailover status
+#  gocdb-autofailover is running... 
 ```
 
 Now check the '/root/autoEngageFailover/pingCheckLog.txt' and

autoEngageFailover/engageFailover.sh

@@ -22,6 +22,9 @@ updateLog=/root/autoEngageFailover/pingCheckLog.txt
 errorEngageFailoverLog=/root/autoEngageFailover/errorEngageFailoverLog.txt
 lockFile=/root/autoEngageFailover/engage.lock
 
+# Dir containing the import DB scripts and log file
+importDBdmpFile=/root/importDBdmpFile
+
 # maintainthe current fail count
 failcount=0
 
@@ -173,7 +176,8 @@ email "Detected successive failures. Attempting to engage the failover - please
 
 
 # - Test that the last goc.dmp imported ok by parsing /root/importDBdmpFile/updateLog.txt 
-cd /root/importDBdmpFile
+#cd /root/importDBdmpFile
+cd $importDBdmpFile
 if [ "$(tail -1 ./updateLog.txt)" != "completed ok" ]; then
   errorLogger "Last import of dmp file did not complete ok, exiting auto-failover early before cert and dns switch "
   exit 1
@@ -186,54 +190,38 @@ mv /etc/cron.hourly/cronRunDbUpdate.sh /root
 
 errorLogger "Swapping server certs"
 
-# - Swap server cert 
-unlink /etc/grid-security/hostcert.pem 
-unlink /etc/pki/tls/private/hostkey.pem
-ln -s /etc/grid-security/goc.egi.eu.cert.pem /etc/grid-security/hostcert.pem
-ln -s /etc/pki/tls/private/goc.egi.eu.key.pem /etc/pki/tls/private/hostkey.pem
-# note, after the main instance has been restored, you will need to revert this swap: 
-# ln -s /etc/pki/tls/private/goc.dl.ac.uk.key.pem /etc/pki/tls/private/hostkey.pem
-# ln -s /etc/grid-security/goc.dl.ac.uk.cert.pem /etc/grid-security/hostcert.pem
+## - Swap server cert 
+## Not needed e.g. if your server cert has a dual SAN
+#unlink /etc/grid-security/hostcert.pem 
+#unlink /etc/pki/tls/private/hostkey.pem
+#ln -s /etc/grid-security/goc.egi.eu.cert.pem /etc/grid-security/hostcert.pem
+#ln -s /etc/pki/tls/private/goc.egi.eu.key.pem /etc/pki/tls/private/hostkey.pem
+## note, after the main instance has been restored, you will need to revert this swap: 
+## ln -s /etc/pki/tls/private/goc.dl.ac.uk.key.pem /etc/pki/tls/private/hostkey.pem
+## ln -s /etc/grid-security/goc.dl.ac.uk.cert.pem /etc/grid-security/hostcert.pem
 
-errorLogger "After server cert swap"
-cd /root/nsupdate_goc
-
-# Old nsupdate scripts when domain was hosted at nikhef
-# Run 1st nsupdate script to delete the goc.egi.eu domain  
-#errorLogger "Running 1st nsupdate script"
-#deleteGocDomainOut=$(./nsupdate-goc.sh ./nsupdateLiveGocFiles/delete_goc.egi.eu 2>&1)
-#deleteGocDomainCode=$?
-#errorLogger "$deleteGocDomainOut"
-#if [ $deleteGocDomainCode != 0 ]; then
-#  errorLogger "Deleting GOC Domain failed: $deleteGocDomainCode"
-#  exit $deleteGocDomainCode 
-#fi
+#errorLogger "After server cert swap"
 
-# Run 2nd nsupdate script to point goc.egi.eu to goc.dl.ac.uk 
-#errorLogger "Running 2nd nsupdate script"
-#addGocDomainOut=$(./nsupdate-goc.sh ./nsupdateLiveGocFiles/point_goc.egi.eu_To_goc.dl.ac.uk 2>&1)
-#addGocDomainCode=$?
-#errorLogger "$addGocDomainCode"
-#if [ $addGocDomainCode != 0 ]; then
-#  errorLogger "Adding GOC Domain failed: $addGocDomainCode"
-#  exit $addGocDomainCode
-#fi
-
-errorLogger "Running nsupdate script"
-nsupdateOut=$(./goc_failover.sh 2>&1)
-nsupdateCode=$?
-errorLogger "nspdateCode was: $nsupdateCode"
-errorLogger "nsupdateOut was: $nsupdateOut"
-if [ $nsupdateCode != 0 ]; then
-  errorLogger "nsupdate Failed"
-  # dont need to exit, restart apache won't hurt anyway
-  #exit $nsupdateCode
-fi
 
-
-# Restart apache 
-errorLogger "Restarting apache"
-service httpd restart 
+#errorLogger "Changing DNS"
+#cd /root/nsupdate_goc
+#
+#
+#errorLogger "Running nsupdate script"
+#nsupdateOut=$(./goc_failover.sh 2>&1)
+#nsupdateCode=$?
+#errorLogger "nspdateCode was: $nsupdateCode"
+#errorLogger "nsupdateOut was: $nsupdateOut"
+#if [ $nsupdateCode != 0 ]; then
+#  errorLogger "nsupdate Failed"
+#  # dont need to exit, restart apache won't hurt anyway
+#  #exit $nsupdateCode
+#fi
+#
+#
+## Restart apache 
+#errorLogger "Restarting apache"
+#service httpd restart 
 
 
 # Finally create the lockFile to indicate the failover ran ok 

importDBdmpFile/1_runDbUpdate.sh

@@ -54,7 +54,7 @@ fi
 
 # Drop gocdb user (gocdb5 user is recreated when doing the impdb) 
 ####################################
-dropGocdbUserOutput=`./dropGocdbUser.sh 2>&1`
+dropGocdbUserOutput=`./dropGocdbUser2.sh 2>&1`
 dropGocdbExitCode=$?
 #echo $dropGocdbUserOutput
 if [ $dropGocdbExitCode != 0 ]; then 
@@ -138,6 +138,9 @@ fi
 # Create a copy of the last successfully imported dmp file and 
 # store this in the lastImportedDmpFile dir with the time and date appended 
 # to the file name 
+if [ ! -d lastImportedDmpFile ]; then
+  mkdir lastImportedDmpFile
+fi
 
 # cd into the dir (this directory must exist) 
 cd lastImportedDmpFile

importDBdmpFile/dropGocdbUser2.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+#service oracle-xe restart
+
+# Script below will drop the gocdb5 user only if it exists
+# rather than using a simple drop in the script:  drop user gocdb5 cascade;
+#
+# We drop the user because impdb will create the user for us (most DBAs let the impdb
+# create the user - note, we need to use the imported user's pw).
+# If the user already exists when we call impdb we get an error reporting the user already
+# exists as described at:
+# http://www.dba-oracle.com/t_ora_31684_import_impdp.htm
+# Or, we can use 'exclude=user' in the impdb command as described at same link.
+#
+# http://www.oracle-base.com/articles/misc/oracle-shell-scripting.php
+#
+# Here doc: The minus sign after the << will ignore tab characters (Note: not all whitespace!)
+# at the start of a line, so you can indent your data to increase the readability
+#
+# RESULT=$(sqlplus system/XXXXXXX <<- ENDSQL
+
+# PASSFILE is in "parfile" format for impdp
+PASSFILE=/root/gocdb-failover-scripts/importDBdmpFile/pass_file
+USER_PASS=$(cat "${PASSFILE}" | grep "^userid=" | cut -d "=" -f 2)
+
+# Need to exit gracefully if pass_file doesn't exist or
+# above filter fails
+
+# /nolog runs sqlplus without connecting to anything
+# Having CONNECT in script keeps ${USER_PASS} off the commandline
+# And hence not visible to "ps"
+RESULT=$(sqlplus /nolog <<- ENDSQL
+	CONNECT ${USER_PASS};
+	create or replace directory dmpdir as '/tmp';
+	DROP USER GOCDB5 CASCADE;
+	EXIT;
+	ENDSQL
+)
+echo $RESULT
+
+# sample successfull $RESULT
+#SQL*Plus: Release 11.2.0.2.0 Production on Fri Sep 27 14:28:37 2013 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production SQL> Directory created. SQL> 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 PL/SQL procedure successfully completed. SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
+