Add MariaDB dumpfile fetch and import.

With optional .zip file inflation.

.flake8

@@ -0,0 +1,3 @@
+[flake8]
+# E501: Ignore line length limit of 79
+ignore = E501

README.md

@@ -7,14 +7,16 @@ This repo contains the service and cron scripts used to run a failover gocdb ins
 * autoEngageFailover/
   * Contians a Service script (```gocdb-autofailover.sh```) and child scripts that monitors the main production instance. If a prolonged outage is detected, the GOCDB top DNS alias 'goc.egi.eu' is swtiched from the production instance to the failover instance. This switch can also be performed manually when needed.
 * importDBdmpFile/
-  * Contains a script that should be invoked by cron hourly (```1_runDbUpdate.sh```) to fetch and install a .dmp of the production DB into the local failover DB. This runs separtely from the autoEngageFailover process.
+  * ORACLE only: Contains a script that should be invoked by cron hourly (```1_runDbUpdate.sh```) to fetch and install a .dmp of the production DB into the local failover DB. This runs separtely from the autoEngageFailover process.
+* importMariaDBdmpFile/
+  * MariaDB only: Contains the ```runDBUpdate.py``` script and configuration files for fetching a remote database dump as generated by the ```mysqldump``` utility and loading the dump into the failover DB. Optionally, if the dump file can be provided as a ```.zip``` file archive.
 * nsupdate_goc/
   * Scripts for switching the DNS to/from the production/failover instance.
 * archiveDmpDownload/
   * Contains a script to download/archive dmp files in a separate process
 
 # Packages
-* The following scripts needs to be installed and configuired for your installation:
+* The following scripts need to be installed and configured for your installation:
 ```
 /root/
   autoEngageFailover/         # Scripts to mon the production instance and engage failover
@@ -30,6 +32,11 @@ This repo contains the service and cron scripts used to run a failover gocdb ins
       |_ gatherStats.sh       #   Oracle gathers stats to re-index
       |_ pass_file_exemplar.txt   #   Sample pwd file for DB (rename to pass_file)
 
+  importMariaDBdmpFile/
+      |_ runDbUpdate.py       #   Main script
+      |_ config.ini_template  #   Configuration parameters
+      |_ dbImport.pwd_template#   DB password file template
+
   nsupdate_goc/              # Scripts for switching the DNS to the failover
       |_ goc_failover.sh     #   Points DNS to failover instance
       |_ goc_production.sh   #   Points DNS to production instance
@@ -55,13 +62,13 @@ following:
 * <strike>symbolic links to the server cert/key are updated so they
   point to the 'goc.egi.eu' cert/key</strike> (note, no longer needed as cert contains dual SAN)
 * the dnscripts are invoked to change the dns (see
-  nsupdate_goc below).  
+  nsupdate_goc below).
 
 ## /root/importDBdmpFile/
 Contains scripts that fetches the .dmp file and install this
 dmp file into the local Oracle XE instance. The master script
 is '1_runDbUpdate.sh' which needs to be invoked from an hourly
-cron:   
+cron:
 
 ```
 # more /etc/cron.hourly/cronRunDbUpdate.sh
@@ -76,6 +83,10 @@ key is present on the host with the database dmp file.
 * populate `importDBdmpFile/failover_TEMPLATE.sh` with
 appropriate values and copy it to `/etc/gocdb/failover.sh`
 
+## /root/importMariaDBdmpFile/
+Contains the ```runDBUpdate.py``` script and configuration files for fetching a remote database dump as generated by the ```mysqldump``` utility and loading the dump into the failover DB. Optionally, the dump file can be wrapped as a ```.zip``` file archive.
+Run as ```runDBUpdate.py --config <config file path>``` . With no ```--config``` option specified, the default is ```./importMariaDBdmpFile/config.ini```
+
 ## /root/nsupdate_goc/
 Contains the nsupdate keys and nsupdate scripts for switching
 the 'goc.egi.eu' top level DNS alias to point to either the
@@ -127,15 +138,15 @@ Or to stop if running manually:
 cd /root/autoEngageFailover
 ./gocdb-autofailover.sh stop
 ```
-Engage the failover now:                                 
+Engage the failover now:
 ```
 ./engageFailover.sh now
 ```
 
 ## Restore failover service after failover was engaged
 You will need to manually revert the steps executed by the
 failover so the dns points back to the production instance
-and restore/restart the failover process. This includes:   
+and restore/restart the failover process. This includes:
 * <strike>restore the symlinks to the gocdb.hartree.stfc.ac.uk server cert and key
   (see details below)</strike> (no longer needed as cert contains dual SAN)
 * restore the hourly cron to fetch the dmp of the DB
@@ -155,7 +166,7 @@ cd /root/nsupdate_goc
 ```
 
 Now wait for DNS to settle, this takes approx **2hrs** and during this time the goc.egi.eu domain will
-swtich between the failover instance and the production instance. You should monitor this using nsupdate:  
+swtich between the failover instance and the production instance. You should monitor this using nsupdate:
 
 ```bash
 nslookup goc.egi.eu
@@ -174,7 +185,7 @@ Only after this ~2hr period should we re-start failover service:
 echo First go check production instance and confirm it is up
 echo running ok and that dns is stable
 rm /root/autoEngageFailover/engage.lock
-mv cronRunDbUpdate.sh /etc/cron.hourly   
+mv cronRunDbUpdate.sh /etc/cron.hourly
 
 # Below server cert change no longer needed as cert contains dual SAN
 # This means a server restart is no longer needed.

importMariaDBdmpFile/config.ini_template

@@ -0,0 +1,46 @@
+[remote]
+# Hostname of remote host from which to fetch the dump file
+host=somehost.somedomain.uk
+# Remote user. Note ssh access must be without password i.e. by key
+user=someuser
+# Path on the remote host from which to fetch the dump file
+# If the suffix .zip is used the archive will be inflated and must
+# contain only a single .sql dump
+path=/path/to/db/dump
+
+[local]
+# A local file which, if it exists, causes the script to parse its configuration
+# and then exit. Return status is zero
+noImport=/etc/gocdb/nofailoverimport
+# mysql retry count. The mysql command will be tried up to this many times
+# before failure is reported. (10 = ~1 minute)
+retryCount=10
+# Local 'working' directory to stage and/or inflate the dump file into.
+workDir=/tmp
+# Path to a file containing default options in CNF format to pass to the
+# mysql client program via the --defaults-extra-file= command line option.
+# See mysqlOptions.cnf_template file.
+# The permissions (rwx) mask for 'Others' must be zero to protect the
+# database password.
+mysqlOptions=/path/to/password/file
+# Local directory into which successfully installed dumps are archived
+# a _goc5dump
+archiveDir=/path/to/archive/directory
+# Time format string to generate archived filename
+format=_goc5dump_%%y-%%m-%%d_%%H_%%M_%%S
+
+[logs]
+# Where logged output should go
+file=/path/to/logfile.txt
+# Log level:
+# 'ERROR' - log file only be updated if there is an error
+# 'INFO' - log file always updated with completion message or error
+# 'DEBUG' - log file will contain a step=by-step summary
+level=INFO
+# Formatting string for the logger module.
+format=%%(asctime)s %%(levelname)s:%%(message)s
+# Date format (see format asctime) to the output log:
+# Syslog
+# dateFormat=%%b %%d %%I:%%M:%%S
+# ISO8601
+dateFormat=%%Y-%%m-%%dT%%H:%%M:%%S%%z

importMariaDBdmpFile/mysqlOptions.cnf_template

@@ -0,0 +1,6 @@
+# See https://mariadb.com/kb/en/mysql-command-line-client/
+# Any valid mysql default value can be provided, but the
+# minimum options required are user and password.
+[client]
+user=usuallyroot
+password=myBadPassword

importMariaDBdmpFile/runDBUpdate.py

@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+'''
+Retrieve a mysqldump file from a given location on a remote host and use mysql
+to import the database locally. The remote file can be compressed as a .zip
+archive, in which case it is inflated. After successful load, the dump file
+is archived.
+'''
+import argparse
+import configparser
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+from time import gmtime, strftime, sleep
+import zipfile
+
+
+class Conf:
+    '''
+    Make accessible the parameters, configuration and set up logging
+    '''
+    def __init__(self, path):
+
+        config = configparser.ConfigParser()
+
+        config.read(path)
+
+        self.remoteHost = config.get('remote', 'host')
+        self.remoteUser = config.get('remote', 'user')
+        self.remotePath = config.get('remote', 'path')
+
+        self.workDir = config.get('local', 'workDir')
+        self.archiveDir = config.get('local', 'archiveDir')
+        self.mysqlOptionsPath = config.get('local', 'mysqloptions')
+        self.format = config.get('local', 'format')
+        self.noImport = config.get('local', 'noImport')
+        self.retryCount = config.getint('local', 'retryCount')
+
+        logging.basicConfig(filename=config.get('logs', 'file'),
+                            format=config.get('logs', 'format'),
+                            datefmt=config.get('logs', 'dateFormat'),
+                            level=config.get('logs', 'level').upper())
+
+        self.checkPerms(self.mysqlOptionsPath)
+
+    def checkPerms(self, path):
+
+        if not os.path.exists(path):
+            raise Exception('mysql import options/password file: ' + path + ' does not exist. Import terminated.')
+
+        # The password file must not be world-readable
+
+        if (os.stat(path).st_mode & (os.R_OK | os.W_OK | os.X_OK) != 0):
+            raise Exception('Open permissions found on database password file. Import terminated.')
+
+
+def getConfig():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-c', '--config', default='./importMariaDBdmpFile/config.ini', )
+
+    args = parser.parse_args()
+
+    if os.path.exists(args.config):
+        return args
+
+    raise Exception('Configuration file ' + args.config + ' does not exist')
+
+
+def runCommand(args):
+    '''
+    Run the given argument array as a command
+    '''
+
+    logging.debug('running command:' + ' '.join(args))
+
+    try:
+        subprocess.check_output(args, stderr=subprocess.STDOUT, shell=False)
+
+    except subprocess.CalledProcessError as p:
+        logging.error('command failed: ' + ' '.join(args))
+        raise Exception(p.output.rstrip())
+
+
+def getDump(remoteUser, remoteHost, remotePath, localDir):
+    '''
+    Fetch the file from the remote and save it locally
+    '''
+
+    logging.debug('fetching remote file ... ')
+
+    # there is clear text personal data in the dump so try to make sure the permissions are appropriate
+    os.umask(0o077)
+
+    # scp will replace the local file contents if it already exists
+    # We do not run with '-q' to allow meaningful authentication error messages to be logged.
+    args = ['/usr/bin/scp',
+            '-o', 'PasswordAuthentication=no',
+            remoteUser + '@' + remoteHost + ':' + remotePath,
+            localDir
+            ]
+
+    runCommand(args)
+
+    logging.debug('remote fetch completed')
+
+    localPath = localDir + '/' + os.path.basename(remotePath)
+
+    suffix = os.path.splitext(localPath)[1]
+
+    if suffix == '.zip':
+        logging.debug('inflating fetched zip archive ...')
+        zip = zipfile.ZipFile(localPath, 'r')
+        zipContents = zip.namelist()
+        if len(zipContents) != 1:
+            raise Exception('Error: .zip archive must contain only 1 file.')
+        logging.debug('inflating to ' + localDir)
+        zip.extractall(localDir)
+        localPath = localDir + '/' + zip.namelist()[0]
+        logging.debug('inflated ' + localPath)
+        zip.close
+        logging.debug('zip archive inflation completed')
+
+    return localPath
+
+
+def importDB(optionsPath, importPath, retryCount):
+    '''
+    Use mysql to import the file
+    '''
+    # Tested using dump generated using the command
+    # > mysqldump --databases --lock-tables --dump-date \
+    #             --add-locks -p gocdb -r /tmp/dbdump.sql
+
+    args = ['/usr/bin/mysql',
+            '--defaults-extra-file=' + optionsPath,
+            '-e SOURCE ' + importPath
+            ]
+
+    count = 0
+
+    while count < retryCount:
+        count += 1
+        try:
+            logging.debug('loading database from dump file ...')
+            runCommand(args)
+            break
+        except Exception as exc:
+            logging.debug('mysql command import failed.')
+            if count < retryCount:
+                logging.error(str(sys.exc_info()[1]) + '. Retrying.')
+                snooze = min(0.1 * (pow(2, count) - 1), 20)
+                logging.debug('sleeping (' + '{:.2f}'.format(snooze) + 'secs)')
+                sleep(snooze)
+                logging.debug('retry ' + str(count) + ' of ' + str(retryCount))
+                # insert backoff time wait here
+                pass
+            else:
+                logging.debug('exceeded retry count for database load failures')
+                raise exc
+
+    logging.debug('database load completed')
+
+
+def archiveDump(importPath, archive, format):
+    '''
+    Save the dump file in the archive directory
+    '''
+
+    logging.debug('archiving dump file ...')
+
+    if not os.path.isdir(archive):
+        raise Exception('Archive directory ' + archive + ' does not exist.')
+
+    archivePath = archive + '/' + strftime(format, gmtime())
+
+    logging.debug('moving ' + importPath + ' to ' + archivePath)
+
+    shutil.move(importPath, archivePath)
+
+    logging.debug('removing all .dmp files in ' + archive)
+
+    for oldDump in glob.iglob(archive + '/*.dmp'):
+        os.remove(oldDump)
+
+    shutil.move(archivePath, archivePath + '.dmp')
+
+    logging.debug('moving ' + archivePath + ' to ' + archivePath + '.dmp')
+
+    logging.debug('archive completed')
+
+
+def main():
+
+    try:
+        args = getConfig()
+
+        cnf = Conf(args.config)
+
+        if os.path.isfile(cnf.noImport):
+            logging.error(cnf.noImport + ' exists. No import attempted. File contents -')
+            with open(cnf.noImport) as f:
+                logging.error(f.read().rstrip())
+            return 1
+
+        dump = getDump(cnf.remoteUser, cnf.remoteHost, cnf.remotePath, cnf.workDir)
+
+        importDB(cnf.mysqlOptionsPath, dump, max(1, cnf.retryCount))
+
+        archiveDump(dump, cnf.archiveDir, cnf.format)
+
+        logging.info('completed ok')
+        return 0
+
+    except Exception:
+        logging.error(sys.exc_info()[1])
+        return 1
+
+if __name__ == '__main__':
+    sys.exit(main())