Incorporate safety wait into sign_csr

When signing CSRs, it is common that the CSR will have been submitted
just prior. Previously, plans which did both would have to pause after
submitting the CSR to ensure there was time for the server to have fully
processed it before running the sign task, in order to avoid any chance
of ready-to-sign timing problems.

This commit makes the sign_csr task itself attempt to sign up to six
times, waiting one second between attempts, in order to avoid needing to
do that in the plan.

Author: reidmv

plans/action/install.pp

@@ -349,10 +349,6 @@
     # Ensure certificate requests have been submitted, then run Puppet
     unless ($target in $database_targets) {
       run_task('peadm::submit_csr', $target)
-      # TODO: come up with an intelligent way to validate that the expected CSRs
-      # have been submitted and are available for signing, prior to signing them.
-      # For now, waiting a short period of time is necessary to avoid a small race.
-      ctrl::sleep(5)
       run_task('peadm::sign_csr', $master_target, { 'certnames' => [$target.name] } )
       run_task('peadm::puppet_runonce', $target)
     }

tasks/sign_csr.rb

@@ -11,22 +11,31 @@ def csr_signed?(certname)
     File.exist?(File.join(Puppet.settings[:cadir], 'signed', "#{certname}.pem"))
 end
 
+def sign(certnames)
+  cmd = ['/opt/puppetlabs/bin/puppetserver', 'ca', 'sign',
+         '--certname', certnames.join(',')]
+
+  stdout, status = Open3.capture2(*cmd)
+  puts stdout
+  raise StandardError unless status.success?
+end
+
 def main
   Puppet.initialize_settings
   params = JSON.parse(STDIN.read)
-  unsigned = params['certnames'].reject { |name| csr_signed?(name) }
-
-  exit 0 if unsigned.empty?
 
-  cmd = ['/opt/puppetlabs/bin/puppetserver', 'ca', 'sign',
-         '--certname', unsigned.join(',')]
+  attempts = 0
 
-  stdout, status = Open3.capture2(*cmd)
-  puts stdout
-  if status.success?
-    exit 0
-  else
-    exit 1
+  begin
+    unsigned = params['certnames'].reject { |name| csr_signed?(name) }
+    exit 0 if unsigned.empty?
+    sign(unsigned.join(','))
+  rescue
+    exit 1 if attempts > 6
+    attempts += 1
+    puts "Signing attempt #{attempts} failed; waiting 1s and trying again"
+    sleep 1
+    retry
   end
 end