(PIE-9) Add fact generation (#7)

* add fact generation and catalog application times
* Refactors module to use shared util for splunk_hec
Fixes SSL handling with simpler options
Adds terminus for facts to be sent to splunk
Enables profiling for both report and fact submission
Updates splunk_hec module to support customization of facts collected
Updates splunk_hec module to optionally manage reports setting
* Update README.md
* Adds Fact Terminus
Major changes to module were done to enable the Fact Terminus:
- util/splunk_hec.rb created for common access methods
- consistent info and error handling for both reports and facts
- performance profile support for Fact Terminus
- Documentation updated with guide and default facts listed
- Module updated to optionally manage reports setting in puppet.conf
- Module updated to add new parameters and template values
- Fact collection time added to puppet report processor
- SSL handling and documentation improved
* Hard codes puppet:summary sourcetype in event
Without this, it was dependent on the user to create the HEC properly
Now as long as the HEC token provided is correct, the reports will
go to the puppet:summary sourcetype
Example splunk_hec.yaml was updated to reflect new changes to install
* Adjusts metrics hash for consistency with splunk plugin

README.md

@@ -24,34 +24,44 @@ This was tested on both Puppet Enterprise 2018.1.4 & Puppet 6, using stock gems
 Report Processor Installation & Usage
 --------------------
 
-The steps below will help install and troubleshoot the report processor on a single Puppet Master, including manual steps to configure a puppet-server, and to use the included splunk_hec class.
+
+The steps below will help install and troubleshoot the report processor on a single Puppet Master, including manual steps to configure a puppet-server, and to use the included splunk_hec class. Because one is modifying production machines, these steps allow you to validate your settings before deploying the changes live.
 
 1. Install the Puppet Report Viewer Addon in Splunk. This will import the needed sourcetypes that make setting up the HEC easier in the next steps, and also some overview dashboards that make it a lot easier to see if you're sending Puppet run reports into Splunk.
 
-2. Create a Splunk HEC Token (preferably named `puppet:summary` and using the sourcetype `puppet:summary` from the Report Viewer addon). Follow the steps provided by Splunk's [Getting Data In Guide](http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) if you are new to HTTP Endpoint Collectors.
+2. Create a Splunk HEC Token or use an existing one that sends to main index and does not have acknowledgement enabled. Follow the steps provided by Splunk's [Getting Data In Guide](http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) if you are new to HTTP Endpoint Collectors.
 
 3. Install this Puppet module in the environment that manages your Puppet Servers are using (probably `production`)
 
 4. Run `puppet plugin download` on your Puppet Server to sync the content
 
-5. Create a `/etc/puppetlabs/puppet/splunk_hec.yaml` (see examples directory for one) adding your Splunk Server & Token from step 1
-  - You can add 'timeout' as an optional parameter, default value is 2 for both open and read sessions, so take value x2 for real world use
-  - The same is true for port, defaults to 8088 if none is provided
-  - Provide a `puppetdb\_callback\_hostname` variable if the hostname that Splunk will use to lookup further information about a report is different than the Puppet Server processing the reports (i.e., multiple servers, load balancer, external dns name vs internal, etc.) This defaults to the certname of the Puppet Server processing the report. Note that this feature has yet to be enabled in the Puppet Report Viewer.
+5. Create a `/etc/puppetlabs/puppet/splunk_hec.yaml` (see examples directory for one) adding your Splunk Server URL to the collector (usually something like `https://splunk-dev:8088/services/collector`) & Token from step 1
+  - You can add 'timeout' as an optional parameter, default value is 1 second for both open and read sessions, so take value x2 for real world use
+  - Provide a `pe_console` value that is the hostname for the Puppet Enterprise Console, which Splunk can use to lookup further information if the installation is a multimaster setup (it is best practice to set this if you're anticipating scaling out more masters in the future).
 
   ```
 ---
-"server" : "splunk-dev.testing.internal"
+"url" : "https://splunk-dev.testing.local:8088/services/collector"
 "token" : "13311780-EC29-4DD0-A796-9F0CDC56F2AD"
 ```
 
 6. Run `puppet apply -e 'notify { "hello world": }' --reports=splunk_hec` from the Puppet Server, this will load the report processor and test your configuration settings without actually modifying your Puppet Server's running configuration. If you are using the Puppet Report Viewer app in Splunk then you will see the page update with new data. If not, perform a search by the sourcetype you provided with your HEC configuration.
 
-7. Provide the working parameters / values to the splunk_hec class and use it in a profile or add it to the PE Masters subgroup of PE Infrastructure in the classification section of the console. Run Puppet on the MoM first (because it is the Puppet Server all the other compile masters are using) before running puppet on the other compile masters. This will restart the puppet-server processor, so stagger the runs to prevent an outage.
+7. If configured properly the Puppet Report Viewer app in Splunk will show 1 node in the Overview tab.
+
+8. Now it is time to roll these settings out to the fleet of to the Puppet Masters in the installation. For Puppet Enterprise users: 
+	- Navigate to Classification -> PE Infrastructure -> PE Master
+	- Select Configuration
+	- Press Refresh to ensure the splunk_hec class is loaded
+	- Add new class `splunk_hec`
+	- From the `Parameter name` select atleast `url` and `token` and provide the same attributes from the testing configuration file
+	- Optionally set `enable_reports` to `true` if there isn't another component managing the servers reports setting, otherwise manually add `splunk_hec` to the settings as described in the manual steps
+	- Commit changes and run Puppet. It is best to navigate to the PE Certificate Authority Classification gorup and run Puppet there first, before running Puppet on the remaining machines
 
 ### Manual steps:
 
 - Add `splunk_hec` to `/etc/puppetlabs/puppet/puppet.conf` reports line under the master's configuration block
+
 ```
 [master]
 node_terminus = classifier
@@ -67,14 +77,12 @@ reports = puppetdb,splunk_hec
 
 SSL Support
 -----------
-Configuring SSL support for this report processor and tasks requires that the Splunk HEC service being used has a [properly configured SSL certificate](https://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwithSSL). Once the HEC service has a valid SSL certificate, the CA will need to be made available to the report processor to load. One could add the CA to Puppet's trust, or just make the CA file available on the puppet-server (/etc/puppetlabs/puppet/splunk\_hec/splunk\_ca.cert works). Either option is supported.
+Configuring SSL support for this report processor and tasks requires that the Splunk HEC service being used has a [properly configured SSL certificate](https://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwithSSL). Once the HEC service has a valid SSL certificate, the CA will need to be made available to the report processor to load. The supported path is to install a copy of the Splunk CA to a directory called `/etc/puppetlabs/puppet/splunk_hec/` and provide the file name to `splunk_hec` class.
 
 One can update the splunk_hec.yaml file with these settings:
 
-
 ```
-"ssl_verify" : "true"
-"ssl_certificate" : "/etc/puppetlabs/puppet/splunk_hec/splunk_ca.cert"
+"ssl_ca" : "splunk_ca.cert"
 ```
 
 Or create a profile that copies the `splunk_ca.cert` as part of invoking the splunk_hec class:
@@ -97,6 +105,28 @@ class profile::splunk_hec {
 }
 ```
 
+Fact Terminus Support
+-----------
+
+The `splunk_hec` module provides a fact terminus that will send a configurable set of facts to the same HEC that the report processor is using, however as a `puppet:facts` sourcetype. This populates the Details and Inventory tabs in the Puppet Report Viewer. 
+
+- In the PE Master configuration group, add the parameter setting `facts_terminus` and set it to `splunk_hec`.
+- To configure which facts to collect (such as custom facts) add the `collect_facts` parameter in the `splunk_hec` class and modify the array of facts presented. The following facts are collected regardless to ensure the functionality of the Puppet Report Viever:
+
+```
+'os'
+'memory'
+'puppetversion'
+'system_uptime'
+'load_averages
+'ipaddress'
+'fqdn'
+'trusted'
+'producer'
+'environment'
+```
+
+
 Tasks
 -----
 

examples/splunk_hec.yaml

@@ -1,5 +1,5 @@
 # managed by puppet
 ---
-"server" : "splunk-dev.foo.bar.com"
-"token" : "9BEFBCD1-47B2-4A74-90FF-2098EEBE4EE3"
-"puppetdb_callback_hostname": "puppetdb.foo.bar.com"
+"url" : "https://splunk-dev.testing.local:8088/services/collector"
+"token" : "13311780-EC29-4DD0-A796-9F0CDC56F2AD"
+"pe_console": "puppetdb.foo.bar.com"

lib/puppet/indirector/facts/splunk_hec.rb

@@ -0,0 +1,55 @@
+require 'puppet/indirector/facts/puppetdb'
+require 'puppet/util/splunk_hec'
+
+# splunk_hec.rb
+class Puppet::Node::Facts::Splunk_hec < Puppet::Node::Facts::Puppetdb
+  desc "Save facts to Splunk over HEC and PuppetDB.
+       It uses PuppetDB to retrieve facts for catalog compilation."
+
+  include Puppet::Util::Splunk_hec
+
+  def save(request)
+    # puppetdb goes first
+    super(request)
+
+    profile('splunk_facts#save', [:splunk, :facts, :save, request.key]) do
+      begin
+        host = request.instance.name.dup
+        incoming_facts = request.instance.values.dup
+
+        hardcoded = [
+          'os',
+          'memory',
+          'puppetversion',
+          'system_uptime',
+          'load_averages',
+          'ipaddress',
+          'fqdn',
+        ]
+
+        # lets ensure user provided fact names are downcased
+        users = settings['facts'].map(&:downcase)
+
+        keep = (hardcoded + users).uniq
+
+        facts = incoming_facts.select { |k, _v| keep.include?(k) }
+
+        facts['trusted'] = get_trusted_info(request.node)
+        facts['environment'] = request.options[:environment] || request.environment.to_s
+        facts['producer'] = Puppet[:node_name_value]
+        facts['pe_console'] = pe_console
+
+        event = {
+          'host' => host,
+          'sourcetype' => 'puppet:facts',
+          'event' => facts,
+        }
+
+        Puppet.info "Submitting facts to Splunk at #{splunk_url}"
+        submit_request event
+      rescue StandardError => e
+        Puppet.err "Could not send facts to Satellite: #{e}\n#{e.backtrace}"
+      end
+    end
+  end
+end