httpeventcollector doesn't seem to work when hitting 8081[2,3] (index slaves)

[andycmaj]

turned off ssl and tried to submit an http collector event via curl from inside a cluster-slave docker container:

$ curl -vS http://127.0.0.1:8088/services/collector -H "Authorization: Splunk EF211A51-D6AC-4045-8CD6-F730939AC518" -d '{"event": "hello world"}'
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8088 (#0)
> POST /services/collector HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 127.0.0.1:8088
> Accept: */*
> Authorization: Splunk EF211A51-D6AC-4045-8CD6-F730939AC518
> Content-Length: 24
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 24 out of 24 bytes
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

$ curl -kvS https://127.0.0.1:8088/services/collector -H "Authorization: Splunk EF211A51-D6AC-4045-8CD6-F730939AC518" -d '{"event": "hello world"}'
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8088 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* Server certificate:
*    subject: CN=SplunkServerDefaultCert; O=SplunkUser
*    start date: 2016-10-11 01:54:32 GMT
*    expire date: 2019-10-11 01:54:32 GMT
*    issuer: C=US; ST=CA; L=San Francisco; O=Splunk; CN=SplunkCommonCA; [email protected]
*    SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> POST /services/collector HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 127.0.0.1:8088
> Accept: */*
> Authorization: Splunk EF211A51-D6AC-4045-8CD6-F730939AC518
> Content-Length: 24
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 24 out of 24 bytes
< HTTP/1.1 403 Forbidden
< Date: Tue, 11 Oct 2016 02:02:52 GMT
< Content-Type: application/json; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 33
< Vary: Authorization
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
* Server Splunkd is not blacklisted
< Server: Splunkd
<
* Connection #0 to host 127.0.0.1 left intact
{"text":"Invalid token","code":4}

my cluster master config changes:

diff --git a/examples/docker/cluster-master.env_file b/examples/docker/cluster-master.env_file
index 17a7600..1017cfd 100644
--- a/examples/docker/cluster-master.env_file
+++ b/examples/docker/cluster-master.env_file
@@ -3,12 +3,6 @@ SPLUNK_BEFORE_START_CMD_1=version $SPLUNK_START_ARGS
 SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --configure
 SPLUNK_ADD_1=role service_discovery_role -auth admin:changeme
 SPLUNK_ADD_2=user service_discovery_user -password service_discovery_password -role service_discovery_role -auth admin:changeme
-SPLUNK_CMD_1=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-01:8089' '(shc_member|shc_captain)'
-SPLUNK_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-02:8089' '(shc_member|shc_captain)'
-SPLUNK_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-03:8089' '(shc_member|shc_captain)'
-SPLUNK_CMD_4=add search-server shc-member-01:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme
-SPLUNK_CMD_5=add search-server shc-member-02:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme
-SPLUNK_CMD_6=add search-server shc-member-03:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme
 SPLUNK_CMD_7=status
 SPLUNK_HEALTHCHECK_SPLUNKD=https://127.0.0.1:8089
 SPLUNK_HEALTHCHECK_SPLUNKWEB=http://127.0.0.1:8000
@@ -33,9 +27,6 @@ CONF__server__clustering__replication_factor=3
 CONF__server__clustering__search_factor=2
 CONF__server__deployment__pass4SymmKey=deployment-changeme
 CONF__server__indexer_discovery__pass4SymmKey=indexdiscovery-changeme
-CONF__server__shclustering__id=776D2949-B2DA-405E-96B3-B6688C87AB7D
-CONF__server__shclustering__pass4SymmKey=shclustering-changeme
-CONF__server__shclustering__shcluster_label=shcluster
 CONF__serverclass__global__crossServerChecksum=true
 CONF__serverclass__serverClass:data-collector-hec:app:splunk_httpinput__restartIfNeeded=true
 CONF__serverclass__serverClass:data-collector-hec__whitelist.0=data-collector-hec
@@ -51,11 +42,10 @@ CONF__etc/deployment-apps/splunk_httpinput__inputs__http__dedicatedIoThreads=2
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__maxThreads=0
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__allowSslRenegotiation=true
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__disabled=false
-CONF__etc/deployment-apps/splunk_httpinput__inputs__http__useDeploymentServer=false
+CONF__etc/deployment-apps/splunk_httpinput__inputs__http__useDeploymentServer=true
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__port=8088
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__allowSslCompression=true
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__sslVersions=*,-ssl2
-CONF__etc/deployment-apps/splunk_httpinput__inputs__http__enableSSL=true
+CONF__etc/deployment-apps/splunk_httpinput__inputs__http__enableSSL=false
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http__maxSockets=0
 CONF__etc/deployment-apps/splunk_httpinput__inputs__http://default__token=EF211A51-D6AC-4045-8CD6-F730939AC518
-CONF__etc/deployment-apps/splunk_httpinput__inputs__http://default__disabled=False

by the way, this change:

-CONF__etc/deployment-apps/splunk_httpinput__inputs__http__useDeploymentServer=false
+CONF__etc/deployment-apps/splunk_httpinput__inputs__http__useDeploymentServer=true

was needed to get the httpinput config to actually propagate to the cluster slaves

[outcoldman]

@andycmaj You should not use useDeploymentServer=true in deployment-apps as this is what will be deployed on Indexers, and you don't need this behavior on indexers.

Just try to deploy everything for the first time. Go to the cluster master, to the page forwarders management and wait when all indexes will be listed in deployment clients. You can also see when they will download the bundles.

I have not tested HEC for a while, but I believe I have not changed anything which could break it.

[andycmaj]

will try

[andycmaj]

i ended up having to actually add the input and token to the slave CONF. didn't seem to propagate via deployment-apps