Support for HEC acknowledgments in splunkhec exporter

[wojtekzyla]

Component(s)

exporter/splunkhec

Is your feature request related to a problem? Please describe.

Currently Splunk HEC exporter sends data to Splunk without reassuring if it was in fact indexed. It is possible that HTTP response from POST request is 200 but the data wasn't indexed due to some errors in the processing pipelines.

Describe the solution you'd like

Add support for HEC acknowledgment. It could be a configurable option, switched off by default. When switched on, exporter would keep data in the persistent queue and keep retrying to send it until it receives acknowledgement message from Splunk.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/splunkhec: @atoulme @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

This requires additional setup such as moving away from the batch processor which induces asynchronicity of the pipeline. It is also a design choice to not support acknowledgement as it has not proven to be useful.

Before you start the implementation of the feature, please draft a comprehensive integration test plan placing multiple collectors in sequence forming an acknowlegement pipeline.