Is your feature request related to a problem? Please describe.
Currently, there is no exposed way to control the max size of batches sent by the ES output plugin. In cases where batches grow large (in very busy environments), this can cause issues when e.g. an nginx proxy in front of the ES endpoints is configured with a (sensible) maximum request body size.
The ES plugin itself does expose bulk_message_request_threshold for controlling this, however this does not appear to be exposed by the fluent operator's clusteroutput.fluentd resource.
Describe the solution you'd like
The fluent operator's fluentd (and possibly fluentbit) [cluster]output CRDs to expose controls over more params to (at least) the ES output plugin, to avoid fluentd choking on log batches which become too large to ever be processed by the receiver
Additional context
No response
Is your feature request related to a problem? Please describe.
Currently, there is no exposed way to control the max size of batches sent by the ES output plugin. In cases where batches grow large (in very busy environments), this can cause issues when e.g. an nginx proxy in front of the ES endpoints is configured with a (sensible) maximum request body size.
The ES plugin itself does expose bulk_message_request_threshold for controlling this, however this does not appear to be exposed by the fluent operator's clusteroutput.fluentd resource.
Describe the solution you'd like
The fluent operator's fluentd (and possibly fluentbit) [cluster]output CRDs to expose controls over more params to (at least) the ES output plugin, to avoid fluentd choking on log batches which become too large to ever be processed by the receiver
Additional context
No response