You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When using output.elasticsearch with batch_op_type: create (required for Elasticsearch/OpenSearch data streams), file.d reports errors even though documents are successfully created.
Error message:
unknown elasticsearch response, 'index' field in the response is empty
This happens because bulk responses contain "create": {...} instead of "index": {...}, which the plugin does not currently parse.
To Reproduce
Steps to reproduce the behavior:
Configure output.elasticsearch with batch_op_type: create.
Send logs to an Elasticsearch/OpenSearch data stream (index name starts with .ds-).
Observe file.d logs:
unknown elasticsearch response, 'index' field in the response is empty
some events from batch aren't written
Expected behavior
file.d should correctly parse bulk responses containing "create": {...} when batch_op_type: create is used, and treat them as successful writes.
Additional context
Version: file.d [v0.61.0]
Platform: Kubernetes
Elasticsearch/OpenSearch: OpenSearch [2.18]
Example response from server:
{"create":{
"_index":".ds-file-d-dev-cluster-apps-order-service-000069",
"_id":"ymfJcJkBL5nPyj0lnB8M",
"_version":1,
"result":"created",
"_shards":{"total":2,"successful":2,"failed":0},
"_seq_no":2826141,
"_primary_term":1,
"status":201
}}
Just to clarify: it’s not only about suppressing error logs.
Right now some events are actually marked as failed, so logs from search-gateway (and potentially other services) do not appear in OpenSearch when writing into data streams.
We need proper support for bulk responses with "create": {...} in output.elasticsearch (when batch_op_type: create is used).
Suppressing the error is not enough — the plugin should treat these responses as successful so that all events are indexed.
Activity