Collector complains about "Could not dynamically add mapping for field [http.connection.id]"

[mviitane]

Bug Report

Which version of the demo you are using?

Using main latest:
$ git log
commit e3548c6 (HEAD -> main, upstream/main)
Date: Sun Dec 22 08:49:02 2024 -0500

Symptom

Collector batchprocessor and opensearch exporter complains about
Could not dynamically add mapping for field [http.connection.id]
Existing mapping for [attributes.http] must be of type object but found [text]

otel-collector  | 2024-12-23T12:40:13.035Z	warn	batchprocessor@v0.116.0/batch_processor.go:263	Sender failed	{"kind": "processor", "name": "batch", "pipeline": "logs", "error": "not retryable error: Permanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":
...
otel-collector  | 2024-12-23T12:40:13.449Z	error	internal/base_exporter.go:139	Exporting failed. Rejecting data.	{"kind": "exporter", "data_type": "logs", "name": "opensearch", "error": "not retryable error: Permanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":
...

What is the expected behavior?

No complains.

Reproduce

Could you provide the minimum required steps to resolve the issue you're seeing?

docker compose build
docker compose run
docker compose logs otel-collector

Workaround

Disable logs pipeline in the Collector.

Additional context

Related to #1833

However, this occurs with standard demo components without enabling any vendor backends.

[mviitane]

This seems to be related to the envoy access logs and #1768

[puckpuck]

Can you do a docker system prune -a, doing a new fresh build and trying again

[mviitane]

I did and got the same problem. I'm running Docker on x86_64, Ubuntu.

[TrevorSquillario]

I'm seeing this same error and it's causing the logs not to show up in opensearch. Deleted all images, volumes, ran prune. Same error.

Sender failed   
{"kind": "processor", "name": "batch", "pipeline": "logs", "error": "not retryable error: Permanent error: 
{\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing 
mapping for [attributes.http] must be of type object but found [text].\"

2025-01-26T05:50:35.594Z        warn    batchprocessor@v0.116.0/batch_processor.go:263  Sender failed   {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "not retryable error: Permanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}\nPermanent error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"Could not dynamically add mapping for field [http.connection.id]. Existing mapping for [attributes.http] must be of type object but found [text].\",\"caused_by\":{\"type\":\"\",\"reason\":\"\",\"caused_by\":null}}"}

[mviitane]

Hi @TrevorSquillario, does the fix in #1930 work for you, to disable a bunch of attributes in the Envoy access logs?

[TrevorSquillario]

Hi @TrevorSquillario, does the fix in #1930 work for you, to disable a bunch of attributes in the Envoy access logs?

Yea, that seems to fix the error messages. Am I loosing out on anything by removing those?

[mviitane]

Hi @TrevorSquillario, does the fix in #1930 work for you, to disable a bunch of attributes in the Envoy access logs?

Yea, that seems to fix the error messages. Am I loosing out on anything by removing those?

You'll lose those attributes in the Envoy access logs (but not the logs themselves). I think they play only a very small role in the Demo app overall. The Envoy access logs were only recently added.