OpenTelemetryCollector with prometheusremotewrite Exporter and basicauth Extension using Env Vars

[psychosis448]

Hello,

I'm currently encountering a bit of a weird issue, that I'm not able to resolve. Hopefully someone might spot my mistake right away :)
If this is not the correct place to ask this question, please let me know.

Components Used

• Kubernetes Cluster 1.33
• OpenTelemetry Operator via Helm Chart 0.102.0 (i.e. appVersion 0.141.0)
• Prometheus Operator CRDs for Prometheus CRD via Helm Chart 25.0.1
• opentelemetry-collector-contrib Image 0.142.0

What I'm Trying To Do

Export Prometheus metrics to an external Prometheus instance using the Opentelemetry Collecor running in Kubernetes.

OpenTelemetryCollector Configuration

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: opentelemetry-test
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
    serviceAccount: otelcol
    prometheusCR:
      enabled: true
      podMonitorSelector: {}
      serviceMonitorSelector: {}
  env:
  - name: USERNAME
    valueFrom:
      secretKeyRef:
        name: observability-credentials
        key: username
  - name: PASSWORD
    valueFrom:
      secretKeyRef:
        name: observability-credentials
        key: password
  ## base64 encoded basicauth string "username:password"
  - name: BASICAUTH
    valueFrom:
      secretKeyRef:
        name: observability-credentials
        key: basicauth
  - name: MY_POD_IP
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: status.podIP
  config:
    extensions:
      basicauth:
        client_auth:
          ## NOTE: basicauth extension only works with usernam & password set in clear text here
          username: clear_username
          password: clear_password
          ## BUG: does not work with env vars
          # username: ${env:USERNAME}
          # password: ${env:PASSWORD}
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: opentelemetry-collector
              scrape_interval: 10s
              static_configs:
                - targets:
                    - ${env:MY_POD_IP}:8888   
    exporters:
      prometheusremotewrite:
        endpoint: "prometheues-remote-write-endpoint"
        headers:
          authorization: Basic ${env:BASICAUTH}
        ## NOTE: only works, if the basicauth extension is configured using plain text credentials
        # auth:
          # authenticator: basicauth
    service:
      extensions: [basicauth]
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [prometheusremotewrite]

The Problem

The basicauth Extension does only work, if I set the username and password in clear text. If I try to reference them using env vars, the authentication fails with a 403.

On the other hand, setting the Authorization header works using the credentials from an env var.

Is this expected behaviour? Or should configuring the basicauth extension with env vars work?

Thanks in advance!

[psychosis448]

Managed to get to work using a Secret, mounting it and reading the username and password from files, as per https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/basicauthextension/README.md

extraVolumes:
  - name: observability
    secret:
      secretName: observability

extraVolumeMounts:
  - mountPath: /etc/secrets
    name: observability
    readOnly: true

config:
    basicauth/observability:
      client_auth:
        username_file: /etc/secrets/username
        password_file: /etc/secrets/password
  exporters:
    prometheusremotewrite/observability:
      endpoint: ...
      auth:
        authenticator: basicauth/observability
  service:
    extensions:
      - basicauth/observability
    pipelines:
      metrics:
        exporters: [prometheusremotewrite/observability]