Service Graph Metrics mix K8s namespace and OTEL namespace

[mariomac]

Scenario

4 services in the demo K8s namespace, that override the OTEL service.namespace attribute via OTEL_RESOURCE_ATTRIBUTES env var or resource.opentelemetry.io/service.namespace: "ecommerce" annotations.

Service Graph metrics duplicate the services because the server_service_namespace is duplicated as demo and ecommerce:

For example:

traces_service_graph_request_total{
  client="my-backend",
  client_service_namespace="ecommerce",
  instance="kind-worker",
  server="worker-svc",
  server_service_namespace="demo",
}
traces_service_graph_request_total{
  client="my-backend",
  client_service_namespace="ecommerce",
  instance="kind-worker2",
  server="my-worker",
  server_service_namespace="ecommerce",
}

Service Graph view:

The reason is that OBI is able to correctly override the destination Pod K8s namespace by the standard OTEL env vars and annotations.

However, when the destination URL is a service (worker-svc instead of my-worker), the pod annotations and env vars are not visible from the URL.

## Possible solution

Through the Kube API, let OBI to keep track of the K8s Service spec.selector attributes and internally maintain a map of Pods that already match that attribute, then from the service name it's possible to access the Pod replicas that should share the same annotations and env vars.

[grcevski]

I thought we did this in name resolver, we peek into the destination pod environment variables and try to look for name overrides. Perhaps we don't have them resolved?

[mariomac]

@grcevski I think the issue here is that sometimes (I think is when the connection goes between two nodes), we don't see the destination Pod URL but the destination Service URL (which sometimes might have different name). In my two example metrics, we have both server="worker-svc" and server="my-worker".

Maybe it's just an edge case, so we don't need to prioritize it too much. In case, we wanted to solve it, we should also store the Services API information, so if for example, the worker-svc is defined as this:

apiVersion: v1
kind: Service
metadata:
  name: worker-svc
  namespace: demo
spec:
  selector:
    app: worker

We should look at at least one Pod labeled as app: worker, and extract the Service name and namespace from the OTEL annotations or environment variables. In my example, it would match the pods from this deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: demo
spec:
  selector:
    matchLabels:
      app: worker
  replicas: 3
  template:
    metadata:
      labels:
        instrument: beyla
        app: worker
      annotations:
        resource.opentelemetry.io/service.name: "my-worker"
        resource.opentelemetry.io/service.namespace: "ecommerce"
        resource.opentelemetry.io/deployment.environment: "staging"
    spec:
      containers:
        - name: worker
          image: mariomac/ds-example:worker
          ports:
            - containerPort: 5000
              name: grpc

But I'm maybe focusing on an edge use case and not so many people will override the K8s names/namespaces with resource.opentelemetry.io/... annotations or OTEL_RESOURCE_ATTRIBUTES env vars. Let's see if real users bring this up :)

[grcevski]

Oh I get it now, I think we should try to fix it, IMO people do various things and I think we should try to handle it.