How to send knative logs via TCP?

[pfilaretov42]

Hi, I'm looking for a way to send logs produced by knative pods to OpenSearch.

We have a standard way of doing this for our applications running in k8s:

app --[sends logs via TCP Socket Appender to]--> logging-agent --[sends to]--> Kafka --> validator/enricher --> OpenSearch

So, ideally I want to use the same mechanism for knative, with no new components installed in the k8s cluster (fluentbit, sidecars, etc).

The app logging configuration uses logback and looks like this (logging to logging-agent-svc-main:5170 using LogstashTcpSocketAppender):

<configuration>
  <appender name="TCP" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>logging-agent-svc-main:5170</destination>
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
      <providers>
        <pattern>
          <pattern>
            {
            "appName": "my-app",
            "threadName": "%thread",
            "loggerName": "%logger",
            "text": "%message",
            "level": "%level",
            "...": "..."
            }
          </pattern>
        </pattern>
        <!-- ... -->
      </providers>
    </encoder>
  </appender>
  <appender name="ASYNC_TCP" class="ch.qos.logback.classic.AsyncAppender">
    <neverBlock>true</neverBlock>
    <appender-ref ref="TCP"/>
  </appender>
  <root level="INFO">
    <appender-ref ref="ASYNC_TCP"/>
  </root>
</configuration>

And now I want the same thing for knative pods, but I cannot figure out how to configure it. I read the docs on logging and tried to update config-logging ConfigMap, i.e. adding data.zap-logger-config field:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-logging
#...
data:
  zap-logger-config: |
    {
      "level": "info",
      "development": false,
      "outputPaths": ["stdout"],
      "errorOutputPaths": ["stderr"],
      "encoding": "json",
      "encoderConfig": {
        ...
      }
    }

I looked at the zap documentation and tried to set outputPaths with one of the following:

• "outputPaths": ["logging-agent-svc-main:5170"]
• "outputPaths": ["http://logging-agent-svc-main:5170"]

However, it does not work and I can see the warning message in a knative pod (e.g., controller) during the start:

{
  "severity": "WARNING",
  "timestamp": "2025-01-31T11:53:41.106347634Z",
  "logger": "fallback-logger",
  "caller": "logging/config.go:73",
  "message": "Failed to parse logging config - using default zap production config",
  "commit": "6a27004"
}

Could you suggest the way this can be done, please?

[dprotaso]

Parsing happens here
https://github.com/knative/pkg/blob/5be486852d2713f8204aad8707e80da0004b6a5e/logging/config.go#L125

I'd honestly use fluentbit or something - I doubt anyone's done this before and I don't think we'd support this if you have problems. eg. what happens to latency in the activator when logging over the URL is slow.

[pfilaretov42]

Thanks for the reply, I'll look into fluentbit setup since looks like I have no other choice.

In regards to the following:

eg. what happens to latency in the activator when logging over the URL is slow.

Is there something like async TCP appender for zap? (something similar to the config I mentioned for logback)

[skonto]

If you dive deeper (I added some logging here):

diff --git a/vendor/go.uber.org/zap/writer.go b/vendor/go.uber.org/zap/writer.go
index 06768c679..695cdeba1 100644
--- a/vendor/go.uber.org/zap/writer.go
+++ b/vendor/go.uber.org/zap/writer.go
@@ -70,6 +70,7 @@ func open(paths []string) ([]zapcore.WriteSyncer, func(), error) {
        for _, path := range paths {
                sink, err := _sinkRegistry.newSink(path)
                if err != nil {
+                       fmt.Printf("DEBUG: %v\n", err)
                        openErr = multierr.Append(openErr, fmt.Errorf("open sink %q: %w", path, err))
                        continue
                }

the error is this:
a) "outputPaths": ["logging-agent-svc-main:5170"] causes

no sink found for scheme "logging-agent-svc-main"

b) "outputPaths": ["http://logging-agent-svc-main:5170"]

no sink found for scheme "http"

You need a custom async sink it seems as in here which you will have to register e.g. RegisterHTTPSink("http", "http://logging-agent-svc-main:5170") before you parse the configuration. However, as @dprotaso mentioned probably it is easier to handle this via parsing logs with some tooling (especially if it is already there for other purposes).

[dprotaso]

Going to close this out as there isn't anything actionable here

[pfilaretov42]

Okay, let's forget about async/TCP/etc.

How do I configure Knative with something else (fluentbit?) to collect Knative logs into OpenSearch or send them to Kafka? (Kafka - to enrich and send logs to OpenSearch)

I read the docs on logging, however it looks like a configuration for apps running on Knative - not for Knative itself. (documentation suggests to configure logging namespace with fluentbit which listents on HTTP port)

[dprotaso]

it looks like a configuration for apps running on Knative - not for Knative itself.

fluentbit captures logs for all pods - that should include the knative control plane as well

I'd honestly look at the fluentbit docs here - https://docs.fluentbit.io/manual - what we have on that page is using a real old fluenbit verison

[pfilaretov42]

Thanks for the answers!

I have a very related question, so I will post it here. Please let me know if you prefer it to be a separate issue and I'll move it.

How to send Knative logs to a file instead of stdout/stderr? Again, I tried to update config-logging ConfigMap and set outputPaths but nothing seems to work. Here is what I've tried:

"outputPaths": ["servinglog"]
"outputPaths": ["serving.log"]
"outputPaths": ["file://serving.log"]
"outputPaths": ["file:///certs/serving.log"]
"outputPaths": ["stdout", "serving.log"]
"outputPaths": ["stdout", "file://serving.log"]
"outputPaths": ["stdout", "file:///certs/serving.log"]
"outputPaths": ["serving.log", "stdout"]
"outputPaths": ["file://serving.log", "stdout"]
"outputPaths": ["file:///certs/serving.log", "stdout"]

But the logging config is not applied and I only get the following warn message in knative pod logs on the start:

{
  "severity": "WARNING",
  "timestamp": "2025-02-17T11:27:49.692552735Z",
  "logger": "fallback-logger",
  "caller": "logging/config.go:73",
  "message": "Failed to parse logging config - using default zap production config",
  "commit": "6a27004"
}

No other info about the reason is logged.

So, it would be nice

1. to know how to send Knative logs to a file instead of stdout/stderr
2. to have more info on logging config problems - could knative log it as well?

[dprotaso]

we're just parsing json into a zap.Config - you can run a small go app locally to confirm the exact error

https://github.com/knative/pkg/blob/5be486852d2713f8204aad8707e80da0004b6a5e/logging/config.go#L125

to know how to send Knative logs to a file instead of stdout/stderr

I'd say our answer is similar to TCP - we don't support this

to have more info on logging config problems - could knative log it as well?

Feel free to update the code here
https://github.com/knative/pkg/blob/5be486852d2713f8204aad8707e80da0004b6a5e/logging/config.go#L73

[pfilaretov42]

Okay, I finally found how to make knative pods log into a file. The problem was with write permissions for a knative container. I added shared volume for logging, e.g.

volumes:
  - name: shared-volume
    emptyDir: { }
containers:
  - name: controller
    volumeMounts:
      - name: shared-volume
        mountPath: /logs

and added log file in the config-logging ConfigMap:

"outputPaths": ["stdout", "/logs/serving.log"]