[Enhancement]: support for k8s Gateway TLSRoute on Kafka listeners

[lgo]

Related problem

It would be very helpful to have a new tls-route listener type which handles all of the templating for the newer Gateway TLSRoutes: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1alpha2.TLSRoute

I've recently begun to leverage strimzi with a k8s Gateway setup, which vastly simplified a previous setup with nginx (and some hacks to do port-to-broker mappings, in lieu of a TLS passthrough with SNI parsing).

Today, we have this setup with a cluster-ip listener with the CRD explicitly listing the brokers with the respective advertisedHost/advertisedPort for each broker.

    listeners:
    - configuration:
        bootstrap:
          alternativeNames:
          - bootstrap.kafka.example.com
        brokerCertChainAndKey: ...
        brokers:
        - advertisedHost: b1.kafka.example.com
          advertisedPort: 443
          broker: 1
        ...
      name: externaltls
      port: 9095
      tls: true
      type: cluster-ip

In addition, we manually create the TLSRoute entries which look like the following:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: kafka-broker-1
  namespace: kafka
spec:
  hostnames:
  - b1.kafka.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway
    namespace: ingress
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: cluster-broker-externaltls-1
      namespace: kafka
      port: 9095

This is mostly just the broker-specific resources that need creation. There are additional configurations that are general to Gateways and TLSRoutes that are necessary for things to fully work (e.g. which domains they are permitted to attach to, whether to use passthrough or TLS termination on the Gateway, etc.).

Doing the manual management comes with its pitfalls: our tooling needs to effectively track the strimzi results (which nodeIds are assigned) and go and create those resources. We've stumbled around race conditions where adding new brokers (alongside rebalances / topic-partition creation) which have resulted in producer/consumers failing due to those resources (and the listeners list of brokers) being correctly managed.

Suggested solution

Adding a tls-route listener type similar to ingress, where:

• You can get a free TLSRoute for a bootstrap server and each broker
• Allow for parameterization of broker lists or a template (like ingress)

I also believe there's benefit in providing some more generic resources with some minimal-level of templating on a Kafka CRD's resource templates. It would have allowed for us to largely create the TLSRoute without any specific implementation on strimzi's end (ignoring the listener broker ID listing). I do recognize that's a bit of an ask to do!

I see at least two benefits to more generic resource templates:

1. There doesn't need to be a strimzi implementation for TLSRoute, which is currently marked as an alpha resource and I suspect could vary between k8s deployments in-use
2. Flexibility to support other resources that may be necessary

In fact, we need (2) to effectively deploy our use-case. My current deployment uses Envoy as our k8s Gateway implementation and we require an Envoy-specific CRD to configure each TLSRoute appropriately, like so

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: kafka-broker-1  # <-- arbitrary name for resource
  namespace: kafka
spec:
  # ... misc Envoy configurations such as for connection limits
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: TLSRoute
    name: kafka-broker-1  # <-- name of the TLSRoute from above, which could be derived from `{{ brokerId }}` or `{{ tlsRouteName }}` or something

Alternatives

No response

Additional context

No response

[lgo]

I'd love to also gauge whether what I suggested is something you'd consider, both because of the alpha resource as well as the complexity of a more generic resource template configuration. That way I don't end up with some hacks on a fork but can't upstream it (:

[scholzj]

I suspect that we will support the Gateway API directly. We also regularly mentioned it in our roadmap conference talks etx. But I think the TLSRoute API should first move from experimental to GA before we integrate it directly into Strimzi. Until then, you can easily self-manage it. We have a blog post about it - https://strimzi.io/blog/2024/08/16/accessing-kafka-with-gateway-api/ - but I guess you might not need it.

I expect that they way it would work will be similar as it is for Ingress today, just hopefully with a better compatibility. That said I do not understand why would we interfere with any other Gateway API resources instead of just managing the TLSRoute resources.

[scholzj]

That said I do not understand why would we interfere with any other Gateway API resources instead of just managing the TLSRoute resources.

Just to be clear ... feel free to explain what exactly you meant, why would that be good for everyone, etc. But please keep in mind that one of the things we learned for example with the Ingress type listener is that you can never please everyone. So we try to focus on the most common path and one of the main points of the ClusterIP listener is that when you need something special, you can do it yourself that way.

[lgo]

That said I do not understand why would we interfere with any other Gateway API resources instead of just managing the TLSRoute resources.
Just to be clear ... feel free to explain what exactly you meant, why would that be good for everyone, etc. But please keep in mind that one of the things we learned for example with the Ingress type listener is that you can never please everyone. So we try to focus on the most common path and one of the main points of the ClusterIP listener is that when you need something special, you can do it yourself that way.

Yep, the suggestion for a slightly generic template is a little aimed at the "hopefully please everyone!" route but I definitely understand and respect a stance from you provided your wealth of experience and prior art in strimzi!

Specifically, the challenge I've run into is:

• The EnvoyGateway implementation has defaults for TCP backends (TLSRoute entries), such as connection counts
• Kafka blows past any reasonable defaults for the Envoy pods, so it needs specific tuning to get working. (Otherwise, it's connection limit circuit-breaker trips which rejects new connections / causes general connection churn)
• This requires an associated Envoy configuration resource for each of the per-broker TLSRoute

There's some quirks with Gateway setups across namespaces also requiring additional Gateway resources per *Route to permit the references, although I don't recall how necessary they are. (I fumbled on this at an earlier point of setting things up and would need to refresh on the problems)

We do self-manage and have things up and running, so no problem there. I'm just hoping to solve some operational hiccups with new brokers coming up without valid external routes simultaneously that effectively hard-breaks producers/consumers until we get the routes up. The routes only get deployed with a deployment flow, namely:

• Where a replica bump on a KafkaNodePool gets applied simultaneously with a topic bump for new partitions (somewhat easier to solve)
• Where a manual operation might replace a broker without a deployment to push out those routes (harder to solve)