Gateway with multiple certificates for multiple hostnames?

[mloskot]

Background

An interesting use case was discussed on the sig-network-gateway-api channel on Kubernetes Slack of configuring single Gateway with single HTTPS listener, but multiple non-wildcard TLS certificates attached:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared-application-gateway
  namespace: default
spec:
  gatewayClassName: ...
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        group: ""
        name: grafana-tls-secret
      - kind: Secret
        group: ""
        name: prometheus-tls-secret
      mode: Terminate

and HTTPRoute-s for every hostname:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: grafana-route
  namespace: grafana
spec:
  hostnames:
  - "grafana.example.com"
  parentRefs:
  - name: shared-application-gateway
    namespace: default
  rules:
  - backendRefs:
    - name: grafana
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: prometheus-route
  namespace: prometheus
spec:
  hostnames:
  - "prometheus.example.com"
  parentRefs:
  - name: shared-application-gateway
    namespace: default
  rules:
  - backendRefs:
    - name: prometheus
      port: 9090

Since the https://gateway-api.sigs.k8s.io/geps/gep-851/ replaced CertificateRef with CertificateRefs to allow multiple certificates, such configuration is valid according to the Gateway API 1.4 specification.
However, https://gateway-api.sigs.k8s.io/reference/1.4/spec/#listenertlsconfig makes it clear certificateRefs support for multiple certificates is implementation-specific, quoting:

Implementations MAY choose to support attaching multiple certificates to
a Listener, but this behavior is implementation-specific.
(...)
Support: Core - A single reference to a Kubernetes Secret of type kubernetes.io/tls
Support: Implementation-specific (More than one reference or other resource types)

Does kgateway support multiple certificates?

In general case, yes. Since, I think, merge of #12895 the kgateway features:

multiple certificates with different signatures (ECDSA and RSA)
pass all certificateRefs through to Envoy now, as Envoy natively is able to pick the right certificate based on SNI and client capabilities.
[tests] confirm that both use cases of serving two certificates with different signature algorithms as well as different certificates for different host names work as expected.

and there is also this foot note in that PR:

It might also make sense to expand the docs pages for HTTPS and SNI with multi-certificate listener examples.

In a specific case, I think, there is room for clarification.

First, I found that multiple certificates support is documented in https://kgateway.dev/docs/envoy/latest/setup/listeners/sni/
But, it 1) discusses a different use case than the one I copied from the Kubernetes Slack above 2) may be confusing, I think, because it says "Serve multiple hosts on the same HTTPS listener" but the Gateway configuration there clearly presents multiple (two) listeners. Why the document repeats "the same listener" wording? Because internally it merges the configuration into one listener what effectively expose the gateway via single IP?

I think it is may be worth to list possible use cases and clarify their support in kgateway:

1. Single listener per Gateway:
   a) single TLS certificate in single secret, then single or multiple secrets listed in certificateRefs
   b) multiple TLS certificates in single secret, then single secret listed in certificateRefs
2. Multiple listeners per Gateway:
   a) single certificate in single secret listed in certificateRefs per single listener
   • essentially, use case from https://kgateway.dev/docs/envoy/latest/setup/listeners/sni/
     b) multiple certificates per listener, complex case of the 1. above.

where each "TLS certificate" above can be:

• single domain certificate
• multi-domain certificate (SAN)
• wildcard certificate

Are there any limitations of what kinds of certificates can be attached?

Outcome of this discussion may make it easier to decide on improvements of the documentation and the tests too.

[yuval-k]

We should probably have something in our docs too, but here's the relevant envoy docs that describe the logic for multiple certs on the same gateway listener (in TLS listener becomes a filter-chain in envoy):
https://www.envoyproxy.io/docs/envoy/v1.36.4/intro/arch_overview/security/ssl#arch-overview-ssl-cert-select

This is the TL;DR:

• DNS SANs or Subject Common Name is extracted as server name pattern to match SNI during handshake. Subject Common Name is not used if DNS SANs are present in the certificate.
• FQDN like “test.example.com” and wildcard like “*.example.com” are valid at the same time, which will be loaded as two different server name patterns.
• If multiple certificates of a particular type (RSA or ECDSA) are specified for the same name or name pattern, the first one loaded is used for that name.

If there are multiple listeners - I believe the filtering the SNI using the listener hostname happens first, and then the certificate selection logic described above.