User reports for TCPRoute and UDPRoute

[youngnick]

This discussion is for folks who are actively using TCPRoute and UDPRoute. The community would like to know what you're using them for, and how you find the experience?

[pamelia]

I've been collaborating with a client on a project that focuses on secure video meetings. The service leverages WebRTC for real-time communication and incorporates the STUNner project as a WebRTC media gateway, running seamlessly in Kubernetes. STUNner uses Gateway API with UDPRoute for efficient traffic management, which has significantly enhanced scalability and reliability. It's been a fascinating experience working on such cutting-edge technologies to deliver secure and high-performing video communication solutions running natively in Kubernetes.

[LionelJouin]

I investigated Gateway API with TCP and UDP routes for a potential implementation of a Kubernetes Load-Balancer for Telco use-cases.

The first thing I noticed is the port and protocol that must be written in listener (Gateway), ParentReds (Route), BackendRef (Route) and Service. I have had a hard time to understand how the listeners field fits with the Routes on layer 4 (TCP/UDP Route) and potentially lower layers. Is it really required? Maybe a better solution could be to have the possibility to allow any port and any protocol? Here are some related discussion related to this: #130 ; #780 (comment) ; #818 ; #1061

For my use case on telco load-balancer, I would probably also need some kind of SCTP Route. But, in my opinion, a Layer 3 and 4 Route would probably make more sense. The route would match based on Src/Dst IPs and Ports + Protocol, and use some kind of service as a backend which would define the actual endpoints.

Few month ago, I made a PoC in this area, I would be happy to demo what I have to the community if there is an interest: https://github.com/LionelJouin/l-3-4-gateway-api-poc
It is about layer 3 and 4 Routes and (Kubernetes) service decomposition. In this PoC, the Gateway workload is collocated with the service implementation.
The 2 main experiments in this PoC are the following:

1. Service as a Route
   • The Kubernetes service object is considered and used as route + load-balancer (service.kubernetes.io/service-proxy-name indicates in which Gateway the service must be running).
   • It (Kubernetes Service) is considered as a route as we can specify the Port + Protocol + IPs.
   • It (Kubernetes Service) is considered as a load-balancer as we have a selector selecting the endpoints and load-balancing the traffic towards them.
2. Service decomposition: Layer 3/4 Route + Load-Balancer
   • New Route L34Route pointing to a service (backendRefs).
   • The service is a headless service containing only the Selector field (Maybe the service object could contain traffic manipulation fields. E.g. Load-Balancer algorithm, NAT being applied, Port forwarding, ...).

Probably what I said is also related to this discussion: #3351
We (I and some people from SIG-Network) also had some discussion around this topic in the hallway of Kubecon NA.

[kflynn]

Linkerd uses TCPRoute in managing egress. We added an EgressNetwork CRD to be a parentRef to define the class of "traffic leaving the mesh", gave the EgressNetwork the ability to specific default policy for that class, and then we allow Routes to express which egress traffic should be allowed. At present, we support GRPCRoute, HTTPRoute, TCPRoute, and TLSRoute in this role.

There's more detail in the Linkerd egress documentation.

[MeydanOzeri]

I worked on 2 use cases where UDPRoute was needed.

1. Deploying Janus inside Kubernetes to facilitate audio + video calls, requires UDP to work, and before it was supported we needed to deploy Janus on a VM introducing scaling difficulties.
2. Using WebTransport, the WebSockets successor which uses https3 and the QUIC protocol which requires UDP.

[mikemorris]

I'm curious if we may eventually want to add an explicit ProtocolType enum value for Gateway listeners to support handling http3/QUIC with an attached HTTPRoute (or are there capabilities for which we would actually want an entirely new route type?) rather than falling back to the rather basic functionality of UDPRoute.

[MeydanOzeri]

For WebTransport you currently need 2 routes, TLSRoute and UDPRoute, i didn't check but i assume even for regular http3 without WebTransport you still need both routes, or an HTTPRoute with TLS terminate instead of the TLSRoute with passthrough.

Since both TCP and UDP are needed behind the scene I'm not sure how it can be done using only HTTPRoute which attaches only to a TCP oriented gateway listener.
In order to be future proof and allow adopters to use WebTransport once it's not experimental anymore, i advise to support direct UDP capabilities, either with the existing UDPRoute or with a new route type.

WebTransport has 2 modes, reliable and unreliable, so either you use QUIC to facilitate traditional WebSocket like connections just on top of UDP and not TCP, or you use unreliable and send datagrams directly on UDP which is the preferred solution for media transfers, online gaming and other real time apps.

[mikemorris]

Since both TCP and UDP are needed behind the scene I'm not sure how it can be done using only HTTPRoute which attaches only to a TCP oriented gateway listener.

Yea, I'm basically questioning from a spec author and Gateway API implementing project perspective whether/what we might want to add to the spec to better serve this use case. It's really interesting what it sounds like you've been able to get working, but the UX of needing two routes feels a bit cumbersome and something we should maybe try to improve!

[MeydanOzeri]

Well, from this point of view it certainly would be better from a dev UX perspective to achieve it with a single route and listener, spares the need to go low level and deal with the underlying protocols of http3 which i assume a lot of devs might find cumbersome.

Perhaps adding a ProtocolType of type 'quic' and allowing normal HTTPRoutes to attach to it for majority of traditional use cases + a QUICRoute or the existing UDPRoute to allow for more flexibility and control in cases like datagrams on top of WebTransport.

[youngnick]

Great feedback, thanks @MeydanOzeri - we've talked about QUIC before, but I don't think any of us had enough experience with it to feel super comfortable designing something.