Making UDP send latency more predictable (i.e. reducing jitter) (IDFGH-14588)

[h-milz]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hi all,

my application is a low-latency UDP audio bridge using two ESP32-C5 sample Devkits, but I suppose the question can be answered with ESP32-C6 knowledge. I'm using the 5 GHz band with 11ax with the receiver taking the role as AP and the sender is the only station, with nothing in between and no other device in the same SSID. Wifi encryption is WPA3-SAE. The latency requirement is in the range of 10-12 ms. Basically, I got this to work but there are still occasional pops caused by buffer overruns. The reason turned out to be on the sending side. Datagrams are created and sent (using sendto()) every 1.36 ms triggered by the i2s on_sent callback (I do no use i2s_channel_read() or i2s_channel_write() - they are just too laggy). The UDP payload size is 1452 bytes so that no IP fragmenting takes place (and it is deactivated anyway). (For explanation, each datagram contains 60 samples with 8 slots with 3 byte samples each, summing up to 1440 bytes, plus a 4 byte XOR checksum, a 4 byte packet sequence number, plus a 4 byte data field for other stuff. The sample rate is 44.1 kHz. Each second, 735 datagrams are sent.)

Most of the time, as I can see on the receiver and using wireshark, UDP packets are sent in regular intervals with an acceptable jitter (say, 1360 +/- 500 µs). However, at random intervals, datagrams appear to pile up for 10..20 ms or more before actually being sent, and then the piled up datagrams are sent in a fast burst, like after 12000 µs, 150, 220, 180, 150, ... about 10 or so packets.

On the receiver, I can see that in fact not a single packet is lost, and all packets arrive in sequence. My sequence-number based buffering can cope with arrival delays up to 6 datagrams. But at these occasions, packets are delayed by more than 10 or 15 ms, and this at first dries out the buffer and then overruns it. When the incident is over, everything is back to normal within two more packets.

So the question is, how do I prevent UDP packets from piling up on the sender, or how can I force packets to be sent immediately.

• I tried to set the FreeRTOS tick rate to 500 Hz, hoping that this would trigger the Wifi send more often, to no avail.
• Reducing the number of dynamic Tx buffers to 5 and thus forcing a send more often did not help.
• QoS makes no sense because there are no other packets to begin with.
• Wifi power saving is disabled.
• On the sender and the receiver, there is only one user task each, running at priority 15 or so, but competing for CPU time with the WiFi task I suppose. The only other things with higher priorities are the on_sent and on_recv callbacks but they are short and don't take longer than a few µs.
• I also tried to make the payload smaller by reducing the number of I2S frames to, say, 40 or 50, but this makes the system even less robust against UDP jitter because the interrupt rate is then higher, and each arriving packet has less time for processing. 60 seems to be the sweet spot.
• Making the receive buffer large enough to cover real outliers would break the latency requirement, and I would end up somewhere between 25 and 30 ms.
• ESP-NOW I didn't try yet because it is prone to denial of service, lacking authentication - and I don't know if this would solve the problem anyway because ESP-NOW sits on top of the data link layer, like the IP layer, and thus presumably faces the same Wifi Tx buffering issue.
• I also found a reference to using esp_wifi_internal_tx() but that's related to sending raw ieee80211 data.
• Also, it's not the maximum throughput. Using the iperf example, I measured a max sustained UDP data rate for a 1440 byte payload of 62 MBit/s between the two boards using MCS7_SGI, and my application uses only about 8.5. The raw UDP packet latency from sendto() to recvfrom() is about 540 +/- 200 µs including the (rare) outliers.

For now, I ran out of other options. Any help is appreciated!

[bryghtlabs-richard]

FYI: if WPA3-SAE has GTK rekeying like WPA2, we routinely see latency of ~1s during this process. If you're in control of the network, you may want to disable this.

Another thing that can go wrong: if a frame is lost, layer2 will often repeat it at a lower speed. If this is happening due to intermittent interference, sometimes layer2 will drop below minimum acceptable bitrate for your application. In this case, it may be better to disable lower radio rates.

How are you recording with WireShark? Is it sitting at the receiver at Layer3? Or is it with another machine capturing at layer2 with radiotap headers? If you have the radiotap headers, you should be able to tell what MCS the radio is doing per packet, as well as acknowledgements.