Skip to content

Commit 33fc4f8

Browse files
author
dave.seddon
committed
goTrackRTP
1 parent 4911205 commit 33fc4f8

22 files changed

+1877
-3
lines changed

‎.trunk/.gitignore

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
*out
2+
*logs
3+
*actions
4+
*notifications
5+
*tools
6+
plugins
7+
user_trunk.yaml
8+
user.yaml

‎.trunk/configs/.markdownlint.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Autoformatter friendly markdownlint config (all formatting rules disabled)
2+
default: true
3+
blank_lines: false
4+
bullet: false
5+
html: false
6+
indentation: false
7+
line_length: false
8+
spaces: false
9+
url: false
10+
whitespace: false

‎.trunk/configs/.yamllint.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
rules:
2+
quoted-strings:
3+
required: only-when-needed
4+
extra-allowed: ["{|}"]
5+
empty-values:
6+
forbid-in-block-mappings: true
7+
forbid-in-flow-mappings: true
8+
key-duplicates: {}
9+
octal-values:
10+
forbid-implicit-octal: true

‎.trunk/trunk.yaml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# This file controls the behavior of Trunk: https://docs.trunk.io/cli
2+
# To learn more about the format of this file, see https://docs.trunk.io/reference/trunk-yaml
3+
version: 0.1
4+
cli:
5+
version: 1.18.1
6+
# Trunk provides extensibility via plugins. (https://docs.trunk.io/plugins)
7+
plugins:
8+
sources:
9+
- id: trunk
10+
ref: v1.4.1
11+
uri: https://github.com/trunk-io/plugins
12+
# Many linters and tools depend on runtimes - configure them here. (https://docs.trunk.io/runtimes)
13+
runtimes:
14+
enabled:
15+
- [email protected]
16+
- [email protected]
17+
- [email protected]
18+
# This is the section where you manage your linters. (https://docs.trunk.io/check/configuration)
19+
lint:
20+
enabled:
21+
- [email protected]
22+
- [email protected]
23+
- [email protected]
24+
- [email protected]
25+
- [email protected]
26+
- [email protected]
27+
- [email protected]
28+
- git-diff-check
29+
- [email protected]
30+
- [email protected]
31+
- [email protected]
32+
actions:
33+
disabled:
34+
- trunk-announce
35+
- trunk-check-pre-push
36+
- trunk-fmt-pre-commit
37+
enabled:
38+
- trunk-upgrade-available

‎.vscode/launch.json

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
// Use IntelliSense to learn about possible attributes.
3+
// Hover to view descriptions of existing attributes.
4+
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
5+
"version": "0.2.0",
6+
"configurations": [
7+
{
8+
"name": "golang lanuch",
9+
"type": "go",
10+
"request": "launch",
11+
"mode": "auto",
12+
"program": "${file}"
13+
},
14+
{
15+
"name": "Test Current File",
16+
"type": "go",
17+
"request": "launch",
18+
"mode": "test",
19+
"program": "${file}",
20+
"env": {},
21+
"args": [],
22+
"showLog": true
23+
}
24+
]
25+
}

‎LICENSE

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2024 randomizedcoder
3+
Copyright (c) 2023 randomizedcoder
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

‎Makefile

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
all: test
2+
3+
test:
4+
go test -v
5+
6+
i:
7+
go test -v -run TestTrackerInit
8+
9+
tw:
10+
go test -v -run TestTrackerWindow
11+
12+
long:
13+
LONG=true go test -v -run TestLongRunningLoop
14+
15+
lw:
16+
LONG=true go test -v -run TestLongRunningWindow -test.timeout=30m
17+
18+
j:
19+
LONG=true go test -v -run TestLongRunningJumps -test.timeout=30m
20+
21+
# math tests
22+
math: l d
23+
24+
l:
25+
go test -v -run TestIsLess
26+
go test -v -run TestIsLessBranchless
27+
go test -v -run TestIsLessBranch
28+
29+
d:
30+
go test -v -run TestDiff

‎README.md

Lines changed: 243 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,244 @@
1-
# goTrackRTP
1+
# goTrackingRing
22

3-
This repo has a golang based library for tracking RTP sequence numbers
3+
This repo contains a library of code to monitor [RTC 3550 Real Time Protocoo (RTP)](https://www.rfc-editor.org/rfc/rfc3550) sequence numbers.
4+
5+
## RTP Sequence Reporting
6+
7+
It provides the ability to report on:
8+
9+
- RTP sequence loss over time
10+
- Specifically, it monitors loss within a particular "acceptable window" of packets. e.g. How many packets were lost out of the last 100?
11+
- How many packets were out of order?
12+
- How many packets were late?
13+
- How many jumps ( continuous gaps ) in sequence numbers were there?
14+
- How many duplicate packets were received?
15+
16+
This library is intended to used in conjunction with other code that handles packets and decodes the RTP header. Essentially, this library only concerns itself with tracking uint16 sequence numbers. The intended use is to expose Prometheus metrics to allow longer term reporting over time. e.g. Allows a network operator to monitor trends in packet losses.
17+
18+
There is also a very simple example implmentation.
19+
20+
### RTP Sequence numbers
21+
22+
Reminder on RTP sequence numbers:
23+
24+
- RTP sequence numbers are 2^16
25+
- Encoders should be randomly select the starting sequence number on startup
26+
- With no network losses, the RTP sequence numbers should increment by one
27+
- In real networks there are delays, reordering, which this code aims to track and report upon
28+
29+
This library aims to be efficent from a CPU and memory perspective, and so uses a [B-tree](https://en.wikipedia.org/wiki/B-tree) structure.
30+
31+
## Overview
32+
33+
The solution essentially tracks RTP sequence numbers and then categorizes an arriving packet into various categories.
34+
35+
This diagram provides an overview of the categories defined.
36+
37+
<img src="./rtp_sequence_numbers.png" alt="RTP Sequence Numbers" width="80%" height="80%"/>
38+
39+
### Definitions
40+
41+
| Definition | Variable | Description |
42+
| ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
43+
| Max() | Max() | Highest RTP sequence and current reference point. Future packets are all relative to this packet |
44+
| Ahead | | Packet with a higher sequnce number than >Max() |
45+
| Behind | | Packet with a lower sequnce number than <Max() |
46+
| Acceptable Window | <aw,>bw | Packets with sequence number in this range are accaptable, which is to total length bw+aw |
47+
| Ahead Window | aw | Sequence number is within +aw packets of <Max() |
48+
| Behind Window | bw | Sequence number is within -bw packets of >Max() |
49+
| Safety Buffers | | To avoid erronously jumping to a new tracking window, "safety" buffers behind/ahead allow the operator to make sure tracker doesn't reinitilize the window |
50+
| Ahead Buffer | ab | Ahead buffer is a logical gap, where if a packet arrives within this range the existing Max() and acceptable window will NOT be regenerated |
51+
| Behind Buffer | bb | Behind buffer is a logical gap, where if a packet arrives within this range the existing Max() and acceptable window will NOT be regenerated |
52+
| Restart | | If the sequence number jumps ahead/behind by a large amount then the encoder has restarted, so the acceptable window needs to be reinitilized |
53+
| Behind Restart | >ab | Sequence number arriving in this range reinitilizes the window |
54+
| Ahead Restart | <bb | Sequence number arriving in this range reinitilizes the window |
55+
| Sequence Roll | | For now, 2^16 sequence roll will be treated like a restart |
56+
57+
### Positions
58+
59+
| Position | Description |
60+
| -------- | ---------------------------------------- |
61+
| Unknown | |
62+
| Ahead | Within the acceptable |
63+
| Behind | Within the safety buffer, and so ignored |
64+
65+
### Categories
66+
67+
| Category | Description |
68+
| -------- | ------------------------------------------------------------------------------- |
69+
| Unknown | |
70+
| Window | Within the acceptable |
71+
| Buffer | Within the safety buffer, and so ignored |
72+
| Reset | Outside the acceptable window and buffer, causing reinitilization of the window |
73+
74+
### SubCategories
75+
76+
| SubCategory | Description |
77+
| ------------- | ------------------------------------------------------------------------ |
78+
| None | No additional sub catagorization |
79+
| Next Sequence | Next sequence is the packet that will ideally arrive next and is Max()+1 |
80+
| Duplicate | Duplicate packets are also identified |
81+
82+
> **Please note:**
83+
>
84+
> All the windows and buffers are defined in terms of _packets_ NOT _time_
85+
86+
## Roughly how this code works
87+
88+
### Items in the "acceptable window" are added to the B-tree
89+
90+
Items are added using [.ReplaceOrInsert()](https://pkg.go.dev/github.com/google/btree#BTreeG.ReplaceOrInsert).
91+
92+
For the happy path, this is the next sequence number, and so .Max() advances by one, so the B-tree will become slightly longer to the left.
93+
94+
If the packet is not in sequence, it will also be added to the tree, and may adjust the .Max() forward.
95+
96+
Typically, we expect either complete packet loss, or slight delays in packets, so the expectation is there are gaps in the sequence numbers, or "behind" packets, neither of which will advance the .Max().
97+
98+
This means the B-tree always holds:
99+
100+
- All the sequence numbers seen within the "acceptable window"
101+
- Iteration is required to get a list of the missing items, although for small window sizes the iteration is relatively inexpensive, although unless there is a very specific debugging scenario this is probably not required.
102+
- Count of the number of packets in the "acceptable window" ( .Len() ). The number of missing packets is merely the "acceptable window" size, minus the number of packets seen.
103+
104+
### Automatic rebalancing of the B-tree
105+
106+
Overtime, the B-tree will become longer on the left ( more items on the left of the tree ), and so the B-tree will rebalanced via rotation+merging. The rebalance is mostly pointer moves, so it's reasonably efficient.
107+
108+
### Items fall off the back of the "behind window"
109+
110+
Of course, as sequence numbers fall off the back of the "behind window", these items need to be removed. This is done by a simple [.Delete()](https://pkg.go.dev/github.com/google/btree#BTreeG.Delete) of a single item, which is just finding the minimum item, so it's efficent.
111+
112+
### Batch deletes ( jump ahead )
113+
114+
If a new item jumps forward the current position of .Max() by more than +1, then essentially multiple items need to be deleted. To make this operation more efficient the [.DescendLessOrEqual()](https://pkg.go.dev/github.com/google/btree#BTreeG.DescendLessOrEqual) iterator is used, deleting as it visits the node. B-trees are efficent at finding the next lower/higher, so this iteration is reasonably efficent.
115+
116+
( An alternative implmentation would be to repeatedly call [.DeleteMin()](https://pkg.go.dev/github.com/google/btree#BTreeG.DeleteMin) until the tail of the "behind window" is reached ( .Max() - bw ), but each delete would traverse the full tree and would not be as efficient as the .DescendLess. )
117+
118+
### Restart of window ( large jump behind/ahead )
119+
120+
If items arrive that are beyond the behind or ahead buffer ( wihtin the "Restart" zones in the diagram ), then these are deemed to be a restart of the RTP encoder. The existing tree is cleared via [.Clear()](https://pkg.go.dev/github.com/google/btree#BTreeG.Clear), and items are put back on the Freelist.
121+
122+
( Honestly, I haven't looked to closely at how the library manages the memory or garbage collection tuning, but hopefully this library is being used with relatively small windows like <=100, so this should be a pretty small memory footprint. I assume from reading words like "freelist" in the documnetation that the library is holding on to memory, which should keep the garbage collection low. I should probably do some profiling and update the finds here.)
123+
124+
### Safety buffers ( large jump behind/ahead )
125+
126+
Given that restarting the window/B-tree will wipe all the packet sequence history, there is a risk that if the window configuration is smaller than packets that may actually arrive, the window will be wiped.
127+
128+
e.g. You could imagine that occationally a packet gets delayed more than expected ( for some unknown reason ), and even if your audio/video decoder may ignore this late packet, the RTP sequence tracker may restart the window and wipe all your useful RTP sequence data.
129+
130+
You probably don't want this, so to protect against this, the "behind and ahead buffers" exist. Essentially, this allows you to configure a bit more space, most importantly the "behind buffer", so reduce this risk.
131+
132+
Of course, the downside of this approach is that there is a small risk the RTP encoder could legitimately restart and start with a new random sequence number that's within acceptable window + buffer range (bb+bw+aw+ab), but with 2^16 the chances are pretty slim, assuming you keep pretty small windows+buffers.
133+
134+
### Configuration comments
135+
136+
The intention is to allow an network operator to tune the monitoring windows to suit the particular network and reporting requirements.
137+
138+
When configuring the various window and buffer seetings, implementors should consider:
139+
140+
- Objectives of the monitoring,
141+
- RTP packet rates,
142+
- Network design in terms of redundant paths, and particularly how the network topology may change impacting end to end latency, particularly during re-convergence
143+
144+
Please keep in mind that the entire "acceptable window" worth of packet sequence numbers is held within the B-tree.
145+
146+
Please refer to this sheet for some simple Mb/s and packet rate calculations
147+
https://docs.google.com/spreadsheets/d/16Wcjm8JVv4121QuZAHokMtMJ6n_b4_QZN73iNydAT5w/edit?usp=sharing
148+
149+
### Example configuration
150+
151+
#### Network with modest variations
152+
153+
For the following environment:
154+
155+
- Video rate of ~10 Mb/s ( estimated packet size of 1380 bytes is ~725 packets per second )
156+
- Maximum network delay of up to 100 milliseconds ( ~725 packets )
157+
- Maximum network path length change of 100 milliseconds ( ~725 packets )
158+
159+
The following configuration might be a good place to start:
160+
161+
| Variable | Packets | Comment |
162+
| -------- | ------- | -------------- |
163+
| aw | 725 | ~100 ms ahead |
164+
| bw | 725 | ~100 ms behind |
165+
| ab | 3600 | ~500 ms |
166+
| bb | 3600 | ~500 ms |
167+
168+
This would allow for ~0.2 seconds ( 200 ms ) or ~1449 packets of "acceptable window".
169+
170+
#### Crazy network with pretty large variations
171+
172+
For the following environment:
173+
174+
- Video rate of ~1 Mb/s ( estimated packet size of 1380 bytes is ~725 packets per second )
175+
- Maximum network delay of up to 1 second ( 1000 ms is ~725 packets )
176+
- Maximum network path length change of 0.5 seconds ( 500 ms is ~362 packets )
177+
178+
The following configuration might be a good place to start:
179+
180+
| Variable | Packets | Comment |
181+
| -------- | ------- | --------------- |
182+
| aw | 362 | ~500 ms ahead |
183+
| bw | 725 | ~1000 ms behind |
184+
| ab | 500 | ~690 ms |
185+
| bb | 500 | ~690 ms |
186+
187+
This would allow for ~1.5 seconds ( ~1500 ms ) or ~1087 packets of "acceptable window".
188+
189+
Diagram Google Slides link: https://docs.google.com/presentation/d/1gkgs0uZ6YDqRBUeYwPjZWI2JgWBN_54CXdNvueBpjXc/edit?usp=sharing
190+
191+
## Performance considerations
192+
193+
This library was originally designed to monitor RTP video at rates <20 Mb/s, and has not been tested for video rates higher than this. e.g. Not tested with SMPTE-2110 video transport. The b-tree operation times should mostly be <200 ns, so there's a chance it will work ok, but it would need to be carefully tested and potentially some tuning could be done.
194+
195+
Please also note that B-Tree "degree" is currently hard coded to three (3). Tuning this is likely to be required for higher packet rates.
196+
197+
## RTP Header
198+
199+
https://www.rfc-editor.org/rfc/rfc3550#section-5.1
200+
201+
```bash
202+
5.1 RTP Fixed Header Fields
203+
204+
The RTP header has the following format:
205+
206+
0 1 2 3
207+
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
208+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
209+
|V=2|P|X| CC |M| PT | sequence number |
210+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
211+
| timestamp |
212+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
213+
| synchronization source (SSRC) identifier |
214+
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
215+
| contributing source (CSRC) identifiers |
216+
| .... |
217+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
218+
```
219+
220+
## Btree libraries for golang
221+
222+
This is a list of some golang btree implmentations.
223+
224+
For now, I've decided to use [Google's golang btree](https://pkg.go.dev/github.com/google/btree) implmentation, until I know a reason not too.
225+
226+
Functions: https://pkg.go.dev/github.com/google/btree#pkg-functions
227+
228+
The scylladb writeup make it look like a reasonable library:
229+
230+
https://www.scylladb.com/2022/04/27/shaving-40-off-googles-b-tree-implementation-with-go-generics/
231+
232+
We are using the "generic" implemention: https://github.com/google/btree/issues/41
233+
234+
Other AVL tree implementations
235+
236+
https://github.com/VictorLowther/btree
237+
238+
https://github.com/tsuzu/go-avl/blob/master/avl_test.go
239+
240+
https://github.com/ross-oreto/go-tree
241+
242+
### Note for myself
243+
244+
Markdown syntax link: https://www.markdownguide.org/basic-syntax/

0 commit comments

Comments
 (0)