Order for Config change

[maxime-bruno]

As I was implementing #132, I stumbled on a question.

When a cell updates its config, on which packet does it apply?

• On all packets, including the one already arrived but which haven't been dequeued yet
• On packets enqueued after the config change

[BobAnkh]

For this issue, it is up to the cell itself to decide. However, for the 'official' cell we have currently implemented, it will affect all packets that have not yet been dequeued from the cell to varying degrees (I mean including those that haven't enqueued). But the extent of this impact may differ for those packets(this is related to config).

Let me give you an example. For instance, in a bandwidth cell, after changing the bandwidth, it will recalculate the next opportunity to send out a packet, thus having an equal effect on all packets that have not yet been enqueued or dequeued in this cell. On the other hand, if we change the queue length and it becomes less than the number of packets already in the queue, then under our current implementation, none of the packets already in the queue will be discarded while any new packets entering the cell will be dropped.

I’m not sure if my explanation and example clarify your doubts, or if you have any further questions or suggestions regarding our current implementation and design?

[maxime-bruno]

Your explanation is quite clear.
So currently it is an undefined behaviour and each cell behave as it want. And the guide line for default cells is try to apply to all packet not yet dequeued.

This raise an issue, it is not how I designed the DelayPerPacketCell. So I'll have to change it.

I don't think there is a perfect approach and the current one is ok. But we could argue that the undefined behaviour is not the best idea, especially since the documentation is not really good at the moment.

An idea could be, following the timestamp idea of #132, that when updating a config, we send a timestamp alongside the config.
This way the config would start to be applied on this instant. And as such packet that should have been dequeued before this instant, are with the old config, the one in the middle is send accordingly to the proper behaviour (like half old, half new bandwidth), and the one after with the new config.

[BobAnkh]

This raise an issue, it is not how I designed the DelayPerPacketCell. So I'll have to change it.

I think this cell has a middle-ground feel, so I can accept both implementations. However, maybe we need some documentation to explain this behavior? It might be complex for us to modify the delays of all packets within the cell accordingly? For this issue, you can decide whether to make changes or not.

An idea could be, following the timestamp idea of

Regarding this idea, I actually thought of it when I see this issue. Let me briefly state one of our visions: we believe that the internal logic of each cell, due to the strict logical clock for ordering (by #132), will resemble discrete event simulation more closely. I completely agree with the behavior described as 'half new and half old'. This is also why we think a calibrated timer can solve issues related to token buckets or bandwidth cells (#131)—in short, the problem of how the order in which packets enter and leave at both ends of a queue affects packet loss outcomes.

[maxime-bruno]

I think this cell has a middle-ground feel, so I can accept both implementations. However, maybe we need some documentation to explain this behavior?

Indeed I agree that the Cell trait may leave it as an undefined behaviour, but Cell struct should clearly state the chosen behaviour.

It might be complex for us to modify the delays of all packets within the cell accordingly? For this issue, you can decide whether to make changes or not.

I'll see what I do when I modify everything to match #132 .

An idea could be, following the timestamp idea of

Regarding this idea, I actually thought of it when I see this issue. Let me briefly state one of our visions: we believe that the internal logic of each cell, due to the strict logical clock for ordering (by #132), will resemble discrete event simulation more closely. I completely agree with the behavior described as 'half new and half old'. This is also why we think a calibrated timer can solve issues related to token buckets or bandwidth cells (#131)—in short, the problem of how the order in which packets enter and leave at both ends of a queue affects packet loss outcomes.

Indeed, I agree with the issue you are talking about, a solution could be to drop the packet not when they arrive in the cell, but when a packet leaves to ensure that we have at most the right number of packet/ amount of data that is queued until the instant the packet leaves.

On another issue about the reordering of packet, I don't know if you noticed but the loopback interface reorders packet when sending to many packets at once.
I stumbled on this issue while using rattan (hence test to check that rattan cells do not reorder packets).
I'm currently working on a xdp probe to catching packet before leaving the socket, ensuring they have the expected order. But it may be more relatable to #56 and #47.

[BobAnkh]

Indeed, I agree with the issue you are talking about, a solution could be to drop the packet not when they arrive in the cell, but when a packet leaves to ensure that we have at most the right number of packet/ amount of data that is queued until the instant the packet leaves.

On another issue about the reordering of packet, I don't know if you noticed but the loopback interface reorders packet when sending to many packets at once. I stumbled on this issue while using rattan (hence test to check that rattan cells do not reorder packets). I'm currently working on a xdp probe to catching packet before leaving the socket, ensuring they have the expected order. But it may be more relatable to #56 and #47.

Although rattan implements both AF_PACKET and AF_XDP features (the camellia feature to enable AF_XDP), I am not familiar with this part, so @minhuw and @Centaurus99 can provide some answers regarding the above situation. For #56, it is actually just for the reuse of veth and netns; as for #47, I don't remember why it wasn't implemented at that time, so we may need to ask @minhuw and @Centaurus99 to explore a bit.