Add _rb_last_synced_tick to network-synchronization-server, to preven…

[jburdecki]

Closes #630

Adds _NetworkSynchronizationServer::_rb_last_synced_tick as a way to gate sending state. Only new (unsimulated) ticks are sent. This prevents issue #630's root cause of increasingly deep rollbacks for clients with "one-way-latency" greater than input_delay.

Video 1: with input_delay = 0 (fixed)
https://github.com/user-attachments/assets/7422a282-9a43-4aaf-a84b-808a009fefd5

Video 2: with input_delay = 6 (fixed)
https://github.com/user-attachments/assets/b32dcb15-37b4-4243-bb64-2c113fb59d13

[aromancev]

I think this could lead to unexpected behavior because local simulation might rely on intermittent states being correct, not only the latest one.

But the argument about not sending the same tick multiple times is valid and is also addressed in this PR: #618

[jburdecki]

Good find @aromancev 👍

Updated PR to also track the "last synced" snapshot, which covers rollback where intermittent states read/write data.

[aromancev]

Sorry, I meant it a bit differently.

The current change basically never sends the same tick data over network. My point is that it's a more radical change than never sending the same data.

It could matter in cases where local simulation relies on the previous ticks to simulate correctly. For example,

• Ticks 1,2,3 were simulated and all sent to the peer.
• After receiving late inputs, host resimulates ticks 2 and 3.
• Only tick 4 with correct values will be sent to the peer. But local predicted simulations on that peer may rely on ticks 2 and 3 being correct to simulate correctly.

This increases the chances of desync between peers and host simulations, especially if peers rely a lot on predicted behavior such as projectiles.

My point is that we could achieve similar bandwidth savings by remembering what was sent in each tick, so that unchanged fields and not sent again. That would still preserve correctness of the simulation on all peers. This is what is done in #618 already.

[jburdecki]

Is it right to understand the concern here is for state that a game (netfox) doesn't replicate? If so I think this PR (#631) is okay as-is, because my understanding is that for all state that a RollbackSynchronizer broadcasts, the newest received state is complete.

Looking at the example through that lens:

1. Ticks 1,2,3 were simulated and all sent to the peer.
   2. After receiving late inputs, host resimulates ticks 2 and 3.
2. Only tick 4 with correct values will be sent to the peer. But local predicted simulations on that peer may rely on ticks 2 and 3 being correct to simulate correctly.

Step 2. is where "local predicted simulations" diverge between host and peers with this PR. However, in my opinion, that's okay. This divergence of local (non-replicated) state is the result of what state a game chooses to replicate via netfox - and so the affected state in this example is not replicated.

As a specific example, I would see this local state divergence to be expected/acceptable for "side-effects" that are dependent on frame specific triggers, like minor SFX, VFX, and lerps/tweens for UI elements, etc. For any "side-effects" that must not diverge, either those should be included/tracked via a RollbackSynchronizer entry, or a reliable RPC.

[jburdecki]

However, maybe more to your point PredictiveSynchronizer would see a change in behavior with this PR, as resims triggered by old ticks are its only repair path... I think that's also okay since the intended behavior is without reliance on networking - but I haven't used this node and am unfamiliar with it's role:

Similar to [RollbackSynchronizer](https://foxssake.github.io/netfox/latest/class-reference/RollbackSynchronizer/), this class manages local variables in a rollback context for predictive simulation without networking.

[jburdecki]

To #618, I like where it's pointing, as a means to reduce bandwidth - however the field deduplication in it doesn't cover the failure this PR is targeting. Eg: on #618, with a connection where "one-way-latency > input_delay", the resim window will still deepen every frame, up to history_limit, and maintain that depth, causing the jitter/desync shown in the issue videos.

[aromancev]

Yeah, this counter point is valid if we assume that synced state cannot interact with non-synced state that is important for local prediction :)

Here is a specific example.

Two entities:

• Player - Simulated and synced between peers
• Projectile - Simulated and predicted, but not synced between peers

Syncing every projectile would consume a lot of bandwidth. There can be hundreds in flight at any point, and it is not necessary to sync their position every tick.

At the same time, projectiles affect (predicted) gameplay state. They may push a character, reduce HP, break concentration, etc. It's not always better to do it but it is a choice a developer can make. I do it for some cases in my game.

Now here is the interesting part.

The position of the player at each tick is originally simulated and synced as follows:
tick 1 - pos 0
tick 2 - pos 0
tick 3 - pos 0

Effectively, standing still.

And the position of the projectile is as follows:
tick 1 - pos 1
tick 2 - pos 2
tick 3 - pos 3

Effectively, flying in a straight line, missing the player.

After receiving late Inputs, host resimulates the position of the player to the following:
tick 1 - pos 1
tick 2 - pos 0
tick 3 - pos 0

So the player was actually at a different position and only moved to position 0 later. So in this PR the server will sync the correct state for the player only from tick 4. This is fine for the player but not for the projectile.

The projectile should have hit the player on tick 1, which it did on the host, but not on the client. The client never observed player in position 1 on tick 1 so the projectile missed it. The projectile will not correct itself because it is not synced.

This will not lead to a permanently broken state but it will produce a short lived desyc. The probability of this happening will differ from game to game and will depend on the amount of predicted state and the frequency of interactions.

With this change, we are forcing a weaker consistency model on all users.

The problem we are trying to solve is to reduce the amount of traffic we're sending. My point is that we can get 90% there by not sending the exact same values for the exact same tick, but at least sending what was different after a resimulation for each tick.

So there are two ways we can proceed. Either the option I'm suggesting is saving enough traffic and then we can avoid a weaker consistency model or it's not a at the very least we should make it an option so that developers can choose their tradeoff.

I'm actually not sure I understand why resending state will ever extend beyond the natural latency of the peer. If latency is 150ms at 20 ticks per second and Input delay is 1 tick, you can expect to sync 3 - 1 = 2 extra ticks worth of state data every tick. If you diff it against previously sent state, it will amount to almost nothing.

There is no reason for peers to fall behind even more just because they received state updates from the past. This is exactly what I onserved in my tests as well. Even before the recent fix of not using unreliable channels. If anything, CPU would be the bottleneck there, not network.

[jburdecki]

The premise of only broadcasting the latest state from a rollback event, and as a result creating "a weaker consistency model", makes sense to me.

That said, as I haven't contributed to netfox before and haven't joined architectural conversations, I don't have a strong statement on if it should be:

1. the only option
2. the default
3. a secondary option

But I believe one of those should be available to developers.

[jburdecki]

In the context of the "projectile, player" example, I'm interpreting the projectile as a gameplay node/entity (eg: static mesh), rather than an algo (eg: hitscan).

My opinion here is that If a developer wishes for higher consistency/replication on projectiles then that data needs to be replicated, and any effects caused by that projectile on the host also need to be replicated. For instance during resim where the Projectile and the Player intersect at Pos 1, on tick 1, RollbackSynchronizer sends the latest state from the host for tick 4, then either:

1. if the projectile is not tracked in a RollbackSynchronizer, reliable RPC the deletion of the projectile, with info on when, where, and how
2. clients receive that and
   • clean up the projectile
   • play any side-effects (fast-forwarded as needed from their auth pos/time of tick 1, to current tick)

or:

1. include the projectile tracking data in a RollbackSynchronizer
2. when it's updated for clients, following the tick 1-4 resim
   • clean up the projectile
   • play any side-effects (fast-forwarded as needed from their auth pos/time of tick 1, to current tick)

[jburdecki]

Rereading this thread, I think we're closer to agreement than it looks.
I see #618 and this #631 as complementary.

To revisit some points:

1. Re: the difference of coverage between fix: stale auth state poisoning #618 and this PR: I could have said that better - in fix: stale auth state poisoning #618 the resim window doesn't "[continue to deepen]". It does stabilize, eventually, but that's still a problem because with old ticks being sent every peer has to resim from the oldest tick, every frame, and to your point, that's potential for a CPU bottleneck (the problem in the videos).

2. While fix: stale auth state poisoning #618 changes what's in the packets, that may or may not minimize footprint by "90%". Depends on what state is sync'd and how often it changes. For instance, the diffs themselves won't always be "almost nothing" for analog input and/or physics games.

3. My opinion is this should be the stock behavior, because of (1.) and (2.)

4. However, I agree this change could be a netfox setting/option 👍

[aromancev]

If you actually observed perpetual resim increase up to history length, would be curious to see an example project.

I don't see how it would happen but maybe I'm missing something.

As for syncing projectiles or collision events, that's one way to do it, yeah. I prefer not to because that has its own trade-offs. So some optionality would be nice in any case.

I think we should wait for Tamas anyway to make a visionary decision 😅