fix: warn if OperatorAdded events not in sync

[petarjuki7]

Issue Addressed

Adresses issue #519

Proposed Changes

We keep track of all OperatorAdded events in a new variable stored in the database (max_operator_id_seen).
Then, when we get a new OperatorAdded event, we can check does it match the expected OperatorId we should have seen next.

Also, during historical sync we check if we saw any OperatorAdded event, if not, there is likely a sync error.

[Copilot]

Pull Request Overview

Adds tracking and validation for OperatorAdded events to detect synchronization issues by monitoring sequence gaps and missing events during historical sync.

• Introduces max_operator_id_seen field to track the highest operator ID encountered
• Validates that OperatorAdded events arrive in sequential order during live sync
• Warns if no OperatorAdded events are found during historical synchronization

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
anchor/eth/src/event_processor.rs	Adds validation logic for OperatorAdded event sequence and missing events detection
anchor/database/src/table_schema.sql	Adds max_operator_id_seen column to metadata table
anchor/database/src/state.rs	Implements getter for max_operator_id_seen from database and state
anchor/database/src/sql_operations.rs	Defines SQL queries for retrieving and updating max_operator_id_seen
anchor/database/src/lib.rs	Adds max_operator_id_seen field to SingleState and bump operation

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[dknopik]

Nice! Here are my thoughts.

If we decide on not persisting the max operator id seen, I think the implementation might be straight forward enough to warrant inclusion in v1.0.0. wdyt?

[dknopik]

For simplicity, I am not sure if we should persist this. It might be sufficient to keep this as an Option<OperatorId> during sync, initialized on the first seen operatorAdded, and checked subsequently during this anchor instances lifetime

[petarjuki7]

I was thinking about persistence also... I think the problem may be with the historical sync because, if I understood the code correctly, after we do it for the first time we persist the block data, and on the next startup of the anchor instance we only sync from the last_processed_block, but our operator_seen count would start from 0 which could cause inconsistencies.

[dknopik]

That is correct. This is why I suggest using an last_operator_add_seen: Option<OperatorId>, initialized to None. To avoid the inconsistency you mentioned, we would not do the check on the first operatorAdded we see, but use it to initialize the last_operator_add_seen. Then we can do the check when receiving more operatorAdded.

The downside is that we do not catch missing operatorsAdded between the last run and the first operatorAdded we get in the current run. I am on the fence whether that is alright.

[petarjuki7]

Ah okay, I understand.
The downside might be fine, because it if it's truly not catching the events, we will find out soon enough?

[diegomrsantos]

we would not do the check on the first operatorAdded we see, but use it to initialize the last_operator_add_see

That's the crux of the issue, isn't it? A pruned client could have deleted the state containing some initial operators

[diegomrsantos]

The current approach might not be enough. Does the operator id start at zero or one and then continue increasing by 1? If so, we need to guarantee that we have seen all of them up to the current value in the contract.

[petarjuki7]

we need to guarantee that we have seen all of them up to the current value in the contract.

I think that was the downside discussed about this approach when we don't persist the max_operator_id_seen variable. We kind of "trust" the first OperatorAdded event we see will initialize the variable with the highest OperatorId.

I think the purpose of this PR is more to warn users that something with syncing is wrong so they configure their EL client correctly.

But I could implement a check that goes through our saved OperatorIds in the database and checks if they are monotonically increasing by 1 up until the max_seen?

[diegomrsantos]

What I described is a suggestion about how to check if Anchor has synced correctly starting with an empty DB. Additionally, we also need to persist the highest operator id we have seen to be able to do the check when we start syncing after being offline.

[diegomrsantos]

I think we do have to persist the highest_operator_id_seen. It will be created initially with the first value used by the contract.

But I could implement a check that goes through our saved OperatorIds in the database and checks if they are monotonically increasing by 1

This needs to be done during sync, starting with the current value in the DB.

[diegomrsantos]

And before we start syncing, we could read the last operator id from the ssv contract and check that see all ids up to this value. But not sure it's necessary

[petarjuki7]

Yeah, I understand the idea, can do it.
Just tagging @dknopik to confirm, if I remember correctly we decided on the non-persistence path yesterday.

[dknopik]

@diegomrsantos In yesterdays meeting we discussed that we can also not persist the highest_operator_id_seen and initialize it with the first operator ID seen per sync. This would be sufficient to catch any missing operatorAdded for the lifetime of that instance.

In the meeting, I asked if there is agreement to do it this way for now, as it is a tradeoff for not having to modify the database that closely to the release. While it is of course okay to change one's mind, I believe the simpler option is better for a short term change

[dknopik]

@petarjuki7 please rebase onto release-v1.0.0

[dknopik]

I think it would be a bit cleaner to just have set_max_operator_id_seen

[dknopik]

if we do not persist get_max_operator_id_seen, we need to check the operators map instead here, as very short historic syncs (e.g. after a restart) trigger this otherwise.