feat: Operator Doppelgänger Protection

[diegomrsantos]

Issue Addressed

Closes #627

Proposed Changes

Implements Operator Doppelgänger Protection to prevent accidental slashing when multiple instances of Anchor run with the same operator ID. The feature monitors the network during a configurable waiting period to detect if another instance is already active.

Core Implementation

• State Management (operator_doppelganger/state.rs):

  • Tracks operating mode (Monitor/Active) with startup grace period
  • Grace period prevents false positives from own old messages after restart
  • Messages remain in gossipsub mcache for ~4.2s (history_length × heartbeat_interval)

• Service Layer (operator_doppelganger/service.rs):

  • Orchestrates monitoring period based on epoch progression
  • Background task sleeps for grace period, then monitors epochs until waiting period ends
  • Detects single-signer messages with own operator ID after grace period
  • Watch channel broadcasts monitoring status to block duties until safe

• Integration (lib.rs, message_receiver/manager.rs):

  • Integrated into message receiver pipeline
  • Services wait for sync completion AND doppelgänger monitoring
  • Graceful shutdown on twin detection

• Grace Period Configuration (network/behaviour.rs):

  • Automatically derived: (history_length × heartbeat_interval) / 1000 + 1
  • Default: 5 seconds (gossip cache window is 4.2s)
  • Updates automatically when gossipsub parameters change

Configuration

• --operator-dg: Enable protection (default: true)
• --operator-dg-wait-epochs: Monitoring period in epochs (default: 2)

Grace Period Rationale

After restart, gossipsub seen_cache is cleared but messages remain in peers' mcache for up to 4.2s. Without a grace period, the node would receive its own old messages via IHAVE/IWANT and incorrectly detect a twin. The grace period ensures all old messages expire before detection starts.

Testing

9 tests covering:

• Twin detection with single-signer messages
• Grace period prevents false positives during startup
• Multi-signer aggregate messages ignored
• Different operator IDs ignored
• Mode transitions after monitoring period

Additional Info

Design Decisions:

• Grace period approach: simpler than height tracking, works uniformly for all message types
• Derived from gossipsub config: automatically stays synchronized
• Watch channel: allows multiple services to wait for monitoring completion

Future Considerations:

• Metrics for doppelgänger events (monitoring status, detection count)
• Configurable actions on detection (shutdown vs. alarm only)

[dknopik]

I do not think this is necessary. Have you tested this and confirmed there to be an issue without this feature?

[diegomrsantos]

Could you elaborate more on why you think it's not necessary?

[dknopik]

I understand now that this is about preventing replay attacks, not necessarily about this occurring naturally.

[diegomrsantos]

This isn't about replay attacks. What makes you think so?

[dknopik]

What makes you think so?

The sentence "This prevents false positives from replayed old messages."

If another node maliciously holds onto one of our messages, they can use it to crash us once we do not have it in the duplicate cache anymore (due to cache timeout or node restart).

[diegomrsantos]

What makes you think so?

The sentence "This prevents false positives from replayed old messages."

If another node maliciously holds onto one of our messages, they can use it to crash us once we do not have it in the duplicate cache anymore (due to cache timeout or node restart).

The sentence is misleading; the idea is a protection from a protocol behavior. I thought about this other malicious case, but I don't know a solution to it yet. Not only is our gossipsub seen cache empty, but our message counts in validation as well.

[diegomrsantos]

It's still a heavily drafted PR; it is not ready for review

[claude-code-actions-sigp]

Claude Code is working…

I'll analyze this and get back to you.

View job run

[diegomrsantos]

I feel like a lot of this is needlessly complex

It has been simplified substantially

[claude-code-actions-sigp]

Claude finished @diegomrsantos's task —— View job

---

Code Review: Operator Doppelgänger Protection

Comprehensive review of the simplified implementation completed. The refactoring has addressed most of the previous concerns - excellent work on the simplification!

---

Summary

Overall Assessment: This is a well-designed and significantly improved implementation. The recent simplification (removing epoch tracking, generics, and watch channels) has made the code much cleaner while maintaining safety guarantees. The implementation is production-ready with minor suggestions for polish.

Key Strengths:

• Simple, focused state machine (3 states, clear transitions)
• Clean single-sleep timer design (no complex epoch polling)
• Excellent documentation of the grace period rationale
• Comprehensive test coverage using async timers
• Proper thread safety with Mutex and Arc
• Grace period automatically derived from gossipsub parameters

Recommendation: ✅ Approve with minor suggestions. The critical issues from previous reviews have been resolved through the simplification.

---

Architecture & Design ✅

Excellent Simplification

The refactoring successfully addressed @dknopik's "needlessly complex" feedback:

1. Removed unnecessary generics - No more PhantomData<E>, direct use of OwnOperatorId
2. Simplified timing - Single sleep(grace + monitoring) instead of slot-by-slot polling
3. Removed watch channel - Direct state checking instead of broadcasting
4. Eliminated epoch tracking - Calculate duration upfront, no need for monitor_end_epoch
5. State machine internalized - DoppelgangerState is now private, cleaner API

Before: ~200 LOC service with generics, watch channels, epoch polling loop
After: ~150 LOC with single timer, direct state transitions

This is a textbook example of refactoring for simplicity while preserving correctness.

---

Code Quality

1. State Management ✅

Location: anchor/operator_doppelganger/src/service.rs:22-77

The state enum is clean and well-documented:

GracePeriod → Monitoring → Completed

The documentation explaining the grace period (lines 24-41) is excellent - it clearly explains the gossipsub cache vulnerability and why we need to wait.

Minor suggestion: Consider adding a state transition diagram in the module-level docs for discoverability.

2. Timer Logic ✅

Location: anchor/operator_doppelganger/src/service.rs:123-149

The new single-sleep approach is much simpler:

tokio::time::sleep(grace_period).await;
self.state.lock().end_grace_period();
tokio::time::sleep(monitoring_duration).await;
self.end_monitoring_period();

This replaces the previous slot-by-slot polling loop, eliminating complexity and potential timing bugs.

3. Detection Logic ✅

Location: anchor/operator_doppelganger/src/service.rs:173-216

The is_doppelganger() method is clean and has proper early returns:

• Checks monitoring state first (fast path)
• Handles None operator ID gracefully (still syncing)
• Filters multi-signer messages correctly
• Logs detailed context on detection

Good: Separation of detection (is_doppelganger) from action (check_message)

---

Security Considerations

1. Grace Period Calculation ✅

Location: anchor/network/src/behaviour.rs:34-37

pub const OPERATOR_DOPPELGANGER_GRACE_PERIOD_SECS: u64 = {
    let cache_window_millis = GOSSIPSUB_HISTORY_LENGTH as u64 * GOSSIPSUB_HEARTBEAT_INTERVAL_MILLIS;
    cache_window_millis / 1000 + 1
};

Excellent: Automatically derived from gossipsub parameters (4.2s cache + 1s buffer = 5s). This will stay synchronized if gossipsub parameters change.

Consideration: The +1 second buffer is reasonable but not explicitly justified. Consider documenting why 1s is sufficient (network delays, clock skew, etc.). See inline comment.

2. Message Filtering ✅

Location: anchor/operator_doppelganger/src/service.rs:190-201

The single-signer check correctly prevents false positives:

• Multi-signer aggregates are ignored (correct - those are legitimate)
• Only messages with exactly our operator ID are flagged
• Works for both QBFT and partial signature messages (good!)

Note: Previous concern about partial signatures not being checked has been addressed - both message types are now checked in manager.rs:168-180.

3. Shutdown Safety ✅

Location: anchor/operator_doppelganger/src/service.rs:228-231

let _ = self
    .shutdown_sender
    .lock()
    .try_send(ShutdownReason::Failure("Operator doppelgänger detected"));

Good: Uses try_send to avoid blocking. The let _ = is acceptable here since we're shutting down anyway.

Minor suggestion: Consider logging if the send fails, though in practice this shouldn't happen.

---

Performance

1. Lock Contention ⚠️

Location: anchor/operator_doppelganger/src/service.rs:178

let state = self.state.lock();

Every incoming message acquires this lock, even after monitoring completes.

Impact: Low during monitoring (2 epochs = ~12 minutes), but this continues forever after monitoring ends.

Optimization suggestions - See inline comment for details:

• Option A - Early exit without lock (recommended)
• Option B - Use RwLock instead of Mutex
• Option C - Use an AtomicU8 for state (most efficient)

None of these are critical since the monitoring period is short, but Option A would be a simple improvement.

2. Message Dropping During Monitoring ✅

Location: anchor/message_receiver/src/manager.rs:168-180

Messages are dropped during monitoring with a simple early return. This is correct and efficient.

Question: Should there be any metrics or logging for dropped message counts during monitoring? This might help operators understand why duties are being missed during the monitoring window. See inline comment.

---

Testing

Test Coverage ✅

Location: anchor/operator_doppelganger/src/service.rs:244-535

The test suite is comprehensive (300+ lines):

1. State lifecycle - Basic state transitions ✅
2. Grace period - No detection during grace period (prevents false positives) ✅
3. Twin detection - Detects single-signer with our ID ✅
4. Multi-signer filtering - Ignores aggregates ✅
5. Different operator - Ignores other operators ✅
6. Monitoring end - No detection after period ends ✅

Excellent: All tests use #[tokio::test(start_paused = true)] for deterministic async timer testing. This is the correct approach for timer-based tests.

Test quality observations:

• Good use of helper functions (create_test_message, spawn_and_advance_past_grace_period)
• Tests are independent and well-named
• Proper use of tokio::time::advance() for time-based scenarios

Missing test scenarios (non-critical):

1. Operator ID not yet available (sync still in progress) - line 186-188 path
2. Shutdown sender failure scenario
3. Concurrent message checking (multiple threads)

---

Integration & Configuration

1. CLI Configuration ✅

Location: anchor/client/src/cli.rs:488-510

#[clap(
    long,
    default_value_t = true,
    help = "Enable operator doppelgänger protection..."
)]
pub operator_dg: bool,

#[clap(
    long,
    default_value_t = 2,
    requires = "operator_dg"
)]
pub operator_dg_wait_epochs: u64,

Note on requires attribute: @dknopik mentioned this doesn't prevent the option from being listed. If that's correct, then the current implementation is fine. The flag provides helpful context about dependencies.

Suggestion: See inline comment for improved help text that mentions:

• How to disable: --operator-dg=false
• What happens during monitoring (duties are dropped)
• Default grace period (5s)

2. Integration Points ✅

Service Creation: anchor/client/src/lib.rs:500-508

• Created only when enabled and not in impostor mode ✅
• Waits for sync before starting ✅
• Clean separation from other services ✅

Message Pipeline: anchor/message_receiver/src/manager.rs:168-180

• Checks both QBFT and partial signatures ✅
• Drops messages during monitoring ✅
• No processing overhead after monitoring completes ✅

Shutdown: anchor/operator_doppelganger/src/service.rs:228-231

• Uses ShutdownReason::Failure ✅
• Non-blocking with try_send ✅

---

Documentation

1. Code Documentation ✅

Excellent:

• Comprehensive explanation of grace period vulnerability (lines 24-41)
• Clear state transition documentation (lines 12-20)
• Function-level docs explain "why" not just "what"

Good examples:

• spawn_monitor_task explains the grace period rationale
• is_doppelganger clarifies it's detection without side effects
• check_message explains it includes shutdown triggering

2. PR Description ✅

The PR description is well-structured with:

• Clear explanation of core implementation
• Configuration options documented
• Grace period rationale explained
• Test coverage summary

---

Previous Review Concerns - Resolution Status

@dknopik's concerns (2025-10-21):

1. ✅ "needlessly complex" - RESOLVED: Significant simplification completed
2. ✅ "how is committee_id relevant at all?" - RESOLVED: Committee tracking removed entirely
3. ✅ "Why bother with [height tracking]" - RESOLVED: Height tracking removed, now uses simple timer
4. ✅ "why not just sleep until the target epoch starts?" - RESOLVED: Now using single sleep timer
5. ✅ "I do not think we need separate modes" - PARTIALLY: Still has 3 states, but simplified (GracePeriod/Monitoring/Completed seem necessary)
6. ✅ "code needs to cope with not having operator ID yet" - RESOLVED: Line 186-188 handles None gracefully
7. ✅ "requires does not prevent option from being listed" - NOTED: If correct, current implementation is fine

Previous claude-code review concerns (2025-10-21):

1. ✅ Startup crash issue - RESOLVED: Service created but not started until after sync (line 621)
2. ✅ Partial signature checking - RESOLVED: Now checked alongside QBFT messages
3. ✅ Generic removal - RESOLVED: No more unnecessary generics
4. ⚠️ Lock contention - PARTIALLY: Could still optimize with early exit (see Performance section)

---

Recommendations

High Priority: None ✅

All critical issues have been resolved.

Medium Priority:

1. Performance optimization (optional):

   • Add early exit check before acquiring lock in is_doppelganger() (see inline comment)
   • Or use RwLock instead of Mutex

2. Observability (nice-to-have):

   • Add metric for messages dropped during monitoring (see inline comment)
   • Add metric for monitoring state transitions
   • Consider logging when monitoring period ends

3. Documentation polish:

   • Explain the +1 second buffer in grace period calculation (see inline comment)
   • Update CLI help text to mention how to disable and what monitoring does (see inline comment)
   • Add state diagram to module-level docs

Low Priority:

4. Test coverage (optional):

   • Add test for operator ID not yet available scenario
   • Add concurrent message checking test

5. Error handling (nice-to-have):

   • Log if shutdown sender fails (though unlikely to matter)

---

Final Verdict

Status: ✅ Approve with minor suggestions

This is a clean, well-tested implementation that successfully addresses the security concern of preventing operator equivocation. The recent simplification has made the code maintainable and easy to reason about.

Strengths:

• Simple timer-based approach
• Excellent documentation of edge cases
• Comprehensive test coverage
• Proper thread safety
• Clean integration points

Minor improvements:

• Consider lock contention optimization
• Add observability metrics
• Polish documentation

The code is production-ready. The suggested improvements are optimizations and nice-to-haves, not blockers.

Diff Summary:

• New crate: operator_doppelganger/ (3 files, ~540 LOC including tests)
• Integration: Clean additions to client init and message pipeline
• Configuration: Two CLI flags with sensible defaults
• Dependencies: Minimal, appropriate for the functionality

Inline Comments: 4 specific suggestions added to the PR for consideration.

---

[claude-code-actions-sigp]

Documentation suggestion: Consider documenting why +1 second buffer is sufficient.

The calculation is clear, but readers might wonder if 1s accounts for:

• Network propagation delays
• Clock skew between nodes
• Processing time variations

Something like:

// Add 1 second buffer for network delays and clock skew
cache_window_millis / 1000 + 1

This helps future maintainers understand the safety margin.

[claude-code-actions-sigp]

Documentation improvement: Consider enhancing the help text to include:

help = "Enable operator doppelgänger protection. When enabled, the node will monitor \
        for messages signed by its operator ID on startup and shut down if a twin \
        (duplicate operator) is detected. During the monitoring period (~2 epochs + 5s grace), \
        incoming duties will be dropped to ensure safe detection. \
        Use --operator-dg=false to disable. Enabled by default."

This clarifies:

• What happens during monitoring (duties dropped)
• How to disable the feature
• Approximate monitoring duration

Helps operators understand the behavior without reading source code.

[claude-code-actions-sigp]

Observability suggestion: Consider adding metrics or periodic logging for messages dropped during monitoring.

This would help operators understand:

• That monitoring is active
• How many duties are being skipped
• When monitoring period ends

Example:

if service.is_monitoring() {
    // ... existing check_message logic ...
    
    // Optional: track dropped messages
    metrics::inc_counter(&metrics::DOPPELGANGER_MESSAGES_DROPPED);
    return;
}

Not critical, but useful for operational visibility.

[dknopik]

Btw, thinking about it more, doesn't doing it after validation possibly mask detection of some messages? e.g. if we and a doppelganger both send a qbft propose message, validation will only consider our message as valid, as we only accept one propose per round. So I think we only need to make sure that the signature is valid and that the message is fresh for doppelganger protection