Assertoor error msgs during execution

[rodrigo-o]

This is not impacting the execution of assertoor tests right now, probably just a missing endpoint that autoresolves through other nodes, but still something to have a look in the future:

[assertoor] time="2025-01-31T20:26:51Z" level=info msg="received cl block 19 [0x69f9b64d2c50e2e251e8ed3f9628d8bd1e2e42dc96bf1cc8158e0b285b675b10] streamed" client=3-geth-lambda module=consensus
[assertoor] time="2025-01-31T20:26:51Z" level=warning msg="failed processing block event: could not process block: GET failed with status 404: {\"code\":404,\"message\":\"Resource not found\"}" client=3-geth-lambda module=consensus```

These errors appear during assertoor run in the block_proposal check, here is an example: https://github.com/lambdaclass/lambda_ethereum_consensus/actions/runs/13080097954/job/36501495170?pr=1324

[rodrigo-o]

There is another error in syncing, that stopped assertoor on this run: https://github.com/lambdaclass/lambda_ethereum_consensus/actions/runs/13118565938/attempts/1?pr=1324

This is needed to be reproduced locally to be able to tell what's happening in our node but it relates to the current implementation of the syncing endpoint:

[assertoor] time="2025-02-03T17:00:32Z" level=info msg="received cl block 2 [0x7c517e0e4b707fe8f7a7176c053190e332df061e52408e3a3639d1b327eb079c] streamed" client=1-geth-lighthouse module=consensus
[assertoor] time="2025-02-03T17:00:32Z" level=info msg="Check result: false, Failed Clients: [2-geth-lighthouse 1-geth-lighthouse 3-geth-lambda]" RunID=1 TestID=cl-stability-check module=test task=check_consensus_sync_status taskidx=5
[assertoor] time="2025-02-03T17:00:37Z" level=info msg="Check result: false, Failed Clients: [1-geth-lighthouse 2-geth-lighthouse 3-geth-lambda]" RunID=1 TestID=cl-stability-check module=test task=check_execution_sync_status taskidx=6
[assertoor] time="2025-02-03T17:00:42Z" level=info msg="Check result: false, Failed Clients: [1-geth-lighthouse 2-geth-lighthouse 3-geth-lambda]" RunID=1 TestID=cl-stability-check module=test task=check_execution_sync_status taskidx=6
[assertoor] time="2025-02-03T17:00:42Z" level=warning msg="error fetching sync status: GET failed with status 500: {\"message\":\"There has been an error\"}" RunID=1 TestID=cl-stability-check client=3-geth-lambda module=test task=check_consensus_sync_status taskidx=5
[assertoor] time="2025-02-03T17:00:42Z" level=error msg="task execution panic: runtime error: invalid memory address or nil pointer dereference, stack: goroutine 197 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.22.10/x64/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/ethpandaops/assertoor/pkg/coordinator/scheduler.(*TaskScheduler).ExecuteTask.func6()\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/scheduler/task_execution.go:143 +0x99\npanic({0x12db980?, 0x225d550?})\n\t/opt/hostedtoolcache/go/1.22.10/x64/src/runtime/panic.go:770 +0x132\ngithub.com/ethpandaops/assertoor/pkg/coordinator/tasks/check_consensus_sync_status.(*Task).getClientInfo(...)\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/tasks/check_consensus_sync_status/task.go:195\ngithub.com/ethpandaops/assertoor/pkg/coordinator/tasks/check_consensus_sync_status.(*Task).processCheck(0xc001b4b9d0, {0x19c9e90, 0xc001435b30})\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/tasks/check_consensus_sync_status/task.go:124 +0x3fb\ngithub.com/ethpandaops/assertoor/pkg/coordinator/tasks/check_consensus_sync_status.(*Task).Execute(0xc001b4b9d0, {0x19c9e90, 0xc001435b30})\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/tasks/check_consensus_sync_status/task.go:90 +0x45\ngithub.com/ethpandaops/assertoor/pkg/coordinator/scheduler.(*TaskScheduler).ExecuteTask(0xc0016515f0, {0x19c9e90, 0xc0014357c0}, 0xc001b46d60?, 0xc001b46f60)\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/scheduler/task_execution.go:156 +0xa85\ngithub.com/ethpandaops/assertoor/pkg/coordinator/tasks/run_tasks_concurrent.(*Task).Execute.func1(0x0)\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/tasks/run_tasks_concurrent/task.go:139 +0x15c\ncreated by github.com/ethpandaops/assertoor/pkg/coordinator/tasks/run_tasks_concurrent.(*Task).Execute in goroutine 92\n\t/home/runner/work/assertoor/assertoor/pkg/coordinator/tasks/run_tasks_concurrent/task.go:131 +0xe5\n" RunID=1 TestID=cl-stability-check module=test task=check_consensus_sync_status taskidx=5
[assertoor] time="2025-02-03T17:00:42Z" level=info msg="failure limit reached (0 success, 1 failure)" RunID=1 TestID=cl-stability-check module=test task=run_tasks_concurrent taskidx=2
[assertoor] time="2025-02-03T17:00:42Z" level=info msg="task completed" RunID=1 TestID=cl-stability-check module=test task=check_execution_sync_status taskidx=6
[assertoor] time="2025-02-03T17:00:42Z" level=warning msg="task failed with failure result: <nil>" RunID=1 TestID=cl-stability-check module=test task=run_tasks_concurrent taskidx=2
[assertoor] time="2025-02-03T17:00:42Z" level=info msg="test failed!" RunID=1 TestID=cl-stability-check module=test
[assertoor] time="2025-02-03T17:00:42Z" level=error msg="test execution failed: task failed: %!w(<nil>)" RunID=1 TestID=cl-stability-check module=test
2025-02-03 17:00:44  Test Status:
  cl-stability-check               failure 	Check Execution Stability
Run tempdir="/home/runner/work/_temp/elixir-consensus-assertoor"
INFO[2025-02-03T17:00:49Z] Dumped enclave 'elixir-consensus-assertoor' to directory 'elixir-consensus-assertoor--2c694c2c5b3844cab85d3a88bb1e92eb' 

And here is another one:

[assertoor] time="2025-02-03T21:05:11Z" level=info msg="received cl block 39 [0x990810e870caf84f6da58ce549e021b96b8e8ccb5504173d609616551011d27b] streamed" client=3-geth-lambda module=consensus
[assertoor] time="2025-02-03T21:05:11Z" level=warning msg="failed processing block event: could not process block: GET failed with status 404: {\"code\":404,\"message\":\"Resource not found\"}" client=3-geth-lambda module=consensus
[assertoor] time="2025-02-03T21:05:11Z" level=info msg="received el block 39 [0x6c9cae91ab7ec77f962c15d283a41ece398a0cd43d28a2f3128872de6171536b] notified" client=2-geth-lighthouse module=execution
[assertoor] time="2025-02-03T21:05:11Z" level=info msg="processing block 39 with 340 transactions" module=wallet
[assertoor] time="2025-02-03T21:05:23Z" level=info msg="received cl block 40 [0x48606ec91efc3deabd83d7014b8b6c19c9efc5f610f94e03ac5aec814d6ddf13] streamed" client=3-geth-lambda module=consensus
[assertoor] time="2025-02-03T21:05:23Z" level=warning msg="failed processing block event: could not process block: GET failed with status 404: {\"code\":404,\"message\":\"Resource not found\"}" client=3-geth-lambda module=consensus