Database Error: Missing gold_table relation causing indexing and rewards calculation failures

[nii236]

The sidecar service is failing to start properly due to a missing database table gold_table. This is causing cascading failures across multiple components:

• Block indexer (sidecar/blockIndexer.go:26)
• Rewards calculator (rewards/1_goldActiveRewards.go:129, rewards/rewards.go:766)
• Pipeline processor (pipeline/pipeline.go:372)

I imagine at some point table gold_table got dropped but its hard to track down where because GORM is used and there is a lot of string templating.

Main error message: pq: relation "gold_table" does not exist

Earlier I deleted the DB and restored via snapshot because of the same error, but after a few days it occurred again.

Reproduction Steps

• Start the sidecar service
• Restore DB from snapshot
• Service attempts to begin block indexing
• After some time, database query fails due to missing gold_table
• Subsequent rewards calculations also fail with the same error

Logs

fatal: Failed to index from current to tip
  ts: 2025-05-22T01:36:09.121Z
  caller: sidecar/blockIndexer.go:26
  error: pq: relation "gold_table" does not exist
  stacktrace: github.com/Layr-Labs/sidecar/pkg/sidecar.(*Sidecar).StartIndexing
	/build/pkg/sidecar/blockIndexer.go:26
github.com/Layr-Labs/sidecar/pkg/sidecar.(*Sidecar).Start
	/build/pkg/sidecar/sidecar.go:82
error: Failed to generate active rewards
  ts: 2025-05-22T01:53:15.014Z
  caller: rewards/1_goldActiveRewards.go:129
  error: pq: relation "gold_table" does not exist
  stacktrace: github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).Generate1ActiveRewards
	/build/pkg/rewards/1_goldActiveRewards.go:129
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).generateGoldTables
	/build/pkg/rewards/rewards.go:765
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).calculateRewards
	/build/pkg/rewards/rewards.go:617
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).calculateRewardsForSnapshotDate
	/build/pkg/rewards/rewards.go:158
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).CalculateRewardsForSnapshotDate
	/build/pkg/rewards/rewards.go:162
github.com/Layr-Labs/sidecar/pkg/rewardsCalculatorQueue.(*RewardsCalculatorQueue).processMessage
	/build/pkg/rewardsCalculatorQueue/handler.go:53
github.com/Layr-Labs/sidecar/pkg/rewardsCalculatorQueue.(*RewardsCalculatorQueue).Process
	/build/pkg/rewardsCalculatorQueue/handler.go:17
error: Failed to generate active rewards
  ts: 2025-05-22T01:53:15.016Z
  caller: rewards/rewards.go:766
  error: pq: relation "gold_table" does not exist
  stacktrace: github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).generateGoldTables
	/build/pkg/rewards/rewards.go:766
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).calculateRewards
	/build/pkg/rewards/rewards.go:617
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).calculateRewardsForSnapshotDate
	/build/pkg/rewards/rewards.go:158
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).CalculateRewardsForSnapshotDate
	/build/pkg/rewards/rewards.go:162
github.com/Layr-Labs/sidecar/pkg/rewardsCalculatorQueue.(*RewardsCalculatorQueue).processMessage
	/build/pkg/rewardsCalculatorQueue/handler.go:53
github.com/Layr-Labs/sidecar/pkg/rewardsCalculatorQueue.(*RewardsCalculatorQueue).Process
	/build/pkg/rewardsCalculatorQueue/handler.go:17
error: Failed to generate gold tables
  ts: 2025-05-22T01:53:15.025Z
  caller: rewards/rewards.go:619
  error: pq: relation "gold_table" does not exist
  stacktrace: github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).calculateRewards
	/build/pkg/rewards/rewards.go:619
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).calculateRewardsForSnapshotDate
	/build/pkg/rewards/rewards.go:158
github.com/Layr-Labs/sidecar/pkg/rewards.(*RewardsCalculator).CalculateRewardsForSnapshotDate
	/build/pkg/rewards/rewards.go:162
github.com/Layr-Labs/sidecar/pkg/rewardsCalculatorQueue.(*RewardsCalculatorQueue).processMessage
	/build/pkg/rewardsCalculatorQueue/handler.go:53
github.com/Layr-Labs/sidecar/pkg/rewardsCalculatorQueue.(*RewardsCalculatorQueue).Process
	/build/pkg/rewardsCalculatorQueue/handler.go:17
error: Failed to calculate rewards for snapshot date
  ts: 2025-05-22T01:53:15.027Z
  caller: pipeline/pipeline.go:372
  cutoffDate: 2025-05-19
  error: pq: relation "gold_table" does not exist
  blockNumber: 22525930
  distributionRoot: {"Root":"0xdb7376270532042873ed4c2736bd493ef1bfa1a2eddd624df8312a86673f52b6","BlockNumber":22525930,"RootIndex":45,"RewardsCalculationEnd":"2025-05-18T00:00:00Z","RewardsCalculationEndUnit":"snapshot","ActivatedAt":"2025-05-27T18:32:11Z","ActivatedAtUnit":"timestamp","CreatedAtBlockNumber":22525930,"LogIndex":808,"TransactionHash":"0xc33676fc5d7a5ebf573d8b6124fefa985578271af48eaa5e49a02d918faf207e"}
  stacktrace: github.com/Layr-Labs/sidecar/pkg/pipeline.(*Pipeline).RunForFetchedBlock
	/build/pkg/pipeline/pipeline.go:372
github.com/Layr-Labs/sidecar/pkg/pipeline.(*Pipeline).RunForBlockBatch
	/build/pkg/pipeline/pipeline.go:519
github.com/Layr-Labs/sidecar/pkg/sidecar.(*Sidecar).IndexFromCurrentToTip
	/build/pkg/sidecar/blockIndexer.go:292
github.com/Layr-Labs/sidecar/pkg/sidecar.(*Sidecar).StartIndexing
	/build/pkg/sidecar/blockIndexer.go:25
github.com/Layr-Labs/sidecar/pkg/sidecar.(*Sidecar).Start
	/build/pkg/sidecar/sidecar.go:82

[seanmcgary]

Hi @nii236 - I combed through the Sidecar code just to double check this and we don't have any queries or codepaths that would drop that table. Over our 4+ Sidecar instances running Ive not seen this happen either. Im currently running through your reproduction steps to see what I can find (just wanted to follow up and let you know that we're working on it!)

[nii236]

Thanks @seanmcgary! Yeah I had a look through it too, but couldn't find anything. Its not a migration thing, it happens during runtime.

I know rewards tables get dropped (gold_{timestamp}) but not gold_table itself.

The only place I can see it happening is if the wrong arg is passed through at https://github.com/Layr-Labs/sidecar/blob/master/pkg/rewards/rewards.go#L426, but I can't imagine how that would happen.

Anyway Its quite hard to replicate (basically just let it run for a week), so next time it happens I'll paste some more extensive logs.

[seanmcgary]

@nii236 is this the exact procedure you're following?

Start the sidecar service
Restore DB from snapshot

If so that could be the source of your problems. You would need to run sidecar restore-snapshot first without the sidecar actually running and then start the sidecar once the snapshot restore process completes.

[nii236]

That could be it! Sadly I can't remember exactly how I did it exactly. I'll leave this issue up for a few more days or so to see if it happens again, if not then I'll close it.

[nii236]

Hey @seanmcgary it happened again last night! Logs attached.

logs_2025_05_27.txt