Bug: likely deadlock detected, could not get lock after 250ms

[petuhovskiy]

Environment

prod, staging

Subject

Found this panic in the logs of autoscaler-agent:

likely deadlock detected, could not get lock after 250ms

goroutine 529 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/neondatabase/autoscaling/pkg/agent.(*Runner).spawnBackgroundWorker.func1.1()
	/workspace/pkg/agent/runner.go:450 +0x1b5
panic({0x1d23280?, 0xc00c39a000?})
	/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/neondatabase/autoscaling/pkg/agent.(*Runner).Run.(*ChanMutex).DeadlockChecker.func12({0x2380550, 0xc0067a6050})
	/workspace/pkg/util/chanmutex.go:102 +0x1e5
github.com/neondatabase/autoscaling/pkg/agent.(*Runner).Run.ignoreLogger.func13({0x2380550?, 0xc0067a6050?}, 0x0?)
	/workspace/pkg/agent/runner.go:415 +0x22
github.com/neondatabase/autoscaling/pkg/agent.(*Runner).spawnBackgroundWorker.func1()
	/workspace/pkg/agent/runner.go:464 +0xcf
created by github.com/neondatabase/autoscaling/pkg/agent.(*Runner).spawnBackgroundWorker in goroutine 409
	/workspace/pkg/agent/runner.go:430 +0x1c5

Not clear if we waiting for something slow or really have a deadlock there, would be nice to investigate and fix.

Other logs, links

Similar errors in the last 1h: https://neonprod.grafana.net/goto/b_EZNeOHg?orgId=1