Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: panic.go:770: × (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWrite... #140568

Open
cockroach-sentry opened this issue Feb 6, 2025 · 1 comment
Open

Sentry: panic.go:770: × (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWrite... #140568

cockroach-sentry opened this issue Feb 6, 2025 · 1 comment
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-storage Storage Team X-blathers-triaged blathers was able to find an owner

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Feb 6, 2025 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6275062744/?referrer=webhooks_plugin

Panic Message:

panic.go:770: ×
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).destroy
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:1042
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleReadOnly).Close
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:2842
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:240
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:502
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:178
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1517
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1718
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:708
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:170
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:137
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:135
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:718
  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:903
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:788
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:911
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2739
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2187
Wraps: (2) forced error mark
  | ×
  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (3) ×
Error types: (1) *withstack.withStack (2) *markers.withMark (3) *errors.errorString
-- report composition:
*errors.errorString
*markers.withMark
panic.go:770: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2186 to 2188 in 217b43e

prevTok = routingTok
reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2738 to 2740 in 217b43e

} else {
br, err = transport.SendNext(sendCtx, requestToSend)
tEnd := timeutil.Now()

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 191 to 193 in 217b43e

}
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 213 to 215 in 217b43e

log.VEvent(ctx, 2, "sending batch request")
reply, err := iface.Batch(ctx, ba)
log.VEvent(ctx, 2, "received batch response")

https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L910-L912
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L787-L789
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L902-L904

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in 217b43e

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L717-L719
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L818-L820
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L134-L136

cockroach/pkg/util/stop/stopper.go

Lines 336 to 338 in 217b43e

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L136-L138
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L818-L820
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/auth.go#L94-L96
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L818-L820
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L169-L171
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L818-L820

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 217b43e

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/rpc/pkg/rpc/context.go#L707-L709

cockroach/pkg/server/node.go

Lines 1717 to 1719 in 217b43e

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1516 to 1518 in 217b43e

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L177-L179
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L501-L503
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go#L239-L241

cockroach/pkg/storage/pebble.go

Lines 2841 to 2843 in 217b43e

p.iter = nil
p.prefixIter.destroy()
p.normalIter.destroy()

cockroach/pkg/storage/pebble_iterator.go

Lines 1041 to 1043 in 217b43e

}
panic(err)
}

GOROOT/src/runtime/panic.go#L769-L771
https://github.com/cockroachdb/cockroach/blob/217b43e69da61e9712d73e3e19fd5076fbdb26df/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L108-L110
GOROOT/src/runtime/panic.go#L769-L771

pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2187
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2739
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 192
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 214
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 911
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 788
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 903
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 718
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 819
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 135
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 337
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 137
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 819
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 95
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 819
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 170
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 819
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 708
pkg/server/node.go in pkg/server.(*Node).Batch at line 1718
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1517
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 202
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 178
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 502
pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 240
pkg/storage/pebble.go in pkg/storage.(*pebbleReadOnly).Close at line 2842
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).destroy at line 1042
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 109
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770

Tags

Tag Value
Command start-single-node
Environment v24.2.3
Go Version go1.22.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.2.3
Cockroach SHA 217b43e
# of CPUs 8
# of Goroutines 468

Jira issue: CRDB-47234
@cockroach-sentry cockroach-sentry added branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Feb 6, 2025
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Feb 6, 2025

CC'ing via the CODEOWNERS-based sentry heuristic:

  • @cockroachdb/storage

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added T-storage Storage Team X-blathers-triaged blathers was able to find an owner A-storage Relating to our storage engine (Pebble) on-disk storage. labels Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-storage Storage Team X-blathers-triaged blathers was able to find an owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-sentry