kv/kvserver: TestTenantCtx failed

[cockroach-teamcity]

kv/kvserver.TestTenantCtx failed on release-25.4.2-rc @ ff51b7da52b4298ebd4282adccbb5f72f87ffb23:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa345a6]

Stack:

goroutine 2233501 [running]:
database/sql.(*Tx).grabConn(0x0, {0x8d4f6a8, 0xde5bbc0})
	GOROOT/src/database/sql/sql.go:2246 +0x46
database/sql.(*Tx).ExecContext(0x0, {0x8d4f6a8, 0xde5bbc0}, {0x7d12525, 0x1c}, {0xc005a57fb8, 0x1, 0x1})
	GOROOT/src/database/sql/sql.go:2504 +0x4b
database/sql.(*Tx).Exec(...)
	GOROOT/src/database/sql/sql.go:2517
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestTenantCtx.func1.2()
	pkg/kv/kvserver/client_tenant_test.go:491 +0xd8
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestTenantCtx.func1 in goroutine 1869531
	pkg/kv/kvserver/client_tenant_test.go:487 +0x599

Log preceding fatal error

        	            	  | 	pkg/rpc/auth.go:97
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	pkg/rpc/context.go:873
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
        	            	  | 	pkg/rpc/metrics.go:514
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	pkg/rpc/context.go:873
        	            	  | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
        	            	  | 	pkg/rpc/metrics.go:467
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	pkg/rpc/context.go:873
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  | 	pkg/rpc/context.go:139
        	            	  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  | 	pkg/util/stop/stopper.go:351
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  | 	pkg/rpc/context.go:137
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  | 	pkg/rpc/context.go:873
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  | 	pkg/rpc/context.go:772
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func9
        	            	  | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  | 	pkg/rpc/context.go:957
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
        	            	  | 	pkg/rpc/context.go:842
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
        	            	  | 	pkg/rpc/context.go:965
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
        	            	  | 	pkg/kv/kvclient/kvcoord/transport.go:211
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
        	            	  | 	pkg/kv/kvclient/kvcoord/transport.go:189
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
        	            	  | 	pkg/kv/kvclient/kvcoord/dist_sender.go:2768
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
        	            	  | 	pkg/kv/kvclient/kvcoord/dist_sender.go:2227
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
        	            	  | 	pkg/kv/kvclient/kvcoord/dist_sender.go:1775
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
        	            	  | 	pkg/kv/kvclient/kvcoord/dist_sender.go:1280
        	            	Wraps: (2) expected Scan to run as the expected tenant ({10}), but it isn't. tenant request: false, tenantID: {0}
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        	Test:       	TestTenantCtx/shared-process_tenant=true
    client_tenant_test.go:490: 
        	Error Trace:	pkg/kv/kvserver/client_tenant_test.go:490
        	            				src/runtime/asm_amd64.s:1700
        	Error:      	Received unexpected error:
        	            	dial tcp 127.0.0.1:46429: connect: connection reset by peer
        	Test:       	TestTenantCtx/shared-process_tenant=true

Parameters:

• attempt=1
• deadlock=true
• run=1
• shard=45

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-57338