Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestOptimisticEvalNoContention failed #123986

Open
cockroach-teamcity opened this issue May 11, 2024 · 1 comment
Open

kv/kvserver: TestOptimisticEvalNoContention failed #123986

cockroach-teamcity opened this issue May 11, 2024 · 1 comment
Assignees
@nvanbenschoten
Labels
A-testing Testing tools and infrastructure branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
KV
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 11, 2024 •
edited by cockroach-jira-scripts

kv/kvserver.TestOptimisticEvalNoContention failed on release-24.1.0-rc @ 6205244e922606f85761dad2137b842f43a53716:

Fatal error:

panic: test timed out after 1h16m37s
running tests:
	TestOptimisticEvalNoContention (1h16m8s)

Stack:

goroutine 98483 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

=== RUN   TestOptimisticEvalNoContention
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestOptimisticEvalNoContention245023092
    test_log_scope.go:81: use -show-logs to present logs inline
    client_replica_test.go:4982: <nil>

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=23
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-38643
@cockroach-teamcity cockroach-teamcity added branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels May 11, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 11, 2024
@cockroach-teamcity cockroach-teamcity added this to roachtest/unit test backlog in KV May 11, 2024
@nvanbenschoten
Copy link
Member

The test's two goroutines are stuck here:

goroutine 13449 [chan receive, 76 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestOptimisticEvalNoContention(0xc0075b41a0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:5368 +0x413

and

goroutine 15284 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).waitForPush(0xc001141e30, {0xe447258, 0xc007204db0}, 0xc004c383c0, 0xc007204cf0, 0xc0002506e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:615 +0xfc9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush.func2({0xe447258, 0xc007204db0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:561 +0x8e
runtime/pprof.Do({0xe447258, 0xc007204c00}, {{0xc004e5c500?, 0x24?, 0x24?}}, 0xc006603468)
	GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush(0xc001141e30, {0xe447258, 0xc007204c00}, 0xc004c383c0, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:560 +0x166a
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc005005bd0, {0xe447258, 0xc007204c00}, {0x0, {0x17ce6a549611181f, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:372 +0xfd
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc005005bd0, {0xe447258, 0xc007204c00}, 0xc005a31450, 0x3)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:272 +0x210
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc005005bd0, {0xe447258, 0xc007204c00}, 0xc005a31450, {0x0, {0x17ce6a549611181f, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x332
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0013c7b08, {0xe447258, 0xc007204c00}, 0xc008972240, 0xa9c1ed0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x517
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0013c7b08, {0xe447258, 0xc007204bd0}, 0xc008972240)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:186 +0x5d0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc008818408, {0xe447258, 0xc007204ba0}, 0xc008972240)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd12
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc0047f61b0, {0xe447258, 0xc007204ba0}, 0xc008972240)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0xdf
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc008dbf408, {0xe447258, 0xc007204b40}, {0xc00bbdd4a0?}, 0xc008972240)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1437 +0x931
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc008dbf408, {0xe447258, 0xc007204ae0}, 0xc008972240)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1637 +0x4a5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240}, 0xc004592240, 0xc007da5f38)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x72c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240}, 0xc004592240, 0xc004592260)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf1
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc00252a000?, {{0xc0086e9ba8?}, {0xe492620?, 0xc004671c00?}}}, {0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240}, 0xc004592240, 0xc004592280)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x2b7
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xe447258, 0xc007204ae0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0086e9b80, {0xe447258, 0xc007204ae0}, {0x0?, 0x0?}, 0xc005f286a8)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0x186
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240}, 0xc004592240, 0xc0045922a0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x151
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204ae0}, {0xa4aed40, 0xc008972240})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0xe447258, 0xc007204ae0}, {0x4781fd?, 0xc005e0d000?}, {0xa4aed40, 0xc008972240}, {0xa328720, 0xc0017d0400}, 0xc005f28848?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0xe447258, 0xc007204ae0}, {0xa603700, 0x21}, {0xa4aed40, 0xc008972240}, {0xa328720, 0xc0017d0400}, 0x0, 0xc00b36df50, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x5ee
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0xe447258, 0xc007204ae0}, {0xa603700, 0x21}, {0xa4aed40, 0xc008972240}, {0xa328720, 0xc0017d0400}, 0x0, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x19a
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xe447258, 0xc007204ab0}, 0xc008972120, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x463
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc007c19c80, {0xe447258, 0xc007204ab0}, 0x9d0d3a0?, {0xe427820, 0xc0072bd920}, 0xc008972120)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214 +0x283
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc007c19c80, {0xe447258, 0xc007204ab0}, 0xc008972120)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192 +0x13a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xe494048?, 0xc007c19c80?}}, {0xe447258, 0xc007204ab0}, 0xc008972120)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x2ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0038ab208, {0xe447258, 0xc007204ab0}, 0xc008972000, {0xc00cb679e0, 0xc002220680}, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2750 +0x36c9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0038ab208, {0xe447258, 0xc007204ab0}, 0xc008972000, {{0xc0086f1288, 0x1, 0x8}, {0xc0086f1288, 0x2, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2091 +0x9c5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0038ab208, {0xe447258, 0xc007204ab0}, 0xc008972000, {{0xc0086f1288, 0x1, 0x8}, {0xc0086f1288, 0x2, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1616 +0x1031
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0038ab208, {0xe447220, 0xc0002505a0}, 0xc008972000)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1232 +0xa3a
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc004c9b788, {0xe447220, 0xc0002505a0}, 0xc008972000)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:229 +0xf3
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc004c9b720, {0xe447220, 0xc0002505a0}, 0xc008972000, {0xe3f3f20, 0xc004c9b788})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1124 +0x182
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1107
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0xe447220, 0xc0002505a0}, 0xc005f2aed8, 0xc004c5a008)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:956 +0x27e
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc004c9b720, {0xe447220, 0xc0002505a0}, 0xc004c5a008)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:979 +0xa8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions(0xc0049f2500, {_, _}, _, {{0x17ce6a549610708b, 0x0}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:447 +0xa54
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction(0xc0049f2500, {_, _}, _, {{0x17ce6a549610708b, 0x0}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:346 +0x1a5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc005005c20, {0xe447220, _}, {0xc004eb70e0, {0x17ce6a549003ac8f, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:488 +0x77c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn.func3({0xe447258, 0xc007204900})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:363 +0x365
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc005005c20, {0xe447258, 0xc007204900}, {0xc004eb70e0, {0x17ce6a549003ac8f, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:409 +0x7bb
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc005005bd0, {0xe447258, 0xc007204900}, 0xc004ee32b0, 0x4)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:353 +0x10ba
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc005005bd0, {0xe447258, 0xc007204900}, 0xc004ee32b0, {0xc004eb70e0, {0x17ce6a549003ac8f, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x332
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0013c7b08, {0xe447258, 0xc007204900}, 0xc004eb7320, 0xa9c1ec8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x517
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0013c7b08, {0xe447258, 0xc0072048d0}, 0xc004eb7320)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x565
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc008818408, {0xe447258, 0xc0072048a0}, 0xc004eb7320)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd12
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc0047f61b0, {0xe447258, 0xc0072048a0}, 0xc004eb7320)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0xdf
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc008dbf408, {0xe447258, 0xc007204870}, {0xc00bbdd4a0?}, 0xc004eb7320)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1437 +0x931
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc008dbf408, {0xe447258, 0xc007204810}, 0xc004eb7320)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1637 +0x4a5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320}, 0xc004592240, 0xc007da5f38)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x72c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320}, 0xc004592240, 0xc004592260)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf1
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc00252a000?, {{0xc0086e9ba8?}, {0xe492620?, 0xc004671c00?}}}, {0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320}, 0xc004592240, 0xc004592280)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x2b7
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xe447258, 0xc007204810})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0086e9b80, {0xe447258, 0xc007204810}, {0xc008818408?, 0xc00b46f440?}, 0xc005bae228)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0x186
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320}, 0xc004592240, 0xc0045922a0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x151
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe447258, 0xc007204810}, {0xa4aed40, 0xc004eb7320})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0xe447258, 0xc007204810}, {0xc002e0652c?, 0x15?}, {0xa4aed40, 0xc004eb7320}, {0xa328720, 0xc005082800}, 0xc005bae3c8?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0xe447258, 0xc007204810}, {0xa603700, 0x21}, {0xa4aed40, 0xc004eb7320}, {0xa328720, 0xc005082800}, 0x0, 0xc00b36df50, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x5ee
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0xe447258, 0xc007204810}, {0xa603700, 0x21}, {0xa4aed40, 0xc004eb7320}, {0xa328720, 0xc005082800}, 0x0, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x19a
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xe447258, 0xc0072047e0}, 0xc004eb7200, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x463
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc007c19440, {0xe447258, 0xc0072047e0}, 0x1d067c8?, {0xe427820, 0xc0072bd920}, 0xc004eb7200)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214 +0x283
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc007c19440, {0xe447258, 0xc0072047e0}, 0xc004eb7200)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192 +0x13a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xe494048?, 0xc007c19440?}}, {0xe447258, 0xc0072047e0}, 0xc004eb7200)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x2ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0038ab208, {0xe447258, 0xc0072047e0}, 0xc004eb6ea0, {0xc00cb679e0, 0xc002220680}, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2750 +0x36c9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0038ab208, {0xe447258, 0xc0072047e0}, 0xc004eb6ea0, {{0xc0086f1588, 0x1, 0x8}, {0xc0086f16b8, 0x1, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2091 +0x9c5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0038ab208, {0xe447258, 0xc0072047e0}, 0xc004eb6ea0, {{0xc0086f1588, 0x1, 0x8}, {0xc0086f16b8, 0x1, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1616 +0x1031
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0038ab208, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1232 +0xa3a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc004c57170, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x26c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0xc004c57120, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0x1da
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc004c57040, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0, 0x5)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:239 +0x273
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc004c57040, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:167 +0x1d9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0xc004c57000, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:149 +0x95f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc004c56eb0, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:326 +0x245
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc004c56e90, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:117 +0x3d9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0xc004c56de8, {0xe447258, 0xc0072047b0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:246 +0x7bc
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc004c56c08, {0xe4469d8, 0x13d123c0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:536 +0xa77
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc004c9b720, {0xe4469d8, 0x13d123c0}, 0xc004eb6ea0, {0x7f1ad340dca8, 0xc004c56c08})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1124 +0x182
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc003eff290, {0xe4469d8, 0x13d123c0}, 0xc004eb6ea0)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1282 +0x37d
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0xe4469d8, 0x13d123c0}, 0xc0057ff8d0, 0xc004c57208)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:956 +0x27e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run(0xc003eff290, {0xe4469d8, 0x13d123c0}, 0xc004c57208)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:800 +0xa8
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).scan(0xc003eff290, {0xe4469d8, 0x13d123c0}, {0x98a7f60, 0xe3da740}, {0x98a7f60, 0xe3da750}, 0x1, 0x0, 0x1, ...)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:646 +0x218
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Scan(...)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:660
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestOptimisticEvalNoContention.func1.1({0xe4469d8, 0x13d123c0}, 0xc003eff290)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:5361 +0x125
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc003eff290, {0xe4469d8, 0x13d123c0}, 0xc006611fb0)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1049 +0xae
github.com/cockroachdb/cockroach/pkg/kv.runTxn({0xe4469d8, 0x13d123c0}, 0xc003eff290, 0xc0057fffb0)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1089 +0x6c
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0xc004c9b720, {0xe4469d8, 0x13d123c0}, 0x0, 0x0, 0x0, 0xc0057fffb0)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1052 +0xed
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1027
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestOptimisticEvalNoContention.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:5355 +0x99
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestOptimisticEvalNoContention in goroutine 13449
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:5354 +0x3f5

It looks like optimistic evaluation failed for some reason, and now the txn.Scan is blocked on txn1's lock.

@nvanbenschoten nvanbenschoten added A-testing Testing tools and infrastructure P-3 Issues/test failures with no fix SLA C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels May 13, 2024
@nvanbenschoten nvanbenschoten self-assigned this May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
A-testing Testing tools and infrastructure branch-release-24.1.0-rc Used to mark GA and release blockers and technical advisories for 24.1.0-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
KV
roachtest/unit test backlog
Milestone
24.1
Development

No branches or pull requests
2 participants
@nvanbenschoten @cockroach-teamcity