kvcoord: v23.1.20: transaction unexpectedly committed

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5333151409/?referrer=webhooks_plugin

Panic Message:

txn_coord_sender.go:1013: transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.00982825 epo=0 ts=1715386114.889923202,1 min=1715386104.777606487,0 seq=15} lock=true stat=COMMITTED rts=1715386114.889923202,1 wto=false gul=1715386105.277606487,0 int=14 ifw=14: TransactionStatusError: already committed (REASON_TXN_COMMITTED)
(1) assertion failure
Wraps: (2) issue: https://github.com/cockroachdb/cockroach/issues/103817
  | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support.
Wraps: (3)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1013
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:977
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:537
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1036
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1060
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:685
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:700
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1450
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransaction
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1332
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:854
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3140
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2194
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2199
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2116
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:911
  | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:310
  | runtime.goexit
  | 	src/runtime/asm_arm64.s:1172
Wraps: (4) transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.00982825 epo=0 ts=1715386114.889923202,1 min=1715386104.777606487,0 seq=15} lock=true stat=COMMITTED rts=1715386114.889923202,1 wto=false gul=1715386105.277606487,0 int=14 ifw=14
Wraps: (5) TransactionStatusError: already committed (REASON_TXN_COMMITTED)
Error types: (1) *assert.withAssertionFailure (2) *issuelink.withIssueLink (3) *errbase.opaqueWrapper (4) *errutil.withPrefix (5) *kvpb.TransactionStatusError
-- report composition:
*kvpb.TransactionStatusError
*errutil.withPrefix: transaction unexpectedly committed, ba: EndTxn(parallel commit) [/Table/71/9/×/×], [txn: 34160dd6]. txn: "sql txn" meta={id=34160dd6 key=/Table/71/9/×/× pri=0.00982825 epo=0 ts=1715386114.889923202,1 min=1715386104.777606487,0 seq=15} lock=true stat=COMMITTED rts=1715386114.889923202,1 wto=false gul=1715386105.277606487,0 int=14 ifw=14
txn_coord_sender.go:1013: *withstack.withStack (top exception)
*issuelink.withIssueLink: https://github.com/cockroachdb/cockroach/issues/103817
*assert.withAssertionFailure

Stacktrace (expand for inline code snippets):

src/runtime/asm_arm64.s#L1171-L1173

cockroach/pkg/sql/pgwire/conn.go

Lines 309 to 311 in 93a6798

	reservedOwned = false // We're about to pass ownership away.
	retErr = sqlServer.ServeConn(
	ctx,

cockroach/pkg/sql/conn_executor.go

Lines 910 to 912 in 93a6798

	}(ctx, h)
	return h.ex.run(ctx, s.pool, reserved, cancel)
	}

cockroach/pkg/sql/conn_executor.go

Lines 2115 to 2117 in 93a6798

	var err error
	if err = ex.execCmd(); err != nil {
	// Both of these errors are normal ways for the connExecutor to exit.

cockroach/pkg/sql/conn_executor.go

Lines 2198 to 2200 in 93a6798

	return err
	}()
	// Note: we write to ex.statsCollector.PhaseTimes, instead of ex.phaseTimes,

cockroach/pkg/sql/conn_executor.go

Lines 2193 to 2195 in 93a6798

	tcmd.LastInBatch || !implicitTxnForBatch)
	ev, payload, err = ex.execStmt(
	ctx, tcmd.Statement, nil /* portal */, nil /* pinfo */, stmtRes, canAutoCommit,

cockroach/pkg/sql/conn_executor_exec.go

Lines 140 to 142 in 93a6798

	}
	err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3139 to 3141 in 93a6798

	} else {
	err = op(ctx)
	}

cockroach/pkg/sql/conn_executor_exec.go

Lines 141 to 143 in 93a6798

	err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
	return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 853 to 855 in 93a6798

	// CommitTransaction is executed fully here; there's no plan for it.
	ev, payload := ex.commitSQLTransaction(ctx, ast, ex.commitSQLTransactionInternal)
	return ev, payload, nil

cockroach/pkg/sql/conn_executor_exec.go

Lines 1331 to 1333 in 93a6798

	ex.phaseTimes.SetSessionPhaseTime(sessionphase.SessionStartTransactionCommit, timeutil.Now())
	if err := commitFn(ctx); err != nil {
	// For certain retryable errors, we should turn them into client visible

cockroach/pkg/sql/conn_executor_exec.go

Lines 1449 to 1451 in 93a6798

	if err := ex.state.mu.txn.Commit(ctx); err != nil {
	return err

cockroach/pkg/kv/txn.go

Lines 699 to 701 in 93a6798

	return txn.commit(ctx)
	}

cockroach/pkg/kv/txn.go

Lines 684 to 686 in 93a6798

	ba := &kvpb.BatchRequest{Requests: et.unionArr[:]}
	_, pErr := txn.Send(ctx, ba)
	if pErr == nil {

cockroach/pkg/kv/txn.go

Lines 1059 to 1061 in 93a6798

	txn.mu.Unlock()
	br, pErr := txn.db.sendUsingSender(ctx, ba, sender)
	if pErr == nil {

cockroach/pkg/kv/db.go

Lines 1035 to 1037 in 93a6798

	br, pErr := sender.Send(ctx, ba)
	if pErr != nil {

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 536 to 538 in 93a6798

	pErr = tc.updateStateLocked(ctx, ba, br, pErr)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 976 to 978 in 93a6798

	if errTxn := pErr.GetTxn(); errTxn != nil {
	if err := sanityCheckErrWithTxn(ctx, pErr, ba, &tc.testingKnobs); err != nil {
	return kvpb.NewError(err)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 1012 to 1014 in 93a6798

	// are committed.
	err := errors.Wrapf(pErrWithTxn.GoError(),
	"transaction unexpectedly committed, ba: %s. txn: %s",

src/runtime/asm_arm64.s in runtime.goexit at line 1172
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 310
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 911
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2116
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2199
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2194
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3140
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 854
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).commitSQLTransaction at line 1332
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).commitSQLTransactionInternal at line 1450
pkg/kv/txn.go in pkg/kv.(*Txn).Commit at line 700
pkg/kv/txn.go in pkg/kv.(*Txn).commit at line 685
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1060
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1036
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 537
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 977
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 1013

Tags

Tag	Value
Command	server
Environment	v23.1.20
Go Version	go1.19.13
Platform	linux arm64
Distribution	CCL
Cockroach Release	v23.1.20
Cockroach SHA	93a6798
# of CPUs	32
# of Goroutines	14669

Jira issue: CRDB-38634

[yuzefovich]

I'm assuming this was supposed to be fixed via #103817, but apparently not.