Using rd_kafka_topic_conf_set with acks set to 1 on a transactional producer crashes it

[mensfeld]

I'm working on implementing support for per-topic configuration properties for the producer (https://karafka.io/docs/Librdkafka-Configuration/#topic-configuration-properties) to the rdkafka Ruby bindings: karafka/rdkafka-ruby#449

All works well for non-transactional producers, and almost everything works as expected for transactional. Except one thing. Using a custom set of settings via rd_kafka_topic_conf_new + rd_kafka_topic_conf_set works until the producer is transactional and acks is provided with value different than all, causing a crash.

This makes sense as transactional producer requires idempotence and when trying to create it without correct acks I will get the expected:

`acks` must be set to `all` when `enable.idempotence`

However, when I provide acks via rd_kafka_topic_conf_set and then pass the reference from rd_kafka_topic_conf_new to rd_kafka_topic_new and then the topic reference via:

:int, Rdkafka::Bindings::RD_KAFKA_VTYPE_RKT, :pointer, topic_ref,

I get the following crash (part of trace removed for readability):

gems/3.3.0/gems/ffi-1.16.3/lib/ffi/variadic.rb:47: [BUG] Segmentation fault at 0x0000000000000000

-- Control frame information -----------------------------------------------
c:0074 p:---- s:0430 e:000429 CFUNC  :invoke
c:0073 p:0099 s:0424 e:000423 METHOD gems/3.3.0/gems/ffi-1.16.3/lib/ffi/variadic.rb:47
c:0072 p:0013 s:0415 e:000414 METHOD gems/3.3.0/gems/ffi-1.16.3/lib/ffi/variadic.rb:62
c:0071 p:0010 s:0410 e:000409 BLOCK  /rdkafka/lib/rdkafka/producer.rb:404
c:0070 p:0048 s:0406 e:000405 METHOD /rdkafka/lib/rdkafka/native_kafka.rb:77
c:0069 p:0379 s:0402 e:000401 METHOD /rdkafka/lib/rdkafka/producer.rb:403
c:0068 p:0007 s:0378 e:000377 BLOCK  /waterdrop/lib/waterdrop/producer.rb:296
c:0067 p:0002 s:0375 e:000374 BLOCK  /waterdrop/lib/waterdrop/producer/transactions.rb:66 [FINISH]
c:0066 p:---- s:0372 e:000371 CFUNC  :catch
c:0065 p:0021 s:0367 e:000366 BLOCK  /waterdrop/lib/waterdrop/producer/transactions.rb:65
c:0064 p:0007 s:0361 e:000360 METHOD gems/3.3.0/gems/karafka-core-2.4.0/lib/karafka/core/monitoring/notifications.rb:122
c:0063 p:0038 s:0355 e:000354 METHOD gems/3.3.0/gems/karafka-core-2.4.0/lib/karafka/core/monitoring/notifications.rb:98
c:0062 p:0057 s:0345 e:000344 METHOD gems/3.3.0/gems/karafka-core-2.4.0/lib/karafka/core/monitoring/monitor.rb:34
c:0061 p:0030 s:0337 e:000336 METHOD /waterdrop/lib/waterdrop/producer/transactions.rb:163
c:0060 p:0007 s:0330 e:000329 BLOCK  /waterdrop/lib/waterdrop/producer/transactions.rb:57 [FINISH]
c:0059 p:---- s:0327 e:000326 CFUNC  :synchronize
c:0058 p:0020 s:0323 e:000322 METHOD /waterdrop/lib/waterdrop/producer/transactions.rb:56
c:0057 p:0105 s:0319 e:000318 METHOD /waterdrop/lib/waterdrop/producer.rb:296
c:0056 p:0006 s:0311 e:000309 BLOCK  /waterdrop/lib/waterdrop/producer/sync.rb:27
c:0055 p:0007 s:0307 e:000306 METHOD gems/3.3.0/gems/karafka-core-2.4.0/lib/karafka/core/monitoring/notifications.rb:122
c:0054 p:0038 s:0301 e:000300 METHOD gems/3.3.0/gems/karafka-core-2.4.0/lib/karafka/core/monitoring/notifications.rb:98
c:0053 p:0057 s:0291 e:000290 METHOD gems/3.3.0/gems/karafka-core-2.4.0/lib/karafka/core/monitoring/monitor.rb:34
c:0052 p:0028 s:0283 e:000282 METHOD /waterdrop/lib/waterdrop/producer/sync.rb:22
c:0051 p:0026 s:0276 e:000275 METHOD /waterdrop/lib/waterdrop/producer/context.rb:81
c:0050 p:0011 s:0270 e:000269 BLOCK  /waterdrop/spec/lib/waterdrop/producer/context_spec.rb:75 [FINISH]
c:0049 p:---- s:0267 e:000266 CFUNC  :instance_exec
(...)
c:0002 p:0081 s:0008 E:000f20 EVAL   /3.3.1/bin/bundle:25 [FINISH]
c:0001 p:0000 s:0003 E:0021c0 DUMMY  [FINISH]

-- Ruby level backtrace information ----------------------------------------
/3.3.1/bin/bundle:25:in `<main>'
/3.3.1/bin/bundle:25:in `load'
gems/3.3.0/gems/bundler-2.5.9/exe/bundle:20:in `<top (required)>'
3.3.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
gems/3.3.0/gems/bundler-2.5.9/exe/bundle:28:in `block in <top (required)>'
3.3.0/bundler/cli.rb:28:in `start'
3.3.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
3.3.0/bundler/cli.rb:34:in `dispatch'
3.3.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
3.3.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
3.3.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
3.3.0/bundler/cli.rb:451:in `exec'
3.3.0/bundler/cli/exec.rb:23:in `run'
3.3.0/bundler/cli/exec.rb:58:in `kernel_load'
3.3.0/bundler/cli/exec.rb:58:in `load'
/3.3.1/bin/rspec:25:in `<top (required)>'
/3.3.1/bin/rspec:25:in `load'
gems/3.3.0/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'
gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'
gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'
gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'
(...)
/waterdrop/lib/waterdrop/producer.rb:296:in `block in produce'
/rdkafka/lib/rdkafka/producer.rb:403:in `produce'
/rdkafka/lib/rdkafka/native_kafka.rb:77:in `with_inner'
/rdkafka/lib/rdkafka/producer.rb:404:in `block in produce'
gems/3.3.0/gems/ffi-1.16.3/lib/ffi/variadic.rb:62:in `rd_kafka_producev'
gems/3.3.0/gems/ffi-1.16.3/lib/ffi/variadic.rb:47:in `call'
gems/3.3.0/gems/ffi-1.16.3/lib/ffi/variadic.rb:47:in `invoke'

-- Threading information ---------------------------------------------------
Total ractor count: 1
Ruby thread count for this ractor: 4

-- Machine register context ------------------------------------------------
 RIP: 0x00007f119c89ff24 RBP: 0x000055c9c70a7840 RSP: 0x00007fff36c526d8
 RAX: 0x0000000000000018 RBX: 0x0000000000000000 RCX: 0x0000000000000001
 RDX: 0x00007fff36c527e0 RDI: 0x0000000000000000 RSI: 0x0000000000000002
  R8: 0x0000000000000002  R9: 0x0000000000000004 R10: 0x0000000000000000
 R11: 0x00007f119c899520 R12: 0x0000000000000000 R13: 0x0000000000000000
 R14: 0x00007f119ca1306c R15: 0x00007f119cb3e110 EFL: 0x0000000000010202

-- C level backtrace information -------------------------------------------
/3.3.1/lib/libruby.so.3.3(rb_print_backtrace+0x14) [0x7f11b8bc74c1] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_dump.c:820
/3.3.1/lib/libruby.so.3.3(rb_vm_bugreport) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_dump.c:1151
/3.3.1/lib/libruby.so.3.3(rb_bug_for_fatal_signal+0x104) [0x7f11b89be314] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/error.c:1065
/3.3.1/lib/libruby.so.3.3(sigsegv+0x4d) [0x7f11b8b12a9d] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/signal.c:926
/lib/x86_64-linux-gnu/libc.so.6(0x7f11b84d7520) [0x7f11b84d7520]
/rdkafka/lib/rdkafka/../../ext/librdkafka.so(rd_kafka_rkt_is_lw+0x0) [0x7f119c89ff24] /rdkafka/ext/tmp/x86_64-linux-gnu/ports/librdkafka/2.3.0/librdkafka-2.3.0/src/rdkafka_topic.c:252
/rdkafka/lib/rdkafka/../../ext/librdkafka.so(rd_kafka_rkt_get_lw) /rdkafka/ext/tmp/x86_64-linux-gnu/ports/librdkafka/2.3.0/librdkafka-2.3.0/src/rdkafka_topic.h:71
/rdkafka/lib/rdkafka/../../ext/librdkafka.so(rd_kafka_topic_proper) /rdkafka/ext/tmp/x86_64-linux-gnu/ports/librdkafka/2.3.0/librdkafka-2.3.0/src/rdkafka_topic.c:255
/rdkafka/lib/rdkafka/../../ext/librdkafka.so(rd_kafka_topic_keep+0x0) [0x7f119c899893] /rdkafka/ext/tmp/x86_64-linux-gnu/ports/librdkafka/2.3.0/librdkafka-2.3.0/src/rdkafka_msg.c:557
/rdkafka/lib/rdkafka/../../ext/librdkafka.so(rd_kafka_producev) /rdkafka/ext/tmp/x86_64-linux-gnu/ports/librdkafka/2.3.0/librdkafka-2.3.0/src/rdkafka_msg.c:559
/lib/x86_64-linux-gnu/libffi.so.8(0x7f119cb3ce2e) [0x7f119cb3ce2e]
/lib/x86_64-linux-gnu/libffi.so.8(0x7f119cb39493) [0x7f119cb39493]
gems/3.3.0/gems/ffi-1.16.3/lib/ffi_c.so(call_blocking_function+0x1d) [0x7f119cb7748d] gems/3.3.0/gems/ffi-1.16.3/ext/ffi_c/Call.c:336
/3.3.1/lib/libruby.so.3.3(rb_nogvl+0xc6) [0x7f11b8b605a6] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/thread.c:1530
gems/3.3.0/gems/ffi-1.16.3/lib/ffi_c.so(rbffi_do_blocking_call+0x20) [0x7f119cb77460] gems/3.3.0/gems/ffi-1.16.3/ext/ffi_c/Call.c:344
/3.3.1/lib/libruby.so.3.3(rb_vrescue2+0x11e) [0x7f11b89c789e] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/eval.c:919
/3.3.1/lib/libruby.so.3.3(rb_rescue2+0x8e) [0x7f11b89c7b2e] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/eval.c:900
gems/3.3.0/gems/ffi-1.16.3/lib/ffi_c.so(variadic_invoke+0x8d9) [0x7f119cb83f89] gems/3.3.0/gems/ffi-1.16.3/ext/ffi_c/Variadic.c:308
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(invoke_block_from_c_bh+0x113) [0x7f11b8bc0253] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1509
/3.3.1/lib/libruby.so.3.3(vm_yield_with_cref) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(vm_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1642
/3.3.1/lib/libruby.so.3.3(rb_yield_0) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1366
/3.3.1/lib/libruby.so.3.3(catch_i) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:2287
/3.3.1/lib/libruby.so.3.3(vm_catch_protect+0xfc) [0x7f11b8ba19dc] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:2369
/3.3.1/lib/libruby.so.3.3(rb_catch_obj+0x51) [0x7f11b8ba1c91] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:2395
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(rb_yield+0xc2) [0x7f11b8bb5382] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(rb_ensure+0x126) [0x7f11b89c7e86] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/eval.c:1009
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(yield_under+0x19d) [0x7f11b8bb433d] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b98c84] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_other) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3544
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(vm_yield_with_cref+0x42) [0x7f11b8bb5338] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(vm_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1642
/3.3.1/lib/libruby.so.3.3(rb_yield_0) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1366
/3.3.1/lib/libruby.so.3.3(rb_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1382
/3.3.1/lib/libruby.so.3.3(rb_ary_collect+0x5c) [0x7f11b890838c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/array.c:3630
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(vm_yield_with_cref+0x42) [0x7f11b8bb5338] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(vm_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1642
/3.3.1/lib/libruby.so.3.3(rb_yield_0) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1366
/3.3.1/lib/libruby.so.3.3(rb_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1382
/3.3.1/lib/libruby.so.3.3(rb_ary_collect+0x5c) [0x7f11b890838c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/array.c:3630
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(vm_yield_with_cref+0x42) [0x7f11b8bb5338] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(vm_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1642
/3.3.1/lib/libruby.so.3.3(rb_yield_0) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1366
/3.3.1/lib/libruby.so.3.3(rb_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1382
/3.3.1/lib/libruby.so.3.3(rb_ary_collect+0x5c) [0x7f11b890838c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/array.c:3630
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(vm_yield_with_cref+0x42) [0x7f11b8bb5338] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1634
/3.3.1/lib/libruby.so.3.3(vm_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:1642
/3.3.1/lib/libruby.so.3.3(rb_yield_0) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1366
/3.3.1/lib/libruby.so.3.3(rb_yield) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_eval.c:1382
/3.3.1/lib/libruby.so.3.3(rb_ary_collect+0x5c) [0x7f11b890838c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/array.c:3630
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0xc4) [0x7f11b8bab103] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:814
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(raise_load_if_failed+0x0) [0x7f11b8a3131e] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:835
/3.3.1/lib/libruby.so.3.3(rb_load_internal) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:837
/3.3.1/lib/libruby.so.3.3(rb_f_load) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:910
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b9890c] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_sendish+0x9e) [0x7f11b8ba9aa7] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:834
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(raise_load_if_failed+0x0) [0x7f11b8a3131e] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:835
/3.3.1/lib/libruby.so.3.3(rb_load_internal) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:837
/3.3.1/lib/libruby.so.3.3(rb_f_load) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:910
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b98c84] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_other) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3544
/3.3.1/lib/libruby.so.3.3(vm_sendish+0x9e) [0x7f11b8ba9aa7] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:834
/3.3.1/lib/libruby.so.3.3(vm_exec_loop+0xa) [0x7f11b8baffe7] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2513
/3.3.1/lib/libruby.so.3.3(rb_vm_exec) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2492
/3.3.1/lib/libruby.so.3.3(raise_load_if_failed+0x0) [0x7f11b8a3131e] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:835
/3.3.1/lib/libruby.so.3.3(rb_load_internal) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:837
/3.3.1/lib/libruby.so.3.3(rb_f_load) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/load.c:910
/3.3.1/lib/libruby.so.3.3(vm_cfp_consistent_p+0x0) [0x7f11b8b98c84] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3490
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame_) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3492
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_with_frame) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3518
/3.3.1/lib/libruby.so.3.3(vm_call_cfunc_other) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:3544
/3.3.1/lib/libruby.so.3.3(vm_sendish+0x9e) [0x7f11b8ba9aa7] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm_insnhelper.c:5581
/3.3.1/lib/libruby.so.3.3(vm_exec_core) /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/insns.def:834
/3.3.1/lib/libruby.so.3.3(rb_vm_exec+0x19a) [0x7f11b8bafd5a] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/vm.c:2486
/3.3.1/lib/libruby.so.3.3(rb_ec_exec_node+0xb5) [0x7f11b89c6075] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/eval.c:287
/3.3.1/lib/libruby.so.3.3(ruby_run_node+0x93) [0x7f11b89c9c73] /tmp/ruby-build.20240423151506.1269237.GtgICM/ruby-3.3.1/eval.c:328
/3.3.1/bin/ruby(rb_main+0x21) [0x55c9c3e1d187] ./main.c:39
/3.3.1/bin/ruby(main) ./main.c:58
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_call_main+0x80) [0x7f11b84bed90] ../sysdeps/nptl/libc_start_call_main.h:58
/lib/x86_64-linux-gnu/libc.so.6(call_init+0x0) [0x7f11b84bee40] ../csu/libc-start.c:392
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main_impl) ../csu/libc-start.c:379
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main) (null):0
[0x55c9c3e1d1d5]

-- Other runtime information -----------------------------------------------

* Loaded script: /3.3.1/bin/rspec

* Loaded features:

    0 enumerator.so
    1 thread.rb
    2 fiber.so
    3 rational.so
    4 complex.so
    5 ruby2_keywords.rb
    6 3.3.0/x86_64-linux/enc/encdb.so
    7 3.3.0/x86_64-linux/enc/trans/transdb.so
    8 3.3.0/x86_64-linux/rbconfig.rb
    9 3.3.0/rubygems/compatibility.rb
   10 3.3.0/rubygems/defaults.rb
   11 3.3.0/rubygems/deprecate.rb
   12 3.3.0/rubygems/errors.rb
   13 3.3.0/rubygems/unknown_command_spell_checker.rb
   14 3.3.0/rubygems/exceptions.rb
   15 3.3.0/rubygems/basic_specification.rb
   16 3.3.0/rubygems/stub_specification.rb
   17 3.3.0/rubygems/platform.rb
   18 3.3.0/rubygems/util/list.rb
   19 3.3.0/rubygems/version.rb
   20 3.3.0/rubygems/requirement.rb
   21 3.3.0/rubygems/specification.rb
   22 3.3.0/rubygems/util.rb
   23 3.3.0/rubygems/dependency.rb
   24 3.3.0/rubygems/core_ext/kernel_gem.rb
   25 3.3.0/x86_64-linux/monitor.so
   26 3.3.0/monitor.rb
   27 3.3.0/rubygems.rb
   28 3.3.0/bundled_gems.rb
   29 3.3.0/rubygems/path_support.rb
   30 3.3.0/error_highlight/version.rb
   31 3.3.0/error_highlight/base.rb
   (...)
  698 gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/formatters/base_formatter.rb
  699 gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/formatters/base_text_formatter.rb
  700 gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/formatters/progress_formatter.rb
  701 gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/profiler.rb
  702 gems/3.3.0/gems/rspec-expectations-3.13.0/lib/rspec/matchers/built_in/raise_error.rb

* Process memory map:

55c9c3e1c000-55c9c3e1d000 r--p 00000000 fd:01 6844619                    /3.3.1/bin/ruby
55c9c3e1d000-55c9c3e1e000 r-xp 00001000 fd:01 6844619                    /3.3.1/bin/ruby
55c9c3e1e000-55c9c3e1f000 r--p 00002000 fd:01 6844619                    /3.3.1/bin/ruby
55c9c3e1f000-55c9c3e20000 r--p 00002000 fd:01 6844619                    /3.3.1/bin/ruby
55c9c3e20000-55c9c3e21000 rw-p 00003000 fd:01 6844619                    /3.3.1/bin/ruby
55c9c5c05000-55c9c7ab0000 rw-p 00000000 00:00 0                          [heap]
7f1174000000-7f1174021000 rw-p 00000000 00:00 0 
7f1174021000-7f1178000000 ---p 00000000 00:00 0 
7f1178c9f000-7f11797ba000 r--s 00000000 fd:03 8409494                    /rdkafka/ext/librdkafka.so
7f11797ba000-7f1179d8f000 rw-p 00000000 00:00 0 
7f1179d8f000-7f117a1c6000 r--s 00000000 fd:01 3801383                    /usr/lib/debug/.build-id/96/2015aa9d133c6cbcfb31ec300596d7f44d3348.debug
7f117a1c6000-7f117ae00000 rw-p 00000000 00:00 0 
7f117ae00000-7f117c000000 r--s 00000000 fd:01 6844635                    /3.3.1/lib/libruby.so.3.3.1
7f117c000000-7f117c021000 rw-p 00000000 00:00 0 
7f117c021000-7f1180000000 ---p 00000000 00:00 0 
7f1180000000-7f1180021000 rw-p 00000000 00:00 0 
7f1180021000-7f1184000000 ---p 00000000 00:00 0 
7f1184000000-7f1184021000 rw-p 00000000 00:00 0 
7f1184021000-7f1188000000 ---p 00000000 00:00 0 
7f1188000000-7f1188176000 rw-p 00000000 00:00 0 
7f1188176000-7f118c000000 ---p 00000000 00:00 0 
7f118c000000-7f118c024000 rw-p 00000000 00:00 0 
7f118c024000-7f1190000000 ---p 00000000 00:00 0 
7f1190000000-7f1190071000 rw-p 00000000 00:00 0 
7f1190071000-7f1194000000 ---p 00000000 00:00 0 
7f1194000000-7f1194021000 rw-p 00000000 00:00 0 
7f1194021000-7f1198000000 ---p 00000000 00:00 0 
7f1198137000-7f1198356000 r--s 00000000 fd:01 139496                     /usr/lib/x86_64-linux-gnu/libc.so.6
7f1198356000-7f1198a3d000 rw-p 00000000 00:00 0 
7f1198a3d000-7f1198a3e000 ---p 00000000 00:00 0 
7f1198a3e000-7f1198b3e000 rw-p 00000000 00:00 0 
7f1198b3e000-7f1198b3f000 ---p 00000000 00:00 0 
7f1198b3f000-7f119933f000 rw-p 00000000 00:00 0 
7f119933f000-7f1199340000 ---p 00000000 00:00 0 
7f1199340000-7f11994e0000 rw-p 00000000 00:00 0 
7f11994e4000-7f11994e5000 ---p 00000000 00:00 0 
7f11994e5000-7f1199ce5000 rw-p 00000000 00:00 0 
7f1199d8e000-7f1199d9a000 r--s 00000000 fd:01 131983                     /usr/lib/x86_64-linux-gnu/libffi.so.8.1.0
7f1199d9a000-7f1199dd0000 r--s 00000000 fd:01 6844619                    /3.3.1/bin/ruby
7f1199dd0000-7f1199de0000 rw-p 00000000 00:00 0 
7f1199de6000-7f1199de7000 ---p 00000000 00:00 0 
7f1199de7000-7f119a5e7000 rw-p 00000000 00:00 0 
7f119a5e7000-7f119a5e8000 ---p 00000000 00:00 0 
7f119a5e8000-7f119ade8000 rw-p 00000000 00:00 0 
7f119ade8000-7f119ade9000 ---p 00000000 00:00 0 
7f119ade9000-7f119b5e9000 rw-p 00000000 00:00 0 
7f119b5e9000-7f119b609000 r--p 00000000 fd:01 131184                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
7f119b609000-7f119b74a000 r-xp 00020000 fd:01 131184                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
7f119b74a000-7f119b790000 r--p 00161000 fd:01 131184                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
7f119b790000-7f119b797000 r--p 001a6000 fd:01 131184                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
7f119b797000-7f119b798000 rw-p 001ad000 fd:01 131184                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
7f119b79f000-7f119b7a1000 r--p 00000000 fd:01 2105456                    /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so.2.0.25
7f119b7a1000-7f119b7a6000 r-xp 00002000 fd:01 2105456                    /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so.2.0.25
7f119b7a6000-7f119b7a8000 r--p 00007000 fd:01 2105456                    /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so.2.0.25
7f119b7a8000-7f119b7a9000 r--p 00008000 fd:01 2105456                    /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so.2.0.25
7f119b7a9000-7f119b7aa000 rw-p 00009000 fd:01 2105456                    /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so.2.0.25
7f119b7aa000-7f119b7ac000 r--p 00000000 fd:01 2105454                    /usr/lib/x86_64-linux-gnu/sasl2/libgs2.so.2.0.25
7f119b7ac000-7f119b7b1000 r-xp 00002000 fd:01 2105454                    /usr/lib/x86_64-linux-gnu/sasl2/libgs2.so.2.0.25
(...)
7f11b8ed9000-7f11b8edb000 rw-p 00039000 fd:01 136273                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fff36459000-7fff36c58000 rw-p 00000000 00:00 0                          [stack]
7fff36cca000-7fff36cce000 r--p 00000000 00:00 0                          [vvar]
7fff36cce000-7fff36cd0000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]


Segmentation fault (core dumped)

How to reproduce

1. Create a transactional producer
2. Create per topic config and use it as I described above
3. Try to send a message

I can reproduce it consistently.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 2.3.0
• Apache Kafka version: confluentinc/cp-kafka:7.6.1 with KRaft
• librdkafka client configuration: defaults
• Operating system: Linux 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue NO - I can bypass that by implementing validations at a higher level

Broker log is pretty chill about this:

kafka  | [2024-05-07 12:30:49,792] INFO [Broker id=1] Leader 07328bbb-53f3-4773-aec5-2e267c5ee7ef-0 with topic id Some(JORL-9WgRFuoJ_G7d_kzwQ) starts at leader epoch 0 from offset 0 with partition epoch 0, high watermark 0, ISR [1], adding replicas [] and removing replicas [] . Previous leader None and previous leader epoch was -1. (state.change.logger)
kafka  | [2024-05-07 12:30:50,864] INFO [TransactionCoordinator id=1] Initialized transactionalId 9d01cf3c-ea98-43b1-be24-b73e52eaa07d with producerId 340 and producer epoch 0 on partition __transaction_state-9 (kafka.coordinator.transaction.TransactionCoordinator)

Proposed solution

This should be validated in the context of librdkafka producer instance and changing acks should not be allowed for transactional producers.