Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in networking stack #20345

Open
rolfbjarne opened this issue Mar 20, 2024 · 0 comments
Open

Crash in networking stack #20345

rolfbjarne opened this issue Mar 20, 2024 · 0 comments
Labels
bug If an issue is a bug or a pull request a bug fix networking If an issue or pull request is related to networking
Milestone
Future

Comments

@rolfbjarne
Copy link
Member

Use this test project: #11799 (comment)

Run the project for a while (typically hours, sometimes overnight - at one point it crashed after 16 hours / just over 1.5M network requests), and it eventually crashes.

lldb shows this stack trace:

(lldb) bt
* thread #1, name = 'tid_103', queue = 'com.apple.main-thread', stop reason = signal SIGSEGV
  * frame #0: 0x000000019d9e5c34 libobjc.A.dylib`objc_msgSend + 52
    frame #1: 0x000000019da18a5c libobjc.A.dylib`objc_object::sidetable_release(bool, bool) + 292
    frame #2: 0x00000001a30f169c CFNetwork`___lldb_unnamed_symbol2909 + 32
    frame #3: 0x00000001a30f1400 CFNetwork`___lldb_unnamed_symbol2903 + 260
    frame #4: 0x000000019dab717c libsystem_blocks.dylib`_call_dispose_helpers_excp + 48
    frame #5: 0x000000019dab6f48 libsystem_blocks.dylib`_Block_release + 252
    frame #6: 0x00000001a3194e7c CFNetwork`___lldb_unnamed_symbol5273 + 28
    frame #7: 0x000000019dab717c libsystem_blocks.dylib`_call_dispose_helpers_excp + 48
    frame #8: 0x000000019dab6f48 libsystem_blocks.dylib`_Block_release + 252
    frame #9: 0x000000019dc0a3e8 libdispatch.dylib`_dispatch_client_callout + 20
    frame #10: 0x000000019dc18bb8 libdispatch.dylib`_dispatch_main_queue_drain + 988
    frame #11: 0x000000019dc187cc libdispatch.dylib`_dispatch_main_queue_callback_4CF + 44
    frame #12: 0x000000019dedb4ac CoreFoundation`__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
    frame #13: 0x000000019de98c30 CoreFoundation`__CFRunLoopRun + 1996
    frame #14: 0x000000019de97e0c CoreFoundation`CFRunLoopRunSpecific + 608
    frame #15: 0x00000001a8633000 HIToolbox`RunCurrentEventLoopInMode + 292
    frame #16: 0x00000001a8632e3c HIToolbox`ReceiveNextEventCommon + 648
    frame #17: 0x00000001a8632b94 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 76
    frame #18: 0x00000001a16f0970 AppKit`_DPSNextEvent + 660
    frame #19: 0x00000001a1ee2dec AppKit`-[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 700
    frame #20: 0x00000001a16e3cb8 AppKit`-[NSApplication run] + 476
    frame #21: 0x00000001a16baf54 AppKit`NSApplicationMain + 880
    frame #22: 0x00000001a190d610 AppKit`_NSApplicationMainWithInfoDictionary + 24
    frame #23: 0x00000001b71290dc UIKitMacHelper`UINSApplicationMain + 972
    frame #24: 0x00000001cd2159b4 UIKitCore`UIApplicationMain + 148
    frame #25: 0x0000000106879854 nsurlsessionhandler`xamarin_UIApplicationMain(argc=0, argv=0x00006000028144c0, principalClassName=0x0000000000000000, delegateClassName="AppDelegate", exception_gchandle=0x000000016fdfe8e0) at bindings.m:126:10
    frame #26: 0x00000001065c6280 nsurlsessionhandler`wrapper_managed_to_native_UIKit_UIApplication_xamarin_UIApplicationMain_int_intptr_intptr_intptr_intptr_ + 176
    frame #27: 0x0000000105db9614 nsurlsessionhandler`UIKit_UIApplication_UIApplicationMain_int_string___intptr_intptr + 100
    frame #28: 0x0000000105db9948 nsurlsessionhandler`UIKit_UIApplication_Main_string___System_Type_System_Type + 280
    frame #29: 0x0000000102492d68 nsurlsessionhandler`Program__Main__string__ + 136
    frame #30: 0x000000010508ea08 nsurlsessionhandler`wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr + 296
    frame #31: 0x0000000106b8ea90 nsurlsessionhandler`mono_jit_runtime_invoke(method=<unavailable>, obj=<unavailable>, params=<unavailable>, exc=<unavailable>, error=0x000000016fdfec90) at mini-runtime.c:3636:3 [opt]
    frame #32: 0x0000000106b2f158 nsurlsessionhandler`mono_runtime_invoke_checked [inlined] do_runtime_invoke(method=0x000000011a914598, obj=0x0000000000000000, params=0x000000016fdfec30, exc=0x0000000000000000, error=0x000000016fdfec90) at object.c:2576:11 [opt]
    frame #33: 0x0000000106b2f11c nsurlsessionhandler`mono_runtime_invoke_checked(method=0x000000011a914598, obj=0x0000000000000000, params=0x000000016fdfec30, error=0x000000016fdfec90) at object.c:2792:9 [opt]
    frame #34: 0x0000000106b35494 nsurlsessionhandler`mono_runtime_exec_main_checked [inlined] do_exec_main_checked(method=0x000000011a914598, args=<unavailable>, error=0x000000016fdfec90) at object.c:0 [opt]
    frame #35: 0x0000000106b35458 nsurlsessionhandler`mono_runtime_exec_main_checked(method=0x000000011a914598, args=<unavailable>, error=0x000000016fdfec90) at object.c:4775:9 [opt]
    frame #36: 0x0000000106b35540 nsurlsessionhandler`mono_runtime_run_main_checked(method=<unavailable>, argc=<unavailable>, argv=<unavailable>, error=<unavailable>) at object.c:4339:9 [opt] [artificial]
    frame #37: 0x0000000106be1434 nsurlsessionhandler`mono_jit_exec at driver.c:1369:13 [opt]
    frame #38: 0x0000000106be1424 nsurlsessionhandler`mono_jit_exec(domain=<unavailable>, assembly=<unavailable>, argc=1, argv=0x000000016fdfed10) at driver.c:1314:7 [opt]
    frame #39: 0x00000001068a2e58 nsurlsessionhandler`xamarin_main(argc=1, argv=0x000000016fdff070, launch_mode=XamarinLaunchModeApp) at monotouch-main.m:495:8
    frame #40: 0x0000000106d47b18 nsurlsessionhandler`main(argc=1, argv=0x000000016fdff070) at main.arm64.mm:416:11
    frame #41: 0x000000019da320e0 dyld`start + 2360

manually symbolicating the stack using backtrace in lldb shows:

(lldb) parray 20 (char **) 0x000000011aa14800
(char **) $1 = 0x000000011aa14800 {
  [0] = 0x000000011aa148a0 "0   ???                                 0x000000010f58c64c 0x0 + 4552443468"
  [1] = 0x000000011aa148ec "1   nsurlsessionhandler                 0x0000000106d47ad8 main + 0"
  [2] = 0x000000011aa14930 "2   CFNetwork                           0x00000001a30f169c CFURLRequestGetMainDocumentURL + 20720"
  [3] = 0x000000011aa14992 "3   CFNetwork                           0x00000001a30f1400 CFURLRequestGetMainDocumentURL + 20052"
  [4] = 0x000000011aa149f4 "4   libsystem_blocks.dylib              0x000000019dab717c _call_dispose_helpers_excp + 48"
  [5] = 0x000000011aa14a4f "5   libsystem_blocks.dylib              0x000000019dab6f48 _Block_release + 252"
  [6] = 0x000000011aa14a9f "6   CFNetwork                           0x00000001a3194e7c CFURLCredentialStorageCopyAllCredentials + 40380"
  [7] = 0x000000011aa14b0b "7   libsystem_blocks.dylib              0x000000019dab717c _call_dispose_helpers_excp + 48"
  [8] = 0x000000011aa14b66 "8   libsystem_blocks.dylib              0x000000019dab6f48 _Block_release + 252"
  [9] = 0x000000011aa14bb6 "9   libdispatch.dylib                   0x000000019dc0a3e8 _dispatch_client_callout + 20"
  [10] = 0x000000011aa14c0f "10  libdispatch.dylib                   0x000000019dc18bb8 _dispatch_main_queue_drain + 988"
  [11] = 0x000000011aa14c6b "11  libdispatch.dylib                   0x000000019dc187cc _dispatch_main_queue_callback_4CF + 44"
  [12] = 0x000000011aa14ccd "12  CoreFoundation                      0x000000019dedb4ac __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16"
  [13] = 0x000000011aa14d40 "13  CoreFoundation                      0x000000019de98c30 __CFRunLoopRun + 1996"
  [14] = 0x000000011aa14d91 "14  CoreFoundation                      0x000000019de97e0c CFRunLoopRunSpecific + 608"
  [15] = 0x000000011aa14de7 "15  HIToolbox                           0x00000001a8633000 RunCurrentEventLoopInMode + 292"
  [16] = 0x000000011aa14e42 "16  HIToolbox                           0x00000001a8632e3c ReceiveNextEventCommon + 648"
  [17] = 0x000000011aa14e9a "17  HIToolbox                           0x00000001a8632b94 _BlockUntilNextEventMatchingListInModeWithFilter + 76"
  [18] = 0x000000011aa14f0b "18  AppKit                              0x00000001a16f0970 _DPSNextEvent + 660"
  [19] = 0x000000011aa14f5a "19  AppKit                              0x00000001a1ee2dec -[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 700"
}

Some investigation in lldb revelead:

  • ___lldb_unnamed_symbol2909: [__NSCFURLSessionConnection dealloc]
  • ___lldb_unnamed_symbol2903: [__NSCFURLLocalSessionConnection dealloc]
  • ___lldb_unnamed_symbol5273: a block dispose method

Some more lldb info:


(lldb) disass
CFNetwork`___lldb_unnamed_symbol2909: # [__NSCFURLSessionConnection dealloc]
    0x1a30f167c <+0>:   pacibsp
    0x1a30f1680 <+4>:   sub    sp, sp, #0x30
    0x1a30f1684 <+8>:   stp    x20, x19, [sp, #0x10]
    0x1a30f1688 <+12>:  stp    x29, x30, [sp, #0x20]
    0x1a30f168c <+16>:  add    x29, sp, #0x20
    0x1a30f1690 <+20>:  mov    x19, x0
    0x1a30f1694 <+24>:  ldr    x0, [x0, #0x8]
    0x1a30f1698 <+28>:  bl     0x1a3359158               ; symbol stub for: objc_release
->  0x1a30f169c <+32>:  ldr    x0, [x19, #0x10]
    0x1a30f16a0 <+36>:  cbz    x0, 0x1a30f16ac           ; <+48>
    0x1a30f16a4 <+40>:  bl     0x1a3357578               ; symbol stub for: dispatch_release
    0x1a30f16a8 <+44>:  str    xzr, [x19, #0x10]
    0x1a30f16ac <+48>:  mov    x0, x19
    0x1a30f16b0 <+52>:  mov    x2, #0x0
    0x1a30f16b4 <+56>:  mov    w3, #0x18
    0x1a30f16b8 <+60>:  bl     0x1a33591c8               ; symbol stub for: objc_setProperty_atomic
    0x1a30f16bc <+64>:  ldr    x0, [x19, #0x28]
    0x1a30f16c0 <+68>:  cbz    x0, 0x1a30f16cc           ; <+80>
    0x1a30f16c4 <+72>:  bl     0x1a3359158               ; symbol stub for: objc_release
    0x1a30f16c8 <+76>:  str    xzr, [x19, #0x28]
    0x1a30f16cc <+80>:  ldr    x0, [x19, #0x30]
    0x1a30f16d0 <+84>:  cbz    x0, 0x1a30f16dc           ; <+96>
    0x1a30f16d4 <+88>:  bl     0x1a3357578               ; symbol stub for: dispatch_release
    0x1a30f16d8 <+92>:  str    xzr, [x19, #0x30]
    0x1a30f16dc <+96>:  adrp   x8, 424846
    0x1a30f16e0 <+100>: ldr    x8, [x8, #0x890]
    0x1a30f16e4 <+104>: stp    x19, x8, [sp]
    0x1a30f16e8 <+108>: adrp   x8, 373089
    0x1a30f16ec <+112>: add    x1, x8, #0x71f
    0x1a30f16f0 <+116>: mov    x0, sp
    0x1a30f16f4 <+120>: bl     0x1a33590f8               ; symbol stub for: objc_msgSendSuper2
    0x1a30f16f8 <+124>: ldp    x29, x30, [sp, #0x20]
    0x1a30f16fc <+128>: ldp    x20, x19, [sp, #0x10]
    0x1a30f1700 <+132>: add    sp, sp, #0x30
    0x1a30f1704 <+136>: retab


(lldb) disass
CFNetwork`___lldb_unnamed_symbol2903: # [__NSCFURLLocalSessionConnection dealloc]
    0x1a30f12fc <+0>:   pacibsp
    0x1a30f1300 <+4>:   sub    sp, sp, #0x30
    0x1a30f1304 <+8>:   stp    x20, x19, [sp, #0x10]
    0x1a30f1308 <+12>:  stp    x29, x30, [sp, #0x20]
    0x1a30f130c <+16>:  add    x29, sp, #0x20
    0x1a30f1310 <+20>:  mov    x19, x0
    0x1a30f1314 <+24>:  adrp   x8, 399283
    0x1a30f1318 <+28>:  ldrsw  x20, [x8, #0x414]
    0x1a30f131c <+32>:  ldr    x0, [x0, x20]
    0x1a30f1320 <+36>:  cbz    x0, 0x1a30f134c           ; <+80>
    0x1a30f1324 <+40>:  ldr    x16, [x0]
    0x1a30f1328 <+44>:  mov    x17, x0
    0x1a30f132c <+48>:  movk   x17, #0x81be, lsl #48
    0x1a30f1330 <+52>:  autda  x16, x17
    0x1a30f1334 <+56>:  ldr    x8, [x16, #0x8]!
    0x1a30f1338 <+60>:  mov    x9, x16
    0x1a30f133c <+64>:  mov    x17, x9
    0x1a30f1340 <+68>:  movk   x17, #0x990e, lsl #48
    0x1a30f1344 <+72>:  blraa  x8, x17
    0x1a30f1348 <+76>:  str    xzr, [x19, x20]
    0x1a30f134c <+80>:  adrp   x8, 399283
    0x1a30f1350 <+84>:  ldrsw  x20, [x8, #0x410]
    0x1a30f1354 <+88>:  ldr    x0, [x19, x20]
    0x1a30f1358 <+92>:  cbz    x0, 0x1a30f1384           ; <+136>
    0x1a30f135c <+96>:  ldr    x16, [x0]
    0x1a30f1360 <+100>: mov    x17, x0
    0x1a30f1364 <+104>: movk   x17, #0x4399, lsl #48
    0x1a30f1368 <+108>: autda  x16, x17
    0x1a30f136c <+112>: ldr    x8, [x16, #0x8]!
    0x1a30f1370 <+116>: mov    x9, x16
    0x1a30f1374 <+120>: mov    x17, x9
    0x1a30f1378 <+124>: movk   x17, #0x3f8b, lsl #48
    0x1a30f137c <+128>: blraa  x8, x17
    0x1a30f1380 <+132>: str    xzr, [x19, x20]
    0x1a30f1384 <+136>: adrp   x8, 399283
    0x1a30f1388 <+140>: ldrsw  x20, [x8, #0x41c]
    0x1a30f138c <+144>: ldr    x0, [x19, x20]
    0x1a30f1390 <+148>: cbz    x0, 0x1a30f139c           ; <+160>
    0x1a30f1394 <+152>: bl     0x1a3359158               ; symbol stub for: objc_release
    0x1a30f1398 <+156>: str    xzr, [x19, x20]
    0x1a30f139c <+160>: adrp   x8, 399283
    0x1a30f13a0 <+164>: ldrsw  x20, [x8, #0x420]
    0x1a30f13a4 <+168>: ldr    x0, [x19, x20]
    0x1a30f13a8 <+172>: cbz    x0, 0x1a30f13b4           ; <+184>
    0x1a30f13ac <+176>: bl     0x1a3357578               ; symbol stub for: dispatch_release
    0x1a30f13b0 <+180>: str    xzr, [x19, x20]
    0x1a30f13b4 <+184>: adrp   x8, 399283
    0x1a30f13b8 <+188>: ldrsw  x20, [x8, #0x424]
    0x1a30f13bc <+192>: ldr    x0, [x19, x20]
    0x1a30f13c0 <+196>: cbz    x0, 0x1a30f13cc           ; <+208>
    0x1a30f13c4 <+200>: bl     0x1a3357578               ; symbol stub for: dispatch_release
    0x1a30f13c8 <+204>: str    xzr, [x19, x20]
    0x1a30f13cc <+208>: adrp   x8, 399283
    0x1a30f13d0 <+212>: ldrsw  x20, [x8, #0x428]
    0x1a30f13d4 <+216>: ldr    x0, [x19, x20]
    0x1a30f13d8 <+220>: cbz    x0, 0x1a30f13e4           ; <+232>
    0x1a30f13dc <+224>: bl     0x1a3359158               ; symbol stub for: objc_release
    0x1a30f13e0 <+228>: str    xzr, [x19, x20]
    0x1a30f13e4 <+232>: adrp   x8, 424846
    0x1a30f13e8 <+236>: ldr    x8, [x8, #0x898]
    0x1a30f13ec <+240>: stp    x19, x8, [sp]
    0x1a30f13f0 <+244>: adrp   x8, 373089
    0x1a30f13f4 <+248>: add    x1, x8, #0x71f
    0x1a30f13f8 <+252>: mov    x0, sp
    0x1a30f13fc <+256>: bl     0x1a33590f8               ; symbol stub for: objc_msgSendSuper2
->  0x1a30f1400 <+260>: ldp    x29, x30, [sp, #0x20]
    0x1a30f1404 <+264>: ldp    x20, x19, [sp, #0x10]
    0x1a30f1408 <+268>: add    sp, sp, #0x30
    0x1a30f140c <+272>: retab

(lldb) disass
CFNetwork`___lldb_unnamed_symbol5273: # block dispose method
    0x1a3194e60 <+0>:  pacibsp
    0x1a3194e64 <+4>:  stp    x20, x19, [sp, #-0x20]!
    0x1a3194e68 <+8>:  stp    x29, x30, [sp, #0x10]
    0x1a3194e6c <+12>: add    x29, sp, #0x10
    0x1a3194e70 <+16>: mov    x19, x0
    0x1a3194e74 <+20>: ldr    x0, [x0, #0x30]
    0x1a3194e78 <+24>: bl     0x1a3359158               ; symbol stub for: objc_release
->  0x1a3194e7c <+28>: ldr    x0, [x19, #0x28]
    0x1a3194e80 <+32>: bl     0x1a3359158               ; symbol stub for: objc_release
    0x1a3194e84 <+36>: ldr    x0, [x19, #0x20]
    0x1a3194e88 <+40>: ldp    x29, x30, [sp, #0x10]
    0x1a3194e8c <+44>: ldp    x20, x19, [sp], #0x20
    0x1a3194e90 <+48>: autibsp
    0x1a3194e94 <+52>: eor    x16, x30, x30, lsl #1
    0x1a3194e98 <+56>: tbz    x16, #0x3e, 0x1a3194ea0   ; <+64>
    0x1a3194e9c <+60>: brk    #0xc471
    0x1a3194ea0 <+64>: b      0x1a3359158               ; symbol stub for: objc_release

(lldb) re re
General Purpose Registers:
        x0 = 0x000000011b8b5180
        x1 = 0x00000001fe25271f
        x2 = 0x0000600003fbe3c0
        x3 = 0x0000600003fbe3c0
        x4 = 0x0000600003fbe440
        x5 = 0x00000000000023c0
        x6 = 0x0000000000000000
        x7 = 0x0000000000000403
        x8 = 0x000000020a5f4e78  "dealloc"
        x9 = 0x000000020a5f4e78  "dealloc"
       x10 = 0x0000000400000041
       x11 = 0x0000000000000000
       x12 = 0x0000000000000000
       x13 = 0x0000000400000041
       x14 = 0x000000011b893ce0
       x15 = 0x000000011b893ce0
       x16 = 0x000000011b893ce0
       x17 = 0x0000000000000001
       x18 = 0x0000000000000000
       x19 = 0x000000011b8b5180
       x20 = 0x0000000000000001
       x21 = 0x0000000205ac7400  libobjc.A.dylib`(anonymous namespace)::SideTablesMap + 3072
       x22 = 0x0000000000000001
       x23 = 0x0000000000000104
       x24 = 0x0000000000000000
       x25 = 0x0000000205ac7ba0  dyld`_main_thread + 224
       x26 = 0x0000600003fbd100
       x27 = 0x000000000000000f
       x28 = 0x0000000000000000
        fp = 0x000000016fdfcea0
        lr = 0x000000019da18a5c  libobjc.A.dylib`objc_object::sidetable_release(bool, bool) + 292
        sp = 0x000000016fdfce50
        pc = 0x000000019d9e5c34  libobjc.A.dylib`objc_msgSend + 52
      cpsr = 0x20001000
@rolfbjarne rolfbjarne added bug If an issue is a bug or a pull request a bug fix networking If an issue or pull request is related to networking labels Mar 20, 2024
@rolfbjarne rolfbjarne added this to the Future milestone Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug If an issue is a bug or a pull request a bug fix networking If an issue or pull request is related to networking
Projects
None yet
Milestone
Future
Development

No branches or pull requests
1 participant
@rolfbjarne