Shutdown Crash After Failure to Launch Extension

[tferguson7337]

Bug report

What operating system and version are you using?

Windows, crash most often seen on Server 2022, but seen on client versions of Win10 as well.

What version of osquery are you using?

5.11.0

What steps did you take to reproduce the issue?

Crash seen occasionally in automation runs (~ once every 1-3 days).

What did you expect to see?

No crash

What did you see instead?

Crash, with call stack:

# 10  Id: 9c0.a74 Suspend: 0 Teb: 0000005e`e1a7f000 Unfrozen
 # Child-SP          RetAddr               Call Site
00 0000005e`e26fe6f8 00007ff8`ac06ffdc     ntdll!NtWaitForMultipleObjects+0x14
01 0000005e`e26fe700 00007ff8`ac06fede     KERNELBASE!WaitForMultipleObjectsEx+0xec
02 0000005e`e26fe9f0 00007ff8`adf72301     KERNELBASE!WaitForMultipleObjects+0xe
03 0000005e`e26fea30 00007ff8`adf71ecd     kernel32!WerpReportFaultInternal+0x425
04 0000005e`e26feb50 00007ff8`adf71e3e     kernel32!WerpReportFault+0x81
05 0000005e`e26feb90 00007ff8`ac0b7e54     kernel32!BasepReportFault+0x1e
06 0000005e`e26febc0 00007ff8`aeb08905     KERNELBASE!UnhandledExceptionFilter+0x344
07 0000005e`e26fece0 00007ff8`aeaee756     ntdll!RtlUserThreadStart$filt$0+0xac
08 0000005e`e26fed20 00007ff8`aeb0471f     ntdll!_C_specific_handler+0x96
09 0000005e`e26fed90 00007ff8`aea9170e     ntdll!RtlpExecuteHandlerForException+0xf
0a 0000005e`e26fedc0 00007ff8`aeb0372e     ntdll!RtlDispatchException+0x26e
0b 0000005e`e26ff500 00007ff6`01d216fd     ntdll!KiUserExceptionDispatch+0x2e
0c 0000005e`e26ffc10 00007ff6`00d787bf     osqueryi!osquery::PlatformProcess::pid+0xd
0d 0000005e`e26ffc40 00007ff6`00d720d1     osqueryi!osquery::WatcherRunner::stopChild+0x3f
0e 0000005e`e26ffd10 00007ff6`00d6ff68     osqueryi!<lambda_b5a4cb303bf14dfd1a506d0c89a75933>::operator()+0x31
0f (Inline Function) --------`--------     osqueryi!std::invoke+0x32
10 0000005e`e26ffd60 00007ff6`021cde90     osqueryi!std::thread::_Invoke<std::tuple<<lambda_b5a4cb303bf14dfd1a506d0c89a75933>,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::shared_ptr<osquery::PlatformProcess> >,0,1,2>+0x48
11 0000005e`e26ffda0 00007ff8`adf64de0     osqueryi!thread_start<unsigned int (__cdecl*)(void *),1>+0x50
12 0000005e`e26ffdd0 00007ff8`aeade40b     kernel32!BaseThreadInitThunk+0x10
13 0000005e`e26ffe00 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

The main thread is currently in the middle of waiting for shutdown after launching the above crashing thread to shutdown extensions:

0:000> ~0 knL
 # Child-SP          RetAddr               Call Site
00 0000005e`e196f178 00007ff8`ac04d52e     ntdll!NtWaitForSingleObject+0x14
01 0000005e`e196f180 00007ff6`02174dc3     KERNELBASE!WaitForSingleObjectEx+0x8e
02 0000005e`e196f220 00007ff6`00d786dc     osqueryi!_Thrd_join+0x1f
03 (Inline Function) --------`--------     osqueryi!std::thread::join+0x63
04 0000005e`e196f250 00007ff6`01d19f9e     osqueryi!osquery::WatcherRunner::stop+0x1cc
05 0000005e`e196f2f0 00007ff6`01d1ac08     osqueryi!osquery::InterruptibleRunnable::interrupt+0x4e
06 0000005e`e196f320 00007ff6`00d6d196     osqueryi!osquery::Dispatcher::stopServices+0x118
07 0000005e`e196f3a0 00007ff6`0224caba     osqueryi!osquery::Initializer::shutdown+0xc6
08 0000005e`e196f4f0 00007ff6`0224c062     osqueryi!osquery::startOsquery+0x3da
09 0000005e`e196f660 00007ff6`021724bc     osqueryi!main+0x92
0a (Inline Function) --------`--------     osqueryi!invoke_main+0x22
0b 0000005e`e196f720 00007ff8`adf64de0     osqueryi!__scrt_common_main_seh+0x10c
0c 0000005e`e196f760 00007ff8`aeade40b     kernel32!BaseThreadInitThunk+0x10
0d 0000005e`e196f790 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

Issue appears to be due to the following:

osquery/osquery/core/watcher.cpp

Lines 758 to 770 in fcbb368

	auto ext_process =
	PlatformProcess::launchExtension(exec_path.string(),
	Flag::getValue("extensions_socket"),
	Flag::getValue("extensions_timeout"),
	Flag::getValue("extensions_interval"),
	Flag::getValue("verbose") == "true");
	if (ext_process == nullptr) {
	// Unrecoverable error, cannot create an extension process.
	LOG(ERROR) << "Cannot create extension process: " << extension;
	requestShutdown(EXIT_FAILURE);
	}
	watcher_->setExtension(extension, ext_process);

The above if check will log extension launch failure and signal shutdown, but does not include an early return. As a result, an empty std::shared_ptr<PlatformProcess> is added to the watcher's extension map, which the crashing thread later chokes on since it's not anticipating nulls to be in the map.

Seems like a simple one liner fix:

  if (ext_process == nullptr) {
    // Unrecoverable error, cannot create an extension process.
    LOG(ERROR) << "Cannot create extension process: " << extension;
    requestShutdown(EXIT_FAILURE);
    return; // Don't pollute the watcher's extension map with a bad entry.
  }