[D148266] [clang][driver] Linking to just-built libc++.dylib when bootstrapping libc++ with clang #77653

ldionne · 2024-01-10T17:59:18Z

This issue tracks picking up https://reviews.llvm.org/D148266 from the Phabricator archive.

Also related to rdar://107060541.

llvmbot · 2024-01-10T17:59:32Z

@llvm/issue-subscribers-clang-driver

Author: Louis Dionne (ldionne)

This issue tracks picking up https://reviews.llvm.org/D148266 from the Phabricator archive.

Also related to rdar://107060541.

hoyhoy · 2024-03-08T17:57:46Z

I just built with your patch, but I'm getting...

 /opt/llvm18/bin/clang++ -v  ~/backup/cmake_bootstrap.cxx -o cmake_bootstrap_5096_test
clang version 18.1.0rc
Target: x86_64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /opt/llvm18/bin
Configuration file: /opt/llvm18/etc/x86_64-apple-darwin23.4.0-clang++.cfg
System configuration file directory: /opt/llvm18/etc
 "/opt/llvm18/bin/clang-18" -cc1 -triple x86_64-apple-macosx14.4.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -dumpdir cmake_bootstrap_5096_test- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name cmake_bootstrap.cxx -mrelocation-model pic -pic-level 2 -mframe-pointer=all -ffp-contract=on -fno-rounding-math -funwind-tables=2 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -fbuiltin-headers-in-system-modules -fdefine-target-os-macros -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -fdebug-compilation-dir=/Users/jperrie/llvm-project-18.1.0.src/build -v -fcoverage-compilation-dir=/Users/jperrie/llvm-project-18.1.0.src/build -resource-dir /opt/llvm18/lib/clang/18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/opt/llvm18/include -internal-isystem /opt/llvm18/bin/../include/c++/v1 -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /opt/llvm18/lib/clang/18/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -fdeprecated-macro -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcolor-diagnostics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o -x c++ /Users/jperrie/backup/cmake_bootstrap.cxx
clang -cc1 version 18.1.0rc based upon LLVM 18.1.0rc default target x86_64-apple-darwin23.4.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /opt/llvm18/include
 /opt/llvm18/bin/../include/c++/v1
 /opt/llvm18/lib/clang/18/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
 "/opt/llvm18/bin/ld64.lld" -demangle -dynamic -arch x86_64 -platform_version macos 14.4.0 14.4.0 -L /opt/llvm18/bin/../lib -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -mllvm -enable-linkonceodr-outlining -o cmake_bootstrap_5096_test -L/opt/llvm18/lib/darwin -L/opt/llvm18/lib -L/opt/llvm18/lib -L/opt/llvm18/lib/clang/18/lib/darwin -rpath /opt/llvm18/lib /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o -lc++ -lSystem /opt/llvm18/lib/clang/18/lib/darwin/libclang_rt.osx.a
ld64.lld: error: undefined symbol: operator new(unsigned long)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol check_cxx17()+0xe)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol main+0x15)

ld64.lld: error: undefined symbol: __cxa_throw
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::__throw_bad_optional_access[abi:ne180100]()+0x38)

ld64.lld: error: undefined symbol: __cxa_allocate_exception
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::__throw_bad_optional_access[abi:ne180100]()+0xe)

ld64.lld: error: undefined symbol: vtable for std::exception
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::exception::exception[abi:ne180100]()+0xf)

ld64.lld: error: undefined symbol: std::terminate()
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol __clang_call_terminate+0xa)

ld64.lld: error: undefined symbol: __cxa_begin_catch
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol __clang_call_terminate+0x5)

ld64.lld: error: undefined symbol: operator delete(void*)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::default_delete<int>::operator()[abi:ne180100](int*) const+0x27)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::default_delete<Class>::operator()[abi:ne180100](Class*) const+0x27)

ld64.lld: error: undefined symbol: __gxx_personality_v0
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol EH_Frame+0x13)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(__compact_unwind+0x130)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(__compact_unwind+0x70)
>>> referenced 3 more times
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I even rebuilt the runtimes with a second stage using the installed llvm, but that didn't work either.

hoyhoy · 2024-03-08T18:05:01Z

Working llvm 17.0.6 vs non-working llvm 18.1.0...

# working llvm 17.0.6
$ sudo dtruss -a /opt/llvm17/bin/clang++  cmake_bootstrap.cxx -o cmake_test 2>&1 | grep 'open.*\.dylib'
86197/0x14cb7a:      1167      27     25 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1201      15     14 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1247      14     13 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1345      11     10 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86197/0x14cb7a:      1394      17     16 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1441      14     13 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1510       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86197/0x14cb7a:      1560      17     16 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1591      13     12 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:     15717      45     43 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:     33877      42     38 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:     34208      19     18 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:       899      23     21 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:       930      15     13 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:       958      13     12 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1040       4      2 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86198/0x14cb7c:      1085      16     15 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1109      13     12 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1170       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86198/0x14cb7c:      1217      16     15 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1241      13     12 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:     14174      27     25 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:     30621      22     20 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:     30886      16     14 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0

# not working llvm 18.0.1
$ sudo dtruss -a /opt/llvm18/bin/clang++  cmake_bootstrap.cxx -o cmake_test 2>&1 | grep 'open.*\.dylib'
86205/0x14cbfd:      1114      74     36 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1157      16     15 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1208      15     14 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1316      11     10 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86205/0x14cbfd:      1354      17     16 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1402      14     13 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1468       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86205/0x14cbfd:      1505      17     16 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1537      14     13 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:     29702      42     40 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:     47315      39     35 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:     47616      19     17 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:       837      23     22 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:       869      13     11 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:       947      35     34 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1021       3      2 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86206/0x14cc08:      1050      14     13 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1071      11     10 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1119       3      2 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86206/0x14cc08:      1148      13     12 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1169      11     10 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:     26020      32     30 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:     41340      22     20 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:     41585      15     14 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:       947      25     23 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:       981      15     14 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1008      14     12 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1106       4      3 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86207/0x14cc0a:      1143      17     16 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1169      14     12 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1229       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86207/0x14cc0a:      1266      17     16 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1292      14     12 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     26033      28     26 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     42373      91     36 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     42637      17     15 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     49265      16     14 open("/opt/llvm18/bin/../lib/libc++.dylib\0", 0x1000000, 0x0)		 = 3 0
86207/0x14cc0a:     70134      17     16 open("/opt/llvm18/bin/../lib/libunwind.dylib\0", 0x1000000, 0x0)		 = 3 0

hoyhoy · 2024-03-08T18:06:49Z

Seems like something with libcxx is messed up. Or maybe because I configured libunwind this time?

hoyhoy · 2024-03-08T18:21:29Z

I also tried manually changing libcxx's rpath out to the llvm 18.1.0 binary ...

llvm-install-name-tool -delete_rpath /opt/llvm17/lib /opt/llvm18/bin/clang-18
llvm-install-name-tool -delete_rpath /opt/llvm17/lib /opt/llvm18/bin/lld
llvm-install-name-tool -change @rpath/libc++.1.dylib /opt/llvm18/lib/libc++.1.dylib /opt/llvm18/bin/clang-18
llvm-install-name-tool -change @rpath/libc++abi.1.dylib /opt/llvm18/lib/libc++abi.1.dylib /opt/llvm18/bin/clang-18
llvm-install-name-tool -change @rpath/libc++.1.dylib /opt/llvm18/lib/libc++.1.dylib /opt/llvm18/bin/lld
llvm-install-name-tool -change @rpath/libc++abi.1.dylib /opt/llvm18/lib/libc++abi.1.dylib /opt/llvm18/bin/lld

But, when I do that, (or, if I deploy my working llvm17 to /opt/llvm18, compile llvm18 with the /opt/llvm18/bin/clang-17, and then overwrite the previous /opt/llvm18 installation directory with llvm 18.1.0), I get this...

 $ /opt/llvm18/bin/clang++ ~/backup/cmake_bootstrap.cxx
dyld[4913]: Symbol not found: ___cxa_pure_virtual
  Referenced from: <4C4C4476-5555-3144-A184-84B6267412E1> /opt/llvm18/bin/clang-18
  Expected in:     <4C4C4429-5555-3144-A139-01A02C015AD3> /opt/llvm18/lib/libc++.1.0.dylib
Abort trap: 6

hoyhoy · 2024-03-08T18:26:40Z

Looks like libunwind is the problem actually.

5187/0x15e640:      1596       4      3 open("@rpath/libunwind.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
 5187/0x15e640:      1598       2      1 open("@rpath\0", 0x100000, 0x0)		 = -1 Err#2
 5187/0x15e640:      1604       3      2 stat64("/opt/llvm17/lib/libunwind.1.dylib\0", 0x7FF7BCFB6DA0, 0x0)		 = -1 Err#2
 5187/0x15e640:      1615       4      2 stat64("/System/Volumes/Preboot/Cryptexes/OS/opt/llvm17/lib/libunwind.1.dylib\0", 0x7FF7BCFB6DA0, 0x0)		 = -1 Err#2
 5187/0x15e640:      1626       9      8 stat64("/opt/llvm18/lib/libunwind.1.dylib\0", 0x7FF7BCFB6DA0, 0x0)		 = 0 0
 5187/0x15e640:      1632       5      4 stat64("/opt/llvm18/lib/libunwind.1.dylib\0", 0x7FF7BCFB6800, 0x0)		 = 0 0
 5187/0x15e640:      1659      26     25 open("/opt/llvm18/lib/libunwind.1.dylib\0", 0x0, 0x0)		 = 3 0
 5187/0x15e640:      1665       7      5 mmap(0x0, 0x106C0, 0x1, 0x40002, 0x3, 0x0)		 = 0x109EA5000 0
 5187/0x15e640:      1667       2      1 fcntl(0x3, 0x32, 0x7FF7BCFB6910)		 = 0 0
 5187/0x15e640:      1669       1      0 close(0x3)		 = 0 0
 5187/0x15e640:      1687      14     13 open("/opt/llvm18/lib/libunwind.1.0.dylib\0", 0x0, 0x0)		 = 3 0
 5187/0x15e640:      1692       5      3 fcntl(0x3, 0x62, 0x7FF7BCFB65C8)		 = 0 0
 5187/0x15e640:      1704      11      9 mmap(0x109EB6000, 0xC000, 0x5, 0x40012, 0x3, 0x0)		 = 0x109EB6000 0
 5187/0x15e640:      1709       4      2 mmap(0x109EC2000, 0x1000, 0x3, 0x40012, 0x3, 0xC000)		 = 0x109EC2000 0
 5187/0x15e640:      1712       3      2 mmap(0x109EC3000, 0x1000, 0x3, 0x40012, 0x3, 0xD000)		 = 0x109EC3000 0
 5187/0x15e640:      1714       3      1 mmap(0x109EC4000, 0x26C0, 0x1, 0x40012, 0x3, 0xE000)		 = 0x109EC4000 0
 5187/0x15e640:      1716       1      0 close(0x3)		 = 0 0
 5187/0x15e640:      1724       3      2 munmap(0x109EA5000, 0x106C0)		 = 0 0
 5187/0x15e640:      1900     107     43 open("/dev/dtracehelper\0", 0x2, 0x0)		 = 3 0
 5187/0x15e640:      2271     375    369 ioctl(0x3, 0x80086804, 0x7FF7BCFB68A8)		 = 0 0
 5187/0x15e640:      2275       4      2 close(0x3)		 = 0 0
 5187/0x15e640:      2506       6      4 write(0x2, "dyld[5187]: \0", 0xC)		 = 12 0
 5187/0x15e640:      2508       2      0 write(0x2, "Symbol not found: ___cxa_pure_vi\0", 0x20)		 = 32 0
 5187/0x15e640:      2509       1      0 write(0x2, "rtual\n  Referenced from: <4C4C44\0", 0x20)		 = 32 0
 5187/0x15e640:      2511       1      0 write(0x2, "76-5555-3144-A184-84B6267412E1> \0", 0x20)		 = 32 0
 5187/0x15e640:      2512       1      0 write(0x2, "/opt/llvm18/bin/clang-18\n  Expec\0", 0x20)		 = 32 0
 5187/0x15e640:      2514       2      0 write(0x2, "ted in:     <4C4C4429-5555-3144-\0", 0x20)		 = 32 0
 5187/0x15e640:      2515       1      0 write(0x2, "A139-01A02C015AD3> /opt/llvm18/l\0", 0x20)		 = 32 0
 5187/0x15e640:      2517       1      0 write(0x2, "ib/libc++.1.0.dylib\n\0", 0x14)		 = 20 0
 5187/0x15e640:      2519       2      0 sigprocmask(0x2, 0x7FF7BCFB703C, 0x0)		 = 0x0 0
 5187/0x15e640:      2526       8      5 abort_with_payload(0x6, 0x4, 0x7FF7BCFB7490)		 = 0 0

hoyhoy · 2024-03-08T18:56:44Z

I tried disabling libunwind, and this happens...

8986/0x17cdff:      1411      14     13 open("/opt/llvm18/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
38986/0x17cdff:      1416       5      4 fcntl(0x3, 0x62, 0x7FF7BE9B9768)		 = 0 0
38986/0x17cdff:      1429      11      9 mmap(0x108C1F000, 0xA4000, 0x5, 0x40012, 0x3, 0x0)		 = 0x108C1F000 0
38986/0x17cdff:      1435       4      2 mmap(0x108CC3000, 0x5000, 0x3, 0x40012, 0x3, 0xA4000)		 = 0x108CC3000 0
38986/0x17cdff:      1438       4      2 mmap(0x108CC8000, 0x2000, 0x3, 0x40012, 0x3, 0xA9000)		 = 0x108CC8000 0
38986/0x17cdff:      1440       3      1 mmap(0x108CD0000, 0x56C88, 0x1, 0x40012, 0x3, 0xAB000)		 = 0x108CD0000 0
38986/0x17cdff:      1442       1      0 close(0x3)		 = 0 0
38986/0x17cdff:      1451       4      2 munmap(0x108B1D000, 0x101C88)		 = 0 0
38986/0x17cdff:      1472       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
38986/0x17cdff:      1474       2      1 open("@rpath\0", 0x100000, 0x0)		 = -1 Err#2
38986/0x17cdff:      1485       9      7 stat64("/opt/llvm18/lib/libc++abi.1.dylib\0", 0x7FF7BE9B9E60, 0x0)		 = 0 0
38986/0x17cdff:      1491       5      4 stat64("/opt/llvm18/lib/libc++abi.1.dylib\0", 0x7FF7BE9B98C0, 0x0)		 = 0 0
38986/0x17cdff:      1517      26     25 open("/opt/llvm18/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
38986/0x17cdff:      1528      11      9 mmap(0x0, 0x4B7F0, 0x1, 0x40002, 0x3, 0x0)		 = 0x1084A1000 0
38986/0x17cdff:      1530       2      0 fcntl(0x3, 0x32, 0x7FF7BE9B99D0)		 = 0 0
38986/0x17cdff:      1531       2      0 close(0x3)		 = 0 0
38986/0x17cdff:      1549      14     13 open("/opt/llvm18/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
38986/0x17cdff:      1554       4      3 fcntl(0x3, 0x62, 0x7FF7BE9B9688)		 = 0 0
38986/0x17cdff:      1566      11     10 mmap(0x108901000, 0x2A000, 0x5, 0x40012, 0x3, 0x0)		 = 0x108901000 0
38986/0x17cdff:      1571       3      2 mmap(0x10892B000, 0x4000, 0x3, 0x40012, 0x3, 0x2A000)		 = 0x10892B000 0
38986/0x17cdff:      1574       3      1 mmap(0x10892F000, 0x1000, 0x3, 0x40012, 0x3, 0x2E000)		 = 0x10892F000 0
38986/0x17cdff:      1576       3      1 mmap(0x108930000, 0x1C7F0, 0x1, 0x40012, 0x3, 0x2F000)		 = 0x108930000 0
38986/0x17cdff:      1578       1      0 close(0x3)		 = 0 0
38986/0x17cdff:      1586       3      2 munmap(0x1084A1000, 0x4B7F0)		 = 0 0
38986/0x17cdff:      1747     106     43 open("/dev/dtracehelper\0", 0x2, 0x0)		 = 3 0
38986/0x17cdff:      2104     358    356 ioctl(0x3, 0x80086804, 0x7FF7BE9B99C8)		 = 0 0
38986/0x17cdff:      2108       4      2 close(0x3)		 = 0 0
38986/0x17cdff:      2332       6      4 write(0x2, "dyld[38986]: \0", 0xD)		 = 13 0
38986/0x17cdff:      2334       2      1 write(0x2, "Symbol not found: ___cxa_pure_vi\0", 0x20)		 = 32 0
38986/0x17cdff:      2336       1      0 write(0x2, "rtual\n  Referenced from: <4C4C44\0", 0x20)		 = 32 0
38986/0x17cdff:      2337       1      0 write(0x2, "76-5555-3144-A184-84B6267412E1> \0", 0x20)		 = 32 0
38986/0x17cdff:      2339       1      0 write(0x2, "/opt/llvm18/bin/clang-18\n  Expec\0", 0x20)		 = 32 0
38986/0x17cdff:      2340       1      0 write(0x2, "ted in:     <4C4C444F-5555-3144-\0", 0x20)		 = 32 0
38986/0x17cdff:      2341       1      0 write(0x2, "A1AE-E57703862411> /opt/llvm18/l\0", 0x20)		 = 32 0
38986/0x17cdff:      2343       1      0 write(0x2, "ib/libc++.1.0.dylib\n\0", 0x14)		 = 20 0
38986/0x17cdff:      2345       2      0 sigprocmask(0x2, 0x7FF7BE9BA11C, 0x0)		 = 0x0 0
38986/0x17cdff:      2353       9      7 abort_with_payload(0x6, 0x4, 0x7FF7BE9BA570)		 = 0 0

hoyhoy · 2024-03-09T18:11:42Z

OK, with your patch, everything works! Thanks a lot.

I ended up having some issues where I was pulling in binaries from /usr/local compiled with previous versions of clang. I just recompiled with the latest XCode 15.3 on macOS 14.4 with mostly everything set to default using libcxx and libcxxabi.

Here's my build script for anyone else who is seeing these issues. My problems were RPATH related. I had to remove a libzstd that was linked to a libc++ that no longer existed.

#!/bin/sh

BUILD_DIR=build

rm -Rf ${BUILD_DIR}
mkdir ${BUILD_DIR}

export PATH="/usr/bin:/bin"
INSTALL_PATH="/opt/llvm18"

MACOS_SDK_ROOT=$(xcrun --show-sdk-path)
MACOS_VERSION=$(sw_vers -productVersion)

unset ASAN_OPTIONS
unset MSAN_OPTIONS
unset TSAN_OPTIONS
unset DYLD_LIBRARY_PATH
unset LD_LIBRARY_PATH
unset LIBPATH
unset LIBRARY_PATH
unset CPLUS_INCLUDE_PATH
unset CPATH

cmake_args=(
  -DCMAKE_BUILD_TYPE:STRING=Release
  -DLLVM_ENABLE_RUNTIMES:STRING="libcxx;libcxxabi;compiler-rt"
  -DLLVM_ENABLE_PROJECTS:STRING="clang;clang-tools-extra;lldb;lld"
  
  -DDEFAULT_SYSROOT:PATH=${MACOS_SDK_ROOT}

  -DLLVM_ENABLE_LIBCXX:BOOL=ON
  -DLIBCXXABI_USE_COMPILER_RT:BOOL=ON
  -DCLANG_DEFAULT_CXX_STDLIB:STRING=libc++

  -DCLANG_CONFIG_FILE_SYSTEM_DIR:PATH="${INSTALL_PATH}/etc"

  -DCMAKE_INSTALL_PREFIX:PATH="${INSTALL_PATH}"
  -DLLVM_INSTALL_BINUTILS_SYMLINKS:BOOL=ON
  -DLLVM_INSTALL_CCTOOLS_SYMLINKS:BOOL=ON

  -DCLANG_INCLUDE_TESTS:BOOL=OFF
  -DLLVM_INCLUDE_TESTS:BOOL=OFF
  -DLLDB_ENABLE_LIBEDIT:BOOL=ON
  -DLLDB_ENABLE_CURSES:BOOL=ON
  -DLLVM_ENABLE_TERMINFO:BOOL=ON

  -DLLVM_BUILD_TOOLS:BOOL=ON

  -DHAVE_CXX_ATOMICS64_WITHOUT_LIB:BOOL=ON
  -DHAVE_CXX_ATOMICS_WITHOUT_LIB:BOOL=ON

  -DLLVM_PARALLEL_COMPILE_JOBS:STRING=17
  -DLLVM_PARALLEL_LINK_JOBS:STRING=2

  -DLLVM_ENABLE_LLD:BOOL=ON

  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING="${MACOS_VERSION}"

  -DCOMPILER_RT_BUILD_BUILTINS:BOOL=ON
  -DCOMPILER_RT_ENABLE_IOS:BOOL=OFF
  -DCOMPILER_RT_ENABLE_WATCHOS:BOOL=OFF
  -DCOMPILER_RT_ENABLE_TVOS:BOOL=OFF
  -DCOMPILER_RT_ENABLE_MACCATALYST:BOOL=OFF

  -DCOMPILER_RT_BUILD_LIBFUZZER:BOOL=ON
  -DCOMPILER_RT_BUILD_MEMPROF:BOOL=ON
  -DCOMPILER_RT_BUILD_PROFILE:BOOL=ON
  -DCOMPILER_RT_BUILD_SANITIZERS:BOOL=ON
  -DCOMPILER_RT_BUILD_XRAY:BOOL=ON

  -DLIBCXX_CXX_ABI:STRING=libcxxabi
  -DLIBCXX_USE_COMPILER_RT:BOOL=ON
  -DLIBCXXABI_USE_COMPILER_RT:BOOL=ON
  -DLIBCXX_HAS_GCC_S_LIB:BOOL=OFF

  -DLLVM_ENABLE_RTTI:BOOL=ON
  -DLLVM_ENABLE_FFI:BOOL=ON
)

cmake "${cmake_args[@]}" -S llvm -G Ninja -B "${BUILD_DIR}" && ninja -C "${BUILD_DIR}"

hoyhoy · 2024-03-09T18:14:52Z

Also, if your clang-18 is having libc++ linking issues, you can just override the load path with the following...

install_name_tool -change /usr/lib/libc++.1.dylib /opt/llvm18/lib/libc++.1.dylib /opt/llvm18/bin/clang-18

(where /opt/llvm18 is your CMAKE_INSTALL_PREFIX, normally /usr/local).

This is the issue that @ldionne was talking about. With the bootstrapped build (i.e. building the runtimes and the projects with a single ninja or cmake --build), clang-18 load path points to the MacOS platform libc++ (i.e. /usr/lib/libc++.1.dylib), and NOT llvm-18.x's libc++ (i.e. ${CMAKE_INSTALL_PREFIX}/lib/libc++.1.dylib).

If you look at the binary with otool -L, it refer to the libc++ built from the llvm source code. clang-18 and lld's LC_RPATH should point to your CMAKE_INSTALL_PREFIX as well.

 $ otool -L /opt/llvm18/bin/clang-18
/opt/llvm18/bin/clang-18:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/opt/xcode15/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/opt/llvm18/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.0)

$ otool -l /opt/llvm18/bin/clang-18
# relevant lines only...
Load command 18
  cmd LC_LOAD_DYLIB
  name /opt/llvm18/lib/libc++.1.dylib (offset 24)
  compatibility version 1.0.0
cmd LC_RPATH
  cmdsize 32
  path /opt/llvm18/lib (offset 12)
Load command 22
  cmd LC_RPATH
  path /opt/llvm18/lib/darwin (offset 12)

The other odd thing I've been seeing is that the boostrap build doesn't deploy lib/darwin RPATH. I end up having to manually symbolically link that after ninja install.

ln -s /opt/llvm18/lib/clang/18/lib/darwin /opt/llvm18/lib/darwin

If I compile the runtimes separately and install them separately, the installation does deploy to lib/darwin, but not lib/clang/18/lib/darwin. Not sure if there is a cmake setting for that or not. Internally, clang seems to need both installed directories for some reason.

hoyhoy · 2024-04-10T08:18:46Z

Just compiled 18.1.3, and it still has issues. Is x86_64 not supported on macOS at this point? It's nearly impossible to get this to install correctly. Also, this diff isn't in 18.1.3. What's the best way to build at this point? Use XCode 15 to build clang 18.1.3, and then clang 18.1.3 to compile the runtimes in two steps?

ldionne · 2024-04-11T21:32:48Z

@hoyhoy I would like to kindly ask that you stop pinging this issue without additional information. We are aware of this bug, we understand it and we will get to it eventually. In the meantime, posting walls of text here is just making things more confusing. So let's please restrict this issue to new and useful information.

hoyhoy · 2024-04-12T01:02:18Z

OK, if you're aware of the issue and it's not fixed, and you haven't even merged the fix, why are you closing the issue?

Just a helpful hint -- here's how LITERALLY EVERY OTHER SOFTWARE PROJECT DOES IT...

(1) Fix issue.
(2) Merge fix
(3) Verify fix
(4) Close Issue

hoyhoy · 2024-04-12T01:05:00Z

I posted all the errors in case anyone else had the problem.

As for the solution, what worked for me is building the all the runtimes in a separate stage and manually specifying the RPATHs to my INSTALLed PATH.

  -DCMAKE_INSTALL_RPATH:FILEPATH="${INSTALL_PATH}/lib;${INSTALL_PATH}/lib/darwin"
  -DCMAKE_INSTALL_NAME_DIR="${INSTALL_PATH}/lib"

hoyhoy · 2024-04-12T01:06:03Z

@ldionne thanks again for the non-help!

ldionne · 2024-04-12T14:13:35Z

I closed #84392 cause it's a duplicate of this, and having duplicate issues lying around doesn't help anyone, it just creates confusion.

And then you proceeded to spam the hell out of this thread. Please don't interact here again until you've learned to do it properly.

sztomi · 2024-04-29T19:30:28Z

For a couple of different reasons, I utilize a compiler wrapper script on macOS. An easy workaround that can be done without patching / recompiling clang is to simply add -L <the clang lib dir> . I'd imagine this works equally well with a .cfg file.

hoyhoy · 2024-04-29T21:14:11Z

@sztomi I created a x86_64-apple-darwin23.4.0-clang.cfg and x86_64-apple-darwin23.4.0-clang++.cfg, but certain builds (like boost) somehow manage to disable it. This is what I'm doing for C++.

# x86_64-apple-darwin23.4.0-clang++.cfg
-fuse-ld=lld -Wl,-rpath,/opt/llvm18.1.4/lib -Wl,-rpath,/opt/llvm18.1.4/lib/darwin -lc++abi

I don't understand why I have to specifically link -lc++abi with -lc++. This never used to be the case. Building the runtimes and lldb correctly is nearly impossible on MacOS 14.4 with x86 right now. It looks like there's a LIBCXX_ENABLE_ABI_LINKER_SCRIPT boolean I'm not setting. But that's disabled on MacOS by default for some reason. It used to not be needed, but I guess now it is?

The latter temporary workarounds llvm/llvm-project#77653 (and llvm/llvm-project#84392)

themightyoarfish · 2024-05-07T09:52:43Z

I'm not sure this is the same problem, but I'm trying to compile things with llvm 18.1.5 on macos 12.6.5 (ARM) and still run into this after upgrading the compiler:

/opt/homebrew/Cellar/llvm/18.1.5/bin/clang++ test.cpp -o test
Undefined symbols for architecture arm64:
  "std::exception_ptr::__from_native_exception_pointer(void*)", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:ne180100]<std::__1::future_error>(std::__1::future_error) in test-a014c2.o
  "___cxa_init_primary_exception", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:ne180100]<std::__1::future_error>(std::__1::future_error) in test-a014c2.o
ld: symbol(s) not found for architecture arm64

With test.cpp

#include <future>
#include <memory>

int main(int argc, char *argv[]) {
  std::promise<std::shared_ptr<int>> p;
  p.set_value(std::make_shared<int>(5));
  return 0;
}

Can I do anything to work around this?

FeignClaims · 2024-05-07T09:58:11Z

I'm not sure this is the same problem, but I'm trying to compile things with llvm 18.1.5 on macos 12.6.5 (ARM) and still run into this after upgrading the compiler:

/opt/homebrew/Cellar/llvm/18.1.5/bin/clang++ test.cpp -o test
Undefined symbols for architecture arm64:
  "std::exception_ptr::__from_native_exception_pointer(void*)", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:ne180100]<std::__1::future_error>(std::__1::future_error) in test-a014c2.o
  "___cxa_init_primary_exception", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:ne180100]<std::__1::future_error>(std::__1::future_error) in test-a014c2.o
ld: symbol(s) not found for architecture arm64

With test.cpp

#include <future>
#include <memory>

int main(int argc, char *argv[]) {
  std::promise<std::shared_ptr<int>> p;
  p.set_value(std::make_shared<int>(5));
  return 0;
}

Can I do anything to work around this?

I'm currently using the workaround commented here.

hoyhoy · 2024-05-07T20:12:59Z

@ldionne thankfully, I "spammed" the issue with my error log.

hoyhoy · 2024-05-07T20:14:37Z

The runtime cmake for llvm has so many bugs. I'm still having the hard-link -lc++abi even though all my rpaths are correct.

HerrCai0907 · 2024-05-08T05:55:35Z

Could this feature enable via command line option?
In my understanding, the header and libc++ in MacOSX.sdk is tested by apple and relative stable, but the just-built libc++ is unstable and not tested in all versions of OSX. It will waste lots of effort to debug incompatibility between libc++ and operator system for most of non-libc++ developer.

carlocab · 2024-05-30T01:36:43Z

I don't understand why I have to specifically link -lc++abi with -lc++. This never used to be the case. Building the runtimes and lldb correctly is nearly impossible on MacOS 14.4 with x86 right now. It looks like there's a LIBCXX_ENABLE_ABI_LINKER_SCRIPT boolean I'm not setting. But that's disabled on MacOS by default for some reason. It used to not be needed, but I guess now it is?

Yes, this seems to have changed in a recent version of LLVM. We avoid this in Homebrew by making sure the appropriate -rpath flags are passed when building the runtimes. These snippets might help:

https://github.com/Homebrew/homebrew-core/blob/cfa748fd18b9495aad647c8ada4304bc540baa52/Formula/l/llvm.rb#L145-L146
https://github.com/Homebrew/homebrew-core/blob/cfa748fd18b9495aad647c8ada4304bc540baa52/Formula/l/llvm.rb#L154
https://github.com/Homebrew/homebrew-core/blob/cfa748fd18b9495aad647c8ada4304bc540baa52/Formula/l/llvm.rb#L216

hoyhoy · 2024-05-30T13:21:42Z

@carlocab thank you! I was actually looking for those. I based my script somewhat on this.

https://src.fedoraproject.org/rpms/clang/blob/rawhide/f/clang.spec

The last problem I don't understand is that the debugserver isn't working (yes, it's signed) and lldb now crashes if UBSAN throws a runtime an error. Also, liblldb.18.1.6.dylib appears to not be in the RPATH either. I'm having to manually manually change it with this after install.

llvm-install-name-tool -change liblldb.18.1.6.dylib /opt/llvm18.1.6/lib/liblldb.18.1.6.dylib /opt/llvm18.1.6/bin/lldb

My one-stage build script is here:
https://gist.github.com/hoyhoy/acbc0c3f6bff8eb95fce5ec945e33408

ldionne added the clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' label Jan 10, 2024

ldionne assigned fahadnayyar Jan 10, 2024

ldionne added the phabricator Related to migration from Phabricator label Jan 10, 2024

ldionne mentioned this issue Mar 8, 2024

libc++ 18.1.0 fails with BOOST_THROW_EXCEPTION (boost/1.84.0) #84392

Closed

ldionne mentioned this issue Mar 26, 2024

llvm 18.1.2 / x86_64-apple-darwin23.4.0: "undefined symbol __from_native_exception_pointer undefined symbol: __cxa_init_primary_exception" #86077

Closed

ldionne mentioned this issue Apr 11, 2024

Undefined symbols "std::__1::__libcpp_verbose_abort(char const*, ...)" #87590

Closed

llvm locked as too heated and limited conversation to collaborators Apr 12, 2024

ZhongRuoyu mentioned this issue Apr 23, 2024

Compilation Failure with Clang from Brewed LLVM Homebrew/homebrew-core#169820

Closed

4 tasks

llvm unlocked this conversation Apr 29, 2024

asl added a commit to ablab/spades that referenced this issue Apr 30, 2024

Be explicit with OS settings and LLVM version.

991f889

The latter temporary workarounds llvm/llvm-project#77653 (and llvm/llvm-project#84392)

asl added a commit to ablab/spades that referenced this issue Apr 30, 2024

Be explicit with OS settings and LLVM version.

bc06798

The latter temporary workarounds llvm/llvm-project#77653 (and llvm/llvm-project#84392)

kevinushey mentioned this issue May 29, 2024

symbol(s) not found for architecture arm64 RcppCore/RcppParallel#218

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[D148266] [clang][driver] Linking to just-built libc++.dylib when bootstrapping libc++ with clang #77653

[D148266] [clang][driver] Linking to just-built libc++.dylib when bootstrapping libc++ with clang #77653

ldionne commented Jan 10, 2024

llvmbot commented Jan 10, 2024

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 8, 2024 •

edited

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 8, 2024 •

edited

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 9, 2024

hoyhoy commented Mar 9, 2024 •

edited

hoyhoy commented Apr 10, 2024 •

edited

ldionne commented Apr 11, 2024

hoyhoy commented Apr 12, 2024

hoyhoy commented Apr 12, 2024

hoyhoy commented Apr 12, 2024

ldionne commented Apr 12, 2024 •

edited

sztomi commented Apr 29, 2024

hoyhoy commented Apr 29, 2024 •

edited

themightyoarfish commented May 7, 2024

FeignClaims commented May 7, 2024 •

edited

hoyhoy commented May 7, 2024

hoyhoy commented May 7, 2024

HerrCai0907 commented May 8, 2024 •

edited

carlocab commented May 30, 2024

hoyhoy commented May 30, 2024

[D148266] [clang][driver] Linking to just-built libc++.dylib when bootstrapping libc++ with clang #77653

[D148266] [clang][driver] Linking to just-built libc++.dylib when bootstrapping libc++ with clang #77653

Comments

ldionne commented Jan 10, 2024

llvmbot commented Jan 10, 2024

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 8, 2024 • edited

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 8, 2024 • edited

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 8, 2024

hoyhoy commented Mar 9, 2024

hoyhoy commented Mar 9, 2024 • edited

hoyhoy commented Apr 10, 2024 • edited

ldionne commented Apr 11, 2024

hoyhoy commented Apr 12, 2024

hoyhoy commented Apr 12, 2024

hoyhoy commented Apr 12, 2024

ldionne commented Apr 12, 2024 • edited

sztomi commented Apr 29, 2024

hoyhoy commented Apr 29, 2024 • edited

themightyoarfish commented May 7, 2024

FeignClaims commented May 7, 2024 • edited

hoyhoy commented May 7, 2024

hoyhoy commented May 7, 2024

HerrCai0907 commented May 8, 2024 • edited

carlocab commented May 30, 2024

hoyhoy commented May 30, 2024

hoyhoy commented Mar 8, 2024 •

edited

hoyhoy commented Mar 8, 2024 •

edited

hoyhoy commented Mar 9, 2024 •

edited

hoyhoy commented Apr 10, 2024 •

edited

ldionne commented Apr 12, 2024 •

edited

hoyhoy commented Apr 29, 2024 •

edited

FeignClaims commented May 7, 2024 •

edited

HerrCai0907 commented May 8, 2024 •

edited