Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to compile on aarch64 #67251

Open
hectorj2f opened this issue May 9, 2024 · 1 comment
Open

Failed to compile on aarch64 #67251

hectorj2f opened this issue May 9, 2024 · 1 comment
Assignees
@tilakrayal
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.16 type:bug Bug type:build/install Build and install issues

Comments

@hectorj2f
Copy link

hectorj2f commented May 9, 2024 •
edited

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

2.16.1

Custom code

No

OS platform and distribution

aarch64

Mobile device

No response

Python version

3.10

Bazel version

bazelisk using the default version requested by tensorflow

GCC/compiler version

clang 17 (I also tried with clang16).

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

When compiling the tensorflow source code on aarch64, I always get the following error:

WARN ERROR: /home/build/tensorflow/BUILD:1263:20: Linking tensorflow/libtensorflow.so.2.16.1 failed: (Exit 1): clang-17 failed: error executing command (from target //tensorflow:libtensorflow.so.2.16.1) /usr/bin/clang-17 @bazel-out/aarch64-opt/bin/tensorflow/libtensorflow.so.2.16.1-2.params

However the compilation works on x86_64.

export PYTHON_BIN_PATH=/usr/bin/python
      export TF_PYTHON_VERSION=3.10
      export USE_DEFAULT_PYTHON_LIB_PATH=1
      export TF_NEED_JEMALLOC=1
      export TF_NEED_KAFKA=1
      export TF_NEED_OPENCL_SYCL=0
      export TF_NEED_AWS=1
      export TF_NEED_GCP=1
      export TF_NEED_HDFS=1
      export TF_NEED_S3=1
      export TF_ENABLE_XLA=1
      export TF_NEED_GDR=0
      export TF_NEED_VERBS=0
      export TF_NEED_OPENCL=0
      export TF_NEED_MPI=0
      export TF_NEED_TENSORRT=0
      export TF_NEED_NGRAPH=0
      export TF_NEED_IGNITE=0
      export TF_NEED_ROCM=0
      export TF_SYSTEM_LIBS="boringssl,curl,gif,icu,libjpeg_turbo,nasm,png,zlib"
      export TF_SET_ANDROID_WORKSPACE=0

      ./configure

      bazel --bazelrc=.tf_configure.bazelrc build \
        --config=opt \
        --config=mkl_threadpool \
        //tensorflow:libtensorflow.so \
        //tensorflow:libtensorflow_cc.so \
        //tensorflow:install_headers \
        //tensorflow/tools/pip_package:build_pip_package

Standalone code to reproduce the issue

export PYTHON_BIN_PATH=/usr/bin/python
      export TF_PYTHON_VERSION=3.10
      export USE_DEFAULT_PYTHON_LIB_PATH=1
      export TF_NEED_JEMALLOC=1
      export TF_NEED_KAFKA=1
      export TF_NEED_OPENCL_SYCL=0
      export TF_NEED_AWS=1
      export TF_NEED_GCP=1
      export TF_NEED_HDFS=1
      export TF_NEED_S3=1
      export TF_ENABLE_XLA=1
      export TF_NEED_GDR=0
      export TF_NEED_VERBS=0
      export TF_NEED_OPENCL=0
      export TF_NEED_MPI=0
      export TF_NEED_TENSORRT=0
      export TF_NEED_NGRAPH=0
      export TF_NEED_IGNITE=0
      export TF_NEED_ROCM=0
      export TF_SYSTEM_LIBS="boringssl,curl,gif,icu,libjpeg_turbo,nasm,png,zlib"
      export TF_SET_ANDROID_WORKSPACE=0

      ./configure

      bazel --bazelrc=.tf_configure.bazelrc build \
        --config=opt \
        --config=mkl_threadpool \
        //tensorflow:libtensorflow.so \
        //tensorflow:libtensorflow_cc.so \
        //tensorflow:install_headers \
        //tensorflow/tools/pip_package:build_pip_package

Relevant log output

2024/05/09 13:03:51 WARN [14,447 / 14,578] Compiling tensorflow/compiler/jit/xla_platform_info.cc; 20s local ... (16 actions running)
2024/05/09 13:03:52 WARN [14,448 / 14,578] Compiling tensorflow/compiler/jit/xla_platform_info.cc; 21s local ... (16 actions running)
2024/05/09 13:03:55 WARN [14,449 / 14,578] Compiling tensorflow/compiler/jit/xla_platform_info.cc; 24s local ... (16 actions, 15 running)
2024/05/09 13:03:57 WARN [14,450 / 14,578] Compiling tensorflow/compiler/jit/xla_platform_info.cc; 26s local ... (16 actions running)
2024/05/09 13:03:58 WARN [14,452 / 14,578] Compiling tensorflow/compiler/jit/get_compiler_ir.cc; 26s local ... (16 actions running)
2024/05/09 13:04:00 WARN [14,454 / 14,578] Compiling tensorflow/compiler/jit/kernels/xla_ops.cc; 27s local ... (16 actions running)
2024/05/09 13:04:01 WARN [14,457 / 14,578] Compiling tensorflow/compiler/jit/xla_cpu_device.cc; 22s local ... (16 actions running)
2024/05/09 13:04:03 WARN [14,459 / 14,578] Compiling tensorflow/compiler/jit/xla_cpu_device.cc; 24s local ... (16 actions running)
2024/05/09 13:04:04 WARN [14,460 / 14,578] Compiling tensorflow/compiler/jit/xla_cpu_device.cc; 25s local ... (16 actions, 15 running)
2024/05/09 13:04:06 WARN [14,461 / 14,578] Compiling tensorflow/compiler/jit/xla_cpu_device.cc; 27s local ... (16 actions running)
2024/05/09 13:04:07 WARN [14,464 / 14,578] Compiling tensorflow/compiler/tf2xla/mlir_tf2xla.cc; 22s local ... (16 actions running)
2024/05/09 13:04:11 WARN [14,465 / 14,578] Compiling tensorflow/compiler/tf2xla/mlir_tf2xla.cc; 26s local ... (16 actions, 15 running)
2024/05/09 13:04:12 WARN [14,467 / 14,578] Compiling tensorflow/compiler/tf2xla/mlir_tf2xla.cc; 27s local ... (16 actions, 15 running)
2024/05/09 13:04:13 WARN [14,468 / 14,578] Compiling tensorflow/compiler/tf2xla/mlir_tf2xla.cc; 28s local ... (16 actions running)
2024/05/09 13:04:14 WARN [14,470 / 14,578] Compiling tensorflow/compiler/tf2xla/mlir_tf2xla.cc; 29s local ... (16 actions, 15 running)
2024/05/09 13:04:16 WARN [14,473 / 14,578] Compiling tensorflow/compiler/aot/codegen.cc; 19s local ... (16 actions, 15 running)
2024/05/09 13:04:17 WARN [14,476 / 14,578] Compiling tensorflow/core/common_runtime/pluggable_device/pluggable_device_plugin_init.cc; 16s local ... (16 actions running)
2024/05/09 13:04:18 WARN [14,478 / 14,578] Compiling tensorflow/core/common_runtime/pluggable_device/pluggable_device_plugin_init.cc; 17s local ... (16 actions, 15 running)
2024/05/09 13:04:20 WARN [14,479 / 14,578] Compiling tensorflow/core/common_runtime/pluggable_device/pluggable_device_plugin_init.cc; 19s local ... (16 actions, 15 running)
2024/05/09 13:04:21 WARN [14,481 / 14,578] Compiling tensorflow/core/common_runtime/pluggable_device/pluggable_device_plugin_init.cc; 20s local ... (16 actions, 14 running)
2024/05/09 13:04:22 WARN [14,482 / 14,578] Compiling tensorflow/core/common_runtime/pluggable_device/pluggable_device_plugin_init.cc; 21s local ... (16 actions, 15 running)
2024/05/09 13:04:23 WARN [14,486 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 21s local ... (16 actions running)
2024/05/09 13:04:25 WARN [14,487 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 23s local ... (16 actions running)
2024/05/09 13:04:26 WARN [14,489 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 24s local ... (16 actions running)
2024/05/09 13:04:28 WARN [14,491 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 25s local ... (16 actions, 15 running)
2024/05/09 13:04:29 WARN [14,492 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 27s local ... (16 actions, 15 running)
2024/05/09 13:04:30 WARN [14,494 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 28s local ... (16 actions, 15 running)
2024/05/09 13:04:31 WARN [14,497 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 29s local ... (14 actions running)
2024/05/09 13:04:32 WARN [14,498 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 30s local ... (13 actions running)
2024/05/09 13:04:34 WARN [14,500 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 32s local ... (11 actions running)
2024/05/09 13:04:36 WARN [14,505 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 34s local ... (6 actions running)
2024/05/09 13:04:38 WARN [14,506 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 36s local ... (5 actions running)
2024/05/09 13:04:40 WARN [14,508 / 14,578] Compiling tensorflow/compiler/mlir/python/mlir.cc; 38s local ... (3 actions running)
2024/05/09 13:04:40 WARN ERROR: /home/build/tensorflow/BUILD:1263:20: Linking tensorflow/libtensorflow.so.2.16.1 failed: (Exit 1): clang-17 failed: error executing command (from target //tensorflow:libtensorflow.so.2.16.1) /usr/bin/clang-17 @bazel-out/aarch64-opt/bin/tensorflow/libtensorflow.so.2.16.1-2.params
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN /usr/bin/../lib64/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld.gold: error: Stub is too far away, try a smaller value for '--stub-group-size'. The current value is 0x7ffbffc.
2024/05/09 13:04:40 WARN clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
2024/05/09 13:04:41 WARN INFO: Elapsed time: 4130.234s, Critical Path: 216.57s
2024/05/09 13:04:41 WARN INFO: 14511 processes: 1423 internal, 13088 local.
2024/05/09 13:04:41 WARN FAILED: Build did NOT complete successfully
@google-ml-butler google-ml-butler bot added the type:bug Bug label May 9, 2024
@tilakrayal tilakrayal added TF 2.16 type:build/install Build and install issues labels May 10, 2024
@tilakrayal
Copy link
Contributor

@hectorj2f,
Thanks for reporting the issue. Looks like this is a known issue while building the TensorFlow v2.16 with aarch64.
Could you please try to refer to the issue for the updates on the same.

#62490

Thank you!

@tilakrayal tilakrayal added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tilakrayal tilakrayal

Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.16 type:bug Bug type:build/install Build and install issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hectorj2f @tilakrayal