Failed to build benchmark "ModuleNotFoundError: No module named 'package_build'" #12023

happyhust · 2024-05-13T08:00:51Z

What happens?

I want to run benchmark under the repo

My test steps:

cd duckdb
cd tools/pythonpkg/
pip3 install duckdb --user
pip3 install -r requirements-dev.txt --user

cd ../../
mkdir build
cd build
BUILD_BENCHMARK=1 BUILD_TPCH=1 BUILD_PYTHON=1 cmake ..
make -j 20


uname -a
Linux mytest 4.18.0-521.el8.x86_64 #1 SMP Mon Oct 30 18:21:17 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux


 gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-21)
Copyright © 2018 Free Software Foundation, Inc.

python --version
Python 3.8.17

To Reproduce

[100%] Built target unittest
[100%] Built target test_sqlite3_api_wrapper
Processing /home/longda/work/repo/db/duckdb/duckdb/tools/pythonpkg
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpp8fzqetz
       cwd: /tmp/pip-req-build-kdhbhq59
  Complete output (15 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-tb1lj80m/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-tb1lj80m/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-tb1lj80m/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 301, in <module>
  ModuleNotFoundError: No module named 'package_build'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpp8fzqetz Check the logs for full command output.
make[2]: *** [CMakeFiles/duckdb_python.dir/build.make:72：CMakeFiles/duckdb_python] 错误 1
make[1]: *** [CMakeFiles/Makefile2:4005：CMakeFiles/duckdb_python.dir/all] 错误 2
make: *** [Makefile:136：all] 错误 2

OS:

centos stream 8

DuckDB Version:

22

DuckDB Client:

python

Full Name:

happyhust

Affiliation:

build

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

Did you include all code required to reproduce the issue?

Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Yes, I have

The text was updated successfully, but these errors were encountered:

Tishj · 2024-05-13T08:02:43Z

package_build is refering to this module, located here relative to the root of the repo: scripts/package_build.py

Tishj · 2024-05-13T08:46:01Z

I might have a hunch as to why this happens, I believe setuptools can copy the sources to a different path, which respects MANIFEST.in but that isn't enough to ensure this invariant holds true:

sys.path.append(os.path.join(script_path, '..', '..', 'scripts'))
This is where it expects to find package_build.py

happyhust · 2024-05-13T13:49:16Z

In fact, I have do the same operation, it have the same error

cd duckdb
BUILD_BENCHMARK=1 BUILD_TPCH=1 BUILD_PYTHON=1  make 
cd build/release
make -j 20

[100%] Building CXX object tools/sqlite3_api_wrapper/CMakeFiles/test_sqlite3_api_wrapper.dir/test/test_sqlite3_udf_api_wrapper.cpp.o
done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpjsdpezqp
       cwd: /tmp/pip-req-build-1blze5ky
  Complete output (15 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-n_8b0tk5/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-n_8b0tk5/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-n_8b0tk5/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 301, in <module>
  ModuleNotFoundError: No module named 'package_build'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpjsdpezqp Check the logs for full command output.
make[2]: *** [CMakeFiles/duckdb_python.dir/build.make:72：CMakeFiles/duckdb_python] 错误 1
make[1]: *** [CMakeFiles/Makefile2:3916：CMakeFiles/duckdb_python.dir/all] 错误 2
make[1]: *** 正在等待未完成的任务....

happyhust · 2024-05-13T14:23:20Z

I might have a hunch as to why this happens, I believe setuptools can copy the sources to a different path, which respects MANIFEST.in but that isn't enough to ensure this invariant holds true:

sys.path.append(os.path.join(script_path, '..', '..', 'scripts')) This is where it expects to find package_build.py

I have a try, but it failed

virtualenv .venv --python=python3.8
source .venv/bin/activate
BUILD_PYTHON=1 make
cd build/release
make -j 20

The error like the following:

[ 99%] Built target benchmark_runner
[100%] Built target test_sqlite3_api_wrapper
Processing /home/longda/work/repo/db/duckdb/duckdb/tools/pythonpkg
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      ERROR: Could not install packages due to an OSError: Missing dependencies for SOCKS support.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make[2]: *** [CMakeFiles/duckdb_python.dir/build.make:72：CMakeFiles/duckdb_python] 错误 1
make[1]: *** [CMakeFiles/Makefile2:3916：CMakeFiles/duckdb_python.dir/all] 错误 2
make: *** [Makefile:136：all] 错误 2

happyhust · 2024-05-13T14:39:24Z

I might have a hunch as to why this happens, I believe setuptools can copy the sources to a different path, which respects MANIFEST.in but that isn't enough to ensure this invariant holds true:
sys.path.append(os.path.join(script_path, '..', '..', 'scripts')) This is where it expects to find package_build.py

I have a try, but it failed

virtualenv .venv --python=python3.8
source .venv/bin/activate
BUILD_PYTHON=1 make
cd build/release
make -j 20

The error like the following:

[ 99%] Built target benchmark_runner
[100%] Built target test_sqlite3_api_wrapper
Processing /home/longda/work/repo/db/duckdb/duckdb/tools/pythonpkg
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      ERROR: Could not install packages due to an OSError: Missing dependencies for SOCKS support.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make[2]: *** [CMakeFiles/duckdb_python.dir/build.make:72：CMakeFiles/duckdb_python] 错误 1
make[1]: *** [CMakeFiles/Makefile2:3916：CMakeFiles/duckdb_python.dir/all] 错误 2
make: *** [Makefile:136：all] 错误 2

this can be resolve by

pip3 install requests[socks] --user

but at last it still occur

Processing /home/longda/work/repo/db/duckdb/duckdb/tools/pythonpkg
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpt1p8jt38
       cwd: /tmp/pip-req-build-ixnxgo6q
  Complete output (15 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-l2oz08fd/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-l2oz08fd/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-l2oz08fd/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 301, in <module>
  ModuleNotFoundError: No module named 'package_build'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpt1p8jt38 Check the logs for full command output.
make[2]: *** [CMakeFiles/duckdb_python.dir/build.make:72：CMakeFiles/duckdb_python] 错误 1
make[1]: *** [CMakeFiles/Makefile2:3916：CMakeFiles/duckdb_python.dir/all] 错误 2
make: *** [Makefile:136：all] 错误 2

happyhust · 2024-05-14T02:30:17Z

I might have a hunch as to why this happens, I believe setuptools can copy the sources to a different path, which respects MANIFEST.in but that isn't enough to ensure this invariant holds true:

sys.path.append(os.path.join(script_path, '..', '..', 'scripts')) This is where it expects to find package_build.py

@Tishj How to resolve this problem?

Tishj · 2024-05-17T07:34:36Z

BUILD_PYTHON=1 make

Cmake should not be called directly

happyhust · 2024-05-19T03:24:05Z

close this issue, this problem occur in an old version, it works well on the latest code path.

happyhust added the needs triage label May 13, 2024

szarnyasg added the under review label May 13, 2024

duckdblabs-bot removed the needs triage label May 13, 2024

szarnyasg added the needs more info label May 13, 2024

happyhust closed this as completed May 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to build benchmark "ModuleNotFoundError: No module named 'package_build'" #12023

Failed to build benchmark "ModuleNotFoundError: No module named 'package_build'" #12023

happyhust commented May 13, 2024

Tishj commented May 13, 2024

Tishj commented May 13, 2024

happyhust commented May 13, 2024

happyhust commented May 13, 2024

happyhust commented May 13, 2024

happyhust commented May 14, 2024

Tishj commented May 17, 2024

happyhust commented May 19, 2024

Failed to build benchmark "ModuleNotFoundError: No module named 'package_build'" #12023

Failed to build benchmark "ModuleNotFoundError: No module named 'package_build'" #12023

Comments

happyhust commented May 13, 2024

What happens?

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Full Name:

Affiliation:

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

Did you include all relevant data sets for reproducing the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Tishj commented May 13, 2024

Tishj commented May 13, 2024

happyhust commented May 13, 2024

happyhust commented May 13, 2024

happyhust commented May 13, 2024

happyhust commented May 14, 2024

Tishj commented May 17, 2024

happyhust commented May 19, 2024