BUG: docker-compose build --no-cache Command Fails on RUN python3 -m ensurepip Step

[jspeed-meyers]

First, let me admit: it is entirely possible that I am the problem here and this is all user error. I am new to Weaviate.

I am trying to use this benchmark to compare the performance of the standard Weaviate Docker Hub container to the Chainguard version of that container. Not that it's germane to this potential bug below, but I expect no difference in performance.

The problem I encountered is that the docker-compose build --no-cache command fails on the RUN python3 -m ensurepip step.

Here are the steps I followed:

I cloned into the repo:

git clone https://github.com/weaviate/weaviate-benchmarking.git

I changed the working directory:

cd weaviate-benchmarking

I added the benchmark-data folder:

mkdir benchmark-data && \
    curl -o ./benchmark-data/deep-image-96-angular.hdf5 http://ann-benchmarks.com/deep-image-96-angular.hdf5 && \
    curl -o ./benchmark-data/mnist-784-euclidean.hdf5 http://ann-benchmarks.com/mnist-784-euclidean.hdf5 && \
    curl -o ./benchmark-data/gist-960-euclidean.hdf5 http://ann-benchmarks.com/gist-960-euclidean.hdf5 && \
    curl -o ./benchmark-data/glove-25-angular.hdf5 http://ann-benchmarks.com/glove-25-angular.hdf5

I ran the docker-compose command:

docker-compose build --no-cache

I then had this output and error:

docker-compose build --no-cache
[+] Building 19.9s (12/18)                                                      
 => [internal] load build definition from Dockerfile-ann                   0.0s
 => => transferring dockerfile: 832B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/golang:alpine           0.7s
 => [auth] library/golang:pull token for registry-1.docker.io              0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 64.05kB                                       0.0s
 => [ 1/13] FROM docker.io/library/golang:alpine@sha256:2a882244fb51835eb  0.0s
 => CACHED [ 2/13] WORKDIR /var/lib/benchmark                              0.0s
 => [ 3/13] COPY benchmark-scripts/ann/requirements.txt /var/lib/benchmar  0.0s
 => [ 4/13] RUN apk update                                                 0.7s
 => [ 5/13] RUN apk add make automake gcc g++ curl python3 python3-dev p  16.3s 
 => [ 6/13] RUN ln -s /usr/include/locale.h /usr/include/xlocale.h         0.3s 
 => ERROR [ 7/13] RUN python3 -m ensurepip                                 1.7s 
------                                                                          
 > [ 7/13] RUN python3 -m ensurepip:                                            
#0 1.641 error: externally-managed-environment                                  
#0 1.641                                                                        
#0 1.641 × This environment is externally managed                               
#0 1.641 ╰─>                                                                    
#0 1.641     The system-wide python installation should be maintained using the system
#0 1.641     package manager (apk) only.
#0 1.641     
#0 1.641     If the package in question is not packaged already (and hence installable via
#0 1.641     "apk add py3-somepackage"), please consider installing it inside a virtual
#0 1.641     environment, e.g.:
#0 1.641     
#0 1.641     python3 -m venv /path/to/venv
#0 1.641     . /path/to/venv/bin/activate
#0 1.641     pip install mypackage
#0 1.641     
#0 1.641     To exit the virtual environment, run:
#0 1.641     
#0 1.641     deactivate
#0 1.641     
#0 1.641     The virtual environment is not deleted, and can be re-entered by re-sourcing
#0 1.641     the activate file.
#0 1.641     
#0 1.641     To automatically manage virtual environments, consider using pipx (from the
#0 1.641     pipx package).
#0 1.641 
#0 1.641 note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
#0 1.641 hint: See PEP 668 for the detailed specification.
#0 1.692 Traceback (most recent call last):
#0 1.692   File "<frozen runpy>", line 198, in _run_module_as_main
#0 1.692   File "<frozen runpy>", line 88, in _run_code
#0 1.692   File "/usr/lib/python3.11/ensurepip/__main__.py", line 5, in <module>
#0 1.692     sys.exit(ensurepip._main())
#0 1.692              ^^^^^^^^^^^^^^^^^
#0 1.692   File "/usr/lib/python3.11/ensurepip/__init__.py", line 286, in _main
#0 1.692     return _bootstrap(
#0 1.692            ^^^^^^^^^^^
#0 1.692   File "/usr/lib/python3.11/ensurepip/__init__.py", line 202, in _bootstrap
#0 1.692     return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
#0 1.692            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 1.692   File "/usr/lib/python3.11/ensurepip/__init__.py", line 103, in _run_pip
#0 1.692     return subprocess.run(cmd, check=True).returncode
#0 1.692            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 1.692   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 1.693     raise CalledProcessError(retcode, process.args,
#0 1.693 subprocess.CalledProcessError: Command '['/usr/bin/python3', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpr99t5w11/setuptools-65.5.0-py3-none-any.whl\', \'/tmp/tmpr99t5w11/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpr99t5w11\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
------
failed to solve: executor failed running [/bin/sh -c python3 -m ensurepip]: exit code: 1

Am I doing something wrong? I'll try to debug.

My system:

• Apple M1 Pro (Sonora: 14.4.1)

My docker details below:

docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.15.0 (93002)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:41 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[jspeed-meyers]

This bug seems related to this issue:

• Dockerfile-ann build fail  #8

And also to this PR:

• Fix Dockerfile for ann #7