problems with unit tests

[alexanderguzhva]

Summary

unexpected crashes while running python3 -m pytest in a docker container

Platform

ubuntu 22.04, docker

standard g++

Faiss version: master (4d06d70)

Installed from: compiled on my own

Faiss compilation options:

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

> docker run -it --rm ubuntu:22.04

> uname -a
Linux 1f75fc908e4e 6.5.0-1018-aws #18~22.04.1-Ubuntu SMP Fri Apr  5 17:44:33 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
> apt update && apt install python3 python3-pip gcc g++ mc git swig sudo libomp-dev libopenblas-dev
> pip3 install numpy scipy pytest
Collecting numpy
  Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 53.6 MB/s eta 0:00:00
Collecting scipy
  Downloading scipy-1.13.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.6/38.6 MB 59.1 MB/s eta 0:00:00
Installing collected packages: numpy, scipy
Successfully installed numpy-1.26.4 scipy-1.13.0
> git clone https://github.com/facebookresearch/faiss
> wget -qO- "https://cmake.org/files/v3.26/cmake-3.26.5-linux-x86_64.tar.gz" | sudo tar --strip-components=1 -xz -C /usr/local
> cd faiss

> /usr/local/bin/cmake -B build -DFAISS_ENABLE_GPU=OFF -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Release .
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Could NOT find MKL (missing: MKL_LIBRARIES) 
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so  
-- Looking for cheev_
-- Looking for cheev_ - found
-- Found LAPACK: /usr/lib/x86_64-linux-gnu/libopenblas.so;-lm;-ldl  
-- Found SWIG: /usr/bin/swig4.0 (found version "4.0.2") found components: python 
-- Found Python: /usr/include/python3.10 (found version "3.10.12") found components: Development NumPy Interpreter Development.Module Development.Embed 
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter 
-- Configuring done (2.4s)
-- Generating done (0.0s)
-- Build files have been written to: /root/faiss/faiss/build

> make -C build -j 8 faiss
> make -C build -j 8 swigfaiss
> cd build/faiss/python && python setup.py install

then, run python3 -m pytest from tests directory.

# python3 -m pytest
======================================================= test session starts ========================================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0
rootdir: /root/faiss/faiss/tests
collected 845 items                                                                                                                

test_autotune.py .....                                                                                                       [  0%]
test_binary_factory.py ......                                                                                                [  1%]
test_binary_hashindex.py ......                                                                                              [  2%]
test_binary_io.py ......                                                                                                     [  2%]
test_build_blocks.py ..................................................                                                      [  8%]
test_callback_py.py .                                                                                                        [  8%]
test_clone.py Fatal Python error: Aborted

Current thread 0x0000703955c5b1c0 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/swigfaiss.py", line 4798 in train
  File "/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/class_wrappers.py", line 298 in replacement_train
  File "/root/faiss/faiss/tests/test_clone.py", line 28 in do_test_clone
  File "/root/faiss/faiss/tests/test_clone.py", line 79 in test_AdditiveQuantizer
  File "/usr/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
  File "/usr/lib/python3.10/unittest/case.py", line 591 in run
  File "/usr/lib/python3.10/unittest/case.py", line 650 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/unittest.py", line 343 in runtest
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 241 in <lambda>
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 341 in from_call
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 240 in call_and_report
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 339 in _main
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 285 in wrap_session
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 178 in main
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/usr/local/lib/python3.10/dist-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib/python3.10/runpy.py", line 86 in _run_code
  File "/usr/lib/python3.10/runpy.py", line 196 in _run_module_as_main

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, faiss._swigfaiss, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation (total: 54)
Aborted (core dumped)

removing test_clone.py won't help, because there will be moooooore problems

compiling with avx2 won't help

[alexanderguzhva]

Depending on the OS version in a docker contailer, compiler version, the results of the unit tests (what unit tests exactly fail) are different. I've tried multiple variants.

[mdouze]

Hmmm that's weird, we don't observe that.