fil-profile run failed with symbol lookup error: <lib root>/filprofiler/_filpreload.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyBytes_Size

[Stonre]

Hi,

I imported filprofiler into our gitfarm and built it (through setuptools setup.py). The built artifacts are the same as what I get from pip install filprofiler. However, when I run fil-profile run within our python env, it failed with symbol lookup error: <lib root>/filprofiler/_filpreload.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyBytes_Size. I have investigated quite a long time and below are my findings:

• I tried python3.8 (minor version is 3.8.11) and it failed with the undefined symbol error. But when I replaced the libpython3.8.so.1.0 with another libpython3.8.so.1.0 (from minor version 3.8.5), it worked fine.
• Same thing happened for python3.7. I tried python3.7 (minor version is 3.7.11) it failed with the undefined symbol error (undefined symbol: Py_IsInitialized). But when I replaced the libpython3.7m.so.1.0 with another libpython3.7m.so.1.0 (from minor version 3.7.10), it worked fine.
• python3.6.14 also not working and throws undefined symbol: PyBytes_Size.
• I did check the original libpython3.8.so.1.0 and it contains symbol PyBytes_Size (by nm -D libpython3.8.so.1.0)

I wonder if there is any incompatibility for filprofiler on higher minor version for python3.6(.14), 3.7(.11) and 3.8(.11)? And why filprofiler cannot find the symbol while the symbol exists in the libpython.so file?

[itamarst]

Some notes:

• PyBytes_Size has been part of the Python API since at least Python 3.3, maybe earlier, and still exists in Python 3.10. So it's not new, and it's not a removed.
• Python 3.8.12 can run Fil from PyPI just fine. So I don't think it's an issue with newer minor versions.

When you say "you tried Python 3.8", did you recompile with Python 3.8?

My first guess here is that you are compiling with a different version than the one you're importing with. You can explicitly set a path to a Python binary to build with by setting export PYO3_PYTHON=/path/to/bin/python3.7 in addition to activating a virtualenv or whatever.

(Unrelatedly, I suggest stopping using Python 3.6: https://pythonspeed.com/articles/stop-using-python-3.6/)

[Stonre]

Hi, @itamarst thanks a lot for your quick response.

• To answer your question "When you say "you tried Python 3.8", did you recompile with Python 3.8?", yes, I recompile filprofiler with Python3.8 under my python environment as my company only allows source import of 3rd party package.
• Regarding your guess "you are compiling with a different version than the one you're importing with", thanks for this suggestion and I will try with export PYO3_PYTHON=/path/to/bin/python3.7 and see whether it works.
• One follow up question, do you have any insights of why when I replace the libpython*.so* file with a different minor version, it can start working (even both the files have the symbol PyBytes_Size)? Is there any hard codes (maybe relates to some version number) in compiled _filpreload.cpython-36m-x86_64-linux-gnu.so that makes it work only with the particular libpython*.so* file?

[itamarst]

In theory when you compile Python extensions they shouldn't be sensitive to minor version changes like that.

[itamarst]

What Linux distribution are you using? Had someone report a similar issue with Nix.

[Stonre]

What Linux distribution are you using? Had someone report a similar issue with Nix.

Hi, @itamarst Thanks for your kind response with more findings. Here is what linux information I used:

I sshed into my AWS EC2 machine and ran cat /etc/os-release and got the following:

cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2:-:internal"
HOME_URL="https://amazonlinux.com/"
VARIANT="internal"

And my linux kernal version is: 5.4.181-109.354.amzn2int.x86_64

Let me know if you need any more info for you to identify the root cause and fix.

Thanks,
Lei

[Stonre]

Hi, @itamarst Just to follow up, do you have the plan to fix this issue (if this is actually an issue for some linux distribution)?

[itamarst]

I guess I should do some testing on Amazon Linux and see if I can reproduce with binaries.

[Stonre]

Hi, @itamarst Do you by any chance have update on this issue? I see you have put it to NEXT stage. Just to check whether pythonspeed has the plan to check and fix this issue.

[itamarst]

I have limited time for Fil (mostly spending time on my profiler for production use, https://pythonspeed.com/sciagraph/), and having others compiling from scratch is not a top priority for Fil either. There's people having issues on macOS that are higher priority. But I will get to it eventually hopefully.