[vcpkg-get-python-packages] Cache packages via pip precandidate

[WangWeiLin-MV]

WIP

Abstract

Download pypi packages by vcpkg_download_distfile, then install packages offline.

Fix #37473 (review). And any other port that depends on pip install.

Changes

• Bootstrap virtual environment with runtime which has neither standard library venv nor pip (i.e. Embedded Python)
• Add pip command precandidate to obtain the candidate package filename during local installation
• Download packages by vcpkg_download_distfile

WIP

[Neumann-A]

a) this depends on the tool RFC
b) simply no. Doesn't solve the real problem. Simply change vcpkg_get_python_packages so that it is asset cacheable without changing its interface?

[Hoikas]

Python tools need to be the exact same version as the python3 port, or you introduce potential interesting compile failures. AFAIK, the Python 3 port is still on 3.11, but this is using 3.12.

This also seems to be completely unnecessary - we already have the tool package and vcpkg_get_python_packages().

[WangWeiLin-MV]

@Neumann-A @Hoikas

The purpose of this port is as a module for finding Python interpreter and packages, and implementing #31748 vcpkg-tool-python3-interpreter, and combine and make vcpkg-get-python-packages cacheable.

• Explicit use external runtime, with download instructions as possible alternatives
• The possible means since a stable cross-platform runtime is not available (actually download python only work for Windows), downloading should not be the default option. Perhaps in the future, download option disabled and managed by vcpkg whether to download the required external tool, include pkgconf, swig.
• Unless ports/python3 is used as a dependency, but it is too expensive for end use. Of course, this is not prohibited
• Allow multi-version interpreters and multi-version packages to meet possible dependency specifications (pypa allows specifying version less than), also allow specify search path of both interpreter and packages
• Replaces the existing vcpkg_find_acquire_program_python
• Added option to create virtual environment
• Explicit to download packages and enabled virtual environment.
• Avoid installing from source dist as it introduces a lot of build dependencies
• Won't modify env:PATH
• Won't modify ${DOWNLOADS} manually
• Won't install packages in ${CURRENT_PACKAGES_DIR}, because pypa package depends on interpreter implementation (also version), even on the same platform

---

Python tools need to be the exact same version as the python3 port, or you introduce potential interesting compile failures. AFAIK, the Python 3 port is still on 3.11, but this is using 3.12.

The ports/python installs python libraries and binary tools, while current script only find or provide a runtime with packages and is not installed as artifacts. I think this will not affect each other.

[Neumann-A]

Just use the same approach as vcpkg_find_acquire_msys and put it into vcpkg-get-python-packages

[WangWeiLin-MV]

Just use the same approach as vcpkg_find_acquire_msys and put it into vcpkg-get-python-packages

vcpkg-get-python-packages does not provide search function. And no optional virtual environment, it makes difficult to integrate for current port purpose.

About same approach as vcpkg_find_acquire_msys, there are some problems to resolve:

For using pypi source dist requires additional build dependencies, I didn't choose this way.

And for pypi binary dist consists of complex dependencies such as https://pypi.org/pypi/pandas/json requires_dist and lots of wheel https://pypi.org/project/pandas/#files depends on platform. The detection function is used here to obtain dependencies.

[Neumann-A]

@WangWeiLin-MV: I am aware of all what your are telling me. I am still saying use the same approach as vcpkg_find_acquire_msys. Just make it download the wheel and declare the runtime dependencies which can be tested using an from package import * after installation.

YOu also want to automate the generation. So you might want to create a powershell/cmake scripts using
"Invoke-WebRequest -Uri \"https://pypi.python.org/pypi/${package_name}/json/\" -Method GET | ConvertFrom-Json | Select-Object -ExpandProperty info | Select-Object -ExpandProperty version"
you can modify the web request to get the url and filename of the wheel.

[WangWeiLin-MV]

Thanks to @Neumann-A and @Hoikas, I'll continue the discussion later that for the find_python function and whether it replaces vcpkg_find_acquire_program_python and vcpkg-get-python-packages.

Before this, I will first make vcpkg-get-python-packages cacheable.