Fix Builds (python_gnupg-0.5.2-py2.py3-none-any.whl.asc 404 not found) #78
Comments
maltfield
changed the title
|
In the past, we've had build issues due to the verification of this same In the above issues, the issue was a tooling issue with the availability of curl/wget on MacOS. But in this case, it appears that the file actually has disappeared from the server. I cannot fetch it, for example, on my local machine |
|
I checked the package's page on the PyPI project's website. If there's a newer version, then it might make sense that the archived version somehow got orphaned from its signature? If so, it's a bug that we should report Anyway, it appears that the version hasn't changed since the one we pegged as a dependency. The latest version is still v0.5.2 |
|
I was digging through my notes and found that my old link to hash checking in PyPI is stale The above link points to a new location which, after some digging, brings you here Woah, that's dangerous. The above link suggests that hash checking in pip somehow adds security to verify packages are not malicious when turned-on. This is dangerous misinformation. pip doesn't have any built-in mechanism to cryptographically verify the authenticity of packages (although they've been working on this since 2019). The whole reason we have this awkward manual gpg check is because pip doesn't have a secure way to download software. Fortunately, Anyway, I opened a ticket with PyPI to fix their documentation to move the hashing section outside of the |
|
I opened an upstream bug about this here: |
|
I switched to fetching the files from GitHub, but now I have an issue with verifying the signature of the libusb1 dependency |
|
I opened an upstream bug about this here: |
|
Per the above ticket, the |
maltfield
added a commit
that referenced
this issue
Mar 6, 2024
it looks like the upstream PyPI signature bugs are more widespread than just python_gnupg I've worked with the devs to ensure releases are getting uploaded to GitHub so our builds aren't blocked by PyPI bugs breaking PGP signatures * #78 * vsajip/python-gnupg#234 * vpelletier/python-libusb1#96 Now we download from GitHub's website instead of PyPI's website, so 404 errors from PyPI won't break the builds. TODO: update this to query the GitHub API and grab the latest release (currently they're just hardcoded & pinned as a quick fix)
maltfield
added a commit
that referenced
this issue
Mar 6, 2024
maltfield
added a commit
that referenced
this issue
Mar 7, 2024
|
I've replaced the download URL for the signed python deps from PyPI to GitHub in all three platform's build scripts, but it looks like MacOS is still having some distinct issues finding the path to |
|
I'm also struggling with the PowerShell on Windows. For some reason I can't get the subcommand to work on this curl call. But if I take the same output from the subcommand and paste it into the curl command manually, it works fine I've asked about this on SE here: |
|
Update: issue with the windows builds in previous comment was a trailing newline. I fixed it by saving the output of |
|
Windows builds are fixed! I fixed the issue with the MacOS builds not being able to find PIP_PATH by adding a wildcard glob at the end of /usr/local/Cellar/python* But, unfortunately, lots of other pip depends are now failing to install It even fails at the end (in the "print debugging info" section) when trying to get the pip version |
|
Looks like I had this issue before, and it was fixed by making sure we used the right python/pip version |
|
Google seems to suggest that these Currently we install Looks like the latest version is |
|
TOFU 1/3 (VPN, exit in Hong Kong) gives sha256sum = |
|
TOFU 2/3 (Tor, exit in Poland) gives sha256sum = |
|
TOFU 3/3 (ISP, exit in Ecuador) gives sha256sum = |
|
Ok, 3/3 TOFU on 3 distinct days matches; adding to the repo |
maltfield
added a commit
that referenced
this issue
Mar 10, 2024
Upgrading pip in attempt to fix the SCHEME_KEYS errors on macos builds. For more info, see: * #78 (comment)
|
After upgrading, the linux & windows builds are still working but the MacOS builds are still failing We fixed the issue with SCHEME_KEYS, however, this is something new. For some reason when it installs |
|
The first noticeable error appears to be when it tries to install PyInstaller |
|
There's also issues installing Kivy. I think I should just go-through and update all the python packages to their latest versions. |
|
While I'm at it, I'll update python to Python v3.12. Python is already securely installed in our linux CI runner, so no manual updates needed there. Python comes installed on our MacOS CI runner. Possibly not secure, but we're relying on shared GitHub runners for MacOS builds. Anyway, no manual updates needed. We do have to install Python on our Windows CI runners, and manually update our installer in this repo. Fortunately, it looks like Python does sign their windows releases with PGP, but I don't have the key in my keyring. It looks like Python has a designated person as a Release Manager for each version of Python. Additionally, they have a designated person just for signing their Windows releases:
I'm going to have to 3TOFU those fingerprints. Here's the commands for this |
|
The documentation on how to verify the signature on python releases wasn't linked-to from the actual download page (where the link to download the signature is available), so I opened this ticket with the Python project to address this: |
|
I switched the macOS build script to use the python.org version of python 3.12 (instead of the brew version). Now the output of There is no |
|
Unfortunately, it looks like the Looks like the |
|
there must be some sort of race condition in the macOS buids, because I made some trivial change and the next build created the
Also, this was a triumph! I'm making a note here: the latest build from a macOS 13 system ran fine on a macOS 10.x system. So, indeed, it appears that switching from the brew-python to the python.org-python fixed this issue where builds don't work on older systems. Hooray! |
maltfield
added a commit
that referenced
this issue
Apr 9, 2024
this commit changes the path of the root_child from SRC_DIR to EXE_DIR. This is necessary since PyInstaller no longer puts symlinks from the .app/Contents/MacOS dir to .app/Contents/Resources dir, which required us to change the SRC_DIR from the 'MacOS' dir to the 'Resources' dir, but that's not where the root_child and buskill binaries live. * #78 (comment) * #78 (comment)
|
Alright, I stepped away for a couple months, partly because I wanted to properly document how to download brew dependencies from the GitHub Packages Container Registry (as described above) for 3TOFU -- because this is definitely going to be an issue again, in the future. I just finished writing the article (titled |
|
As of now, GitHub Actions is green for Linux and macOS builds. It's red for Windows builds. The latest build (from a trivial change to the docs) fails with an error when trying to verify the cryptographic authenticity of our deps Looks like I started looking into this before, so I have some good debug output |
|
I can't understand why the sig check is BAD, so I checked the file (and detached sig file) contents For comparison, here's the file on my local sandbox The contents of the files appears identical, but the hashes of the files differ |
|
I added a hexdump to the output of the most-recent run Here's the hexdump of the sig file on my linux dev machine And here's the hexdump from the GitHub runner during the build (that fails due to BAD signature) The first discrepancy I find is that there's these ASCII table says the |
|
I don't know what's causing the insertion of the windows carriage returns into our files (perhaps it's git), but that explains why the files look the same, but their hashes differ. On my local Windows 10 VM, I do not have the I'm not sure if this could be the cause of the gpg verification failure, however. It appears that gpg has a The If the output of If the output of Here's our packets of our sig Therefore, it appears that we're not using |
|
I created a new signature file of the And I pushed it to the |
maltfield
added a commit
to BusKill/buskill-app-deps
that referenced
this issue
Jun 6, 2024
…t it will work on Windows or other platforms that insert carriage returns before new lines * BusKill/buskill-app#78 (comment)
|
I also updated the build script to output the version of My local Windows 10 VM (which verifies the GOOD signature correctly) has gpg 2.4.5 (same as above), but with libgcrypt 1.10.3 (slightly newer than above) |
|
I manually re-ran the build, and it's working now. So, yep, adding Curiously, though, it's still detecting it as a failure and exiting 1 |
maltfield
added a commit
that referenced
this issue
Jun 6, 2024
this commit updates our documentation such that, when signing text files, we use the '--textmode' argument. This ensures that the signatures will still be valid on dumb systems (cough Windows cough) that insert additional carriage return characters before our newline characters. * #78 (comment) * BusKill/buskill-app-deps@63ccc12
|
ok, I fixed the error by making sure to execute |
|
cool, looks like our windows builds deps are now being installed OK. But we're getting an error on PyInstaller |
|
Well, I'm not familiar with the Curiously, it looks like PyInstaller is bitching about a module that it itself uses. So my best-guess here is that I need to update PyInstaller. It appears that we're currently installing PyInstaller-3.6. Oh wow, that's crazy oudated; it was release Jan 2020 The latest version is PyInstaller-6.7.0 it looks like this PyInstalller-5.12.0 was released on 2023-06-08, but the next release (5.13.0) explicilty states "Add support for Python 3.12" Looks like I'm missing a section to download windows-specific wheels for pyinstaller from the Here's the 3TOFU script |
|
Here's TOFU 1/3 (Tor, exit in Germany) |
|
Here's TOFU 2/3 (VPN, exit in France) |
|
Ah, fuck. I just realized that I've been downloading pyinstaller-6.7.0 (the latest version) though I said I'd use pyinstaller-6.5.0 to be consistent with others. Probably I should have done both |
|
Here's TOFU 1/3 (Tor, exit in the Netherlands) |
|
Here's TOFU 2/3 (VPN, exit in India) |
|
Here's TOFU 3/3 (ISP, exit in Ecuador) |
|
Ok, the hashes match on all 3TOFUs. I'll try upgrading pyinstaller in Windows now. |
maltfield
added a commit
to BusKill/buskill-app-deps
that referenced
this issue
Jun 10, 2024
see reasoning and 3TOFU here * BusKill/buskill-app#78 (comment)
maltfield
added a commit
that referenced
this issue
Jun 10, 2024
for more info, see: * #78 (comment)
|
w00t! The latest windows build completed successfully |
|
Well, I tried to download and run the latest build of BusKill for windows on my Windows 10 VM, but I got an error
Here's the relevant code that it's complaining about
Curiously, it appears this block was only added for running the app in a Windows 10 VM :/ |
|
Apparently the Update: Oh, right, it's because this is a new addition that I made in the |
maltfield
added a commit
that referenced
this issue
Jun 10, 2024
This commit only imports grp on *nix systems, since it isn't available on windows systems, and it will throw an error * #78 (comment) Traceback (most recent call last:): File "main .py", line 22 in <module> import platform, os, grp ModuleNotFoundError: No module named 'grp'
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Currently it's not possible to build the BusKill app due to gpg signing issues with the python-gnupg dependency
To Reproduce
I've tested this on Linux, but it should be an issue on all platforms
Steps to reproduce the behavior:
Execute the following on Debian 11
You'll get the following error
Expected behavior
The build should be able to verify the authenticity of the
python_gnupgpackage and proceed with building the BusKill appThe text was updated successfully, but these errors were encountered: