generic linux package #9351
generic linux package #9351
Conversation
|
After this PR lintian reports the bare minimum of non-fixable "issues": Lintianlintian koreader-2021.10.1-511-amd64.deb --no-tag-display-limit
E: koreader: embedded-library freetype usr/lib/koreader/libs/libfreetype.so.6
E: koreader: embedded-library libjpeg usr/lib/koreader/libs/libjpeg.so.8
E: koreader: embedded-library libjpeg usr/lib/koreader/libs/libturbojpeg.so
E: koreader: embedded-library libpng usr/lib/koreader/libs/libpng16.so.16
E: koreader: embedded-library openjpeg usr/lib/koreader/libs/libmupdf.so
E: koreader: embedded-library zlib usr/lib/koreader/libs/libz.so.1
E: koreader: embedded-library zlib usr/lib/koreader/sdcv
W: koreader: duplicate-font-file usr/lib/koreader/fonts/freefont/FreeSans.ttf also in (fonts-freefont-ttf)
W: koreader: duplicate-font-file usr/lib/koreader/fonts/freefont/FreeSerif.ttf also in (fonts-freefont-ttf)
W: koreader: national-encoding usr/lib/koreader/frontend/ui/elements/font_ui_fallbacks.lua
W: koreader: national-encoding usr/lib/koreader/l10n/af_ZA/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ar/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/be/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/bg_BG/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/bn/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ca/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/cs/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/de/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/el/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/eo/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/es/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/eu/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/fa/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/fi/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/fr/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/gl/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/he/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/hi/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/hr/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/hu/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/id/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ie/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/it_IT/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ja/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ka/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ko_KR/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/lt_LT/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/lv/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ms/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/nb_NO/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/nl_NL/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/pl/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/pt_BR/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/pt_PT/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ro/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ro_MD/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/ru/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/si/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/sk/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/sr/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/sv/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/tr/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/uk/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/vi/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/zh_CN/koreader.po
W: koreader: national-encoding usr/lib/koreader/l10n/zh_TW/koreader.po
W: koreader: opentype-font-wrong-filename [usr/lib/koreader/fonts/freefont/FreeSans.ttf]
W: koreader: opentype-font-wrong-filename [usr/lib/koreader/fonts/freefont/FreeSerif.ttf] |
|
I think most of permission fixes (previously on |
Well, I doubt if they are all really non-fixable. If I understand things correctly, KOReader was born in a hostile environment (like the kindle system) and therefore could not rely on anything there, so it had to include libraries and some hacks to work with hacky kernel-side implementation of various things. And the deb packages as they are now try to be generic so there is a one package works everywhere. That all is understandable and needed. On the other hand distributions e.g. want to ensure that if they fix a security issue in openssl, every package using openssl will use the fixed version, so they tend to avoid having required libraries in every package. So if they package it every distribution will carry on their own patch to untangle things. So that would be duplicate work. |
|
@akemnade: it started in #3108 (comment) with a very specific scope: #3108 (comment) We still keep the same spirit, AFAICT. |
|
Yeah, unvendoring would be a maintenance nightmare for something none of us would actually actively test ;). So, the burden is definitely on downstream for those sorts of endeavors. |
As an aside, sometimes I'll just use my Debian system libraries to write the Lua stuff, keeping building the library as an exercise for later. It's certainly convenient. But yeah, not actively tested for sure. ^_^ |
|
Well, for newcomers like me, you get baffled by the build system, could you dump and share your knowledgte about running KOReader that way in a human + possibly machine readable form ;-) These libkoreader-* stuff are not system libraries, so it involves something more than the lua files. To avoid misunderstandings I am not talking about putting such builds as downloadable release packages. A tarball bringing its own dependencies (also as a step towards building a deb package) is IMHO ok. |
Nope. Some stuff is coded in c/c++ using the lua C API. You can get a nice picture of all of them in https://github.com/koreader/koreader-base Look at
Lol. Nope, No, Nein, Non.
As far as I can tell nobody built KOReader (as you can grasp now it is a bunch of lua code that scripts a lot of native deps) for nothing other than the GNU C library. |
|
Also, in part because of FFI, the pinned versions of our vendored libs matter, which is another strike against devendoring ;). I wouldn't necessarily expect musl to be a huge problem (after all, we manage to deal with bionic), but, yeah, stuff will probably break nonetheless ;). |
|
And some of these thirparty libraries are patched by us. Some for specific platforms stuff (ie Android), some to disable some dependancies, tweaks or remove some behaviour, or to fix or extend them (ie. lunasvg recently). |
And a large part of our patches are not suitable for upstream anyway (because hacks or platform-specific or whatever ;)). |
So you are objecting against your own patches?! That comment was meant as a +1 for this PR as it is now.
Well, it is a bit bumpy, yes. That involves fixing building of libraries, that I can easily get via system package management. |
Makefile
Outdated
|
|
||
| # fix permissions | ||
| find $(INSTALL_DIR)/linux -type d -print0 | xargs -0 chmod 755 | ||
| find $(INSTALL_DIR)/linux -executable -type f -print0 | xargs -0 chmod 755 |
There was a problem hiding this comment.
find … -executable will already include directories:
-executable
Matches files which are executable and directories which are searchable (in a file name resolution sense) by the current user.
But I would just use chmod -R u=rwX,og=r ….
There was a problem hiding this comment.
Could you make a nice suggestion with the recursive chmod instead of find?
There was a problem hiding this comment.
I figure that's simply supposed to read like this? But let's await confirmation.
| find $(INSTALL_DIR)/linux -executable -type f -print0 | xargs -0 chmod 755 | |
| chmod -R u=rwX,og=r $(INSTALL_DIR)/linux |
|
Ok, applied, some issues related to permissions: E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libcrypto.so.1.1]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libczmq.so.1]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libdjvulibre.so.21]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libfreetype.so.6]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libfribidi.so.0]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libgif.so.7]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libharfbuzz.so.0]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libk2pdfopt.so.2]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/liblept.so.5]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/liblodepng.so]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libmupdf.so]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libpng16.so.16]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libsharpyuv.so.0]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libsqlite3.so]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libssl.so.1.1]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libtesseract.so.3]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libunibreak.so.6]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libutf8proc.so.3]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libwebp.so.7]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libwebpdemux.so.2]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libz.so.1]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libzmq.so.4]
E: koreader: shared-library-is-executable 0755 [usr/lib/koreader/libs/libzstd.so.1]and this one I'm not sure about:
maybe I copied something wrong on your link_fonts suggestion change? |
On my Arch Linux system, all the libraries in
Looks like those file have the wrong permissions in the original checkout.
Sorry, my bad, try this: link_fonts() {
syspath="../../../../share/fonts/truetype/$(basename "$1")"
for FILE in "$1"/*.ttf; do
ln -snf "${syspath}/${FILE##*/}" "${FILE}"
done
} |
|
I believe that on *BSD they're not executable either. |
|
I would check for the required tools before doing anything in Diff: Makefile | 5 +--
platform/linux/do_debian_package.sh | 72 ++++++++++++++++++++-----------------
2 files changed, 41 insertions(+), 36 deletions(-)
diff --git i/Makefile w/Makefile
index 4d094c33b..bda882029 100644
--- i/Makefile
+++ w/Makefile
@@ -455,10 +455,7 @@ linuxupdate: all
find $(INSTALL_DIR)/linux -type d \( -name "test" -o -name ".github" \) -print0 | xargs -0 rm -rf
# fix permissions
- find $(INSTALL_DIR)/linux -type d -print0 | xargs -0 chmod 755
- find $(INSTALL_DIR)/linux -executable -type f -print0 | xargs -0 chmod 755
- find $(INSTALL_DIR)/linux -type f \( -name "COPYING" -o -name "git-rev" -o -name "*manifest" -o -name "*.cff" -o -name "*.css" -o -name "*.desktop" -o -name "*.json" -o -name "*.html" -o -name "*.lua" -o -name "*.pattern" -o -name "*.png" -o -name "*.otf" -o -name "*.po*" -o -name "*.so*" -o -name "*.svg" -o -name "*.template" -o -name "*.tpl" -o -name "*.ttf" \) -print0 | xargs -0 chmod 644
- find $(INSTALL_DIR)/linux -type f -name "reader.lua" -print0 | xargs -0 chmod 755
+ chmod -R u=rwX,og=rX $(INSTALL_DIR)/linux
XZ_OPT=9 tar -C $(INSTALL_DIR)/linux/usr -cvJf $(INSTALL_DIR)/$(LINUX_PACKAGE) .
rm -rf $(INSTALL_DIR)/linux
diff --git i/platform/linux/do_debian_package.sh w/platform/linux/do_debian_package.sh
index 8139743e5..4e6c2df2d 100755
--- i/platform/linux/do_debian_package.sh
+++ w/platform/linux/do_debian_package.sh
@@ -9,8 +9,8 @@ command_exists() {
link_fonts() {
syspath="../../../../share/fonts/truetype/$(basename "$1")"
- for FILE in *.ttf; do
- ln -snf "${syspath}/${FILE}" "$1/${FILE}"
+ for FILE in "$1"/*.ttf; do
+ ln -snf "${syspath}/${FILE##*/}" "${FILE}"
done
}
@@ -25,7 +25,8 @@ uname_to_debian() {
write_changelog() {
CHANGELOG_PATH="${1}/share/doc/koreader/changelog.Debian.gz"
- CHANGELOG=$(cat << 'END_HEREDOC'
+ CHANGELOG=$(
+ cat <<'END_HEREDOC'
koreader (0.1) unstable; urgency=low
* Fixes most lintian errors and warnings
@@ -44,26 +45,34 @@ END_HEREDOC
chmod 644 "${CHANGELOG_PATH}"
}
-if [ -z "${1}" ]; then
+if ! [ -r "${1}" ]; then
echo "${0}: can't find KOReader archive, please specify a path to a KOReader tar.gz" 1>&2
exit 1
-else
- mkdir -p tmp-debian/usr
- chmod 0755 tmp-debian/usr
- tar -xf "${1}" -C tmp-debian/usr
- ARCH="$(echo "${1}" | cut -d '-' -f3)"
- VERSION="$(cut -f2 -dv "tmp-debian/usr/lib/koreader/git-rev" | cut -f1,2 -d-)"
- DEB_ARCH="$(uname_to_debian "${ARCH}")"
fi
-# Run only if dpkg-deb exists
-COMMAND="dpkg-deb"
-if command_exists "${COMMAND}"; then
- BASE_DIR="tmp-debian"
+# Check for required tools.
+missing_tools=()
+for tool in dpkg-deb fakeroot; do
+ if ! command_exists "${tool}"; then
+ missing_tools+=("${tool}")
+ fi
+done
+if [[ ${#missing_tools[@]} -ne 0 ]]; then
+ echo "${0}: unable to build Debian package, the following tools are missing: ${missing_tools[*]}" 1>&2
+ exit 1
+fi
- # populate debian control file
- mkdir -p "${BASE_DIR}/DEBIAN"
- cat >"${BASE_DIR}/DEBIAN/control" <<EOF
+mkdir -p tmp-debian/usr
+chmod 0755 tmp-debian/usr
+tar -xf "${1}" -C tmp-debian/usr
+ARCH="$(echo "${1}" | cut -d '-' -f3)"
+VERSION="$(cut -f2 -dv "tmp-debian/usr/lib/koreader/git-rev" | cut -f1,2 -d-)"
+DEB_ARCH="$(uname_to_debian "${ARCH}")"
+BASE_DIR="tmp-debian"
+
+# populate debian control file
+mkdir -p "${BASE_DIR}/DEBIAN"
+cat >"${BASE_DIR}/DEBIAN/control" <<EOF
Section: graphics
Priority: optional
Depends: libsdl2-2.0-0, fonts-noto-hinted, fonts-droid-fallback, libc6 (>= 2.2.3)
@@ -80,24 +89,23 @@ Description: Ebook reader application supporting PDF, DjVu, EPUB, FB2 and many m
It’s available for Kindle, Kobo, PocketBook, Android and desktop Linux.
EOF
- # use absolute path to luajit in reader.lua
- sed -i 's,./luajit,/usr/lib/koreader/luajit,' "${BASE_DIR}/usr/lib/koreader/reader.lua"
+# use absolute path to luajit in reader.lua
+sed -i 's,./luajit,/usr/lib/koreader/luajit,' "${BASE_DIR}/usr/lib/koreader/reader.lua"
- # use debian packaged fonts instead of our embedded ones to save a couple of MB.
- # Note: avoid linking against fonts-noto-cjk-extra, cause it weights ~200MB.
- link_fonts "${BASE_DIR}/usr/lib/koreader/fonts/noto"
+# use debian packaged fonts instead of our embedded ones to save a couple of MB.
+# Note: avoid linking against fonts-noto-cjk-extra, cause it weights ~200MB.
+link_fonts "${BASE_DIR}/usr/lib/koreader/fonts/noto"
- # DroidSansMono has a restrictive license. Replace it with DroidSansFallback
- ln -snf ../../../../share/fonts-droid-fallback/truetype/DroidSansFallback.ttf "${BASE_DIR}/usr/lib/koreader/fonts/droid/DroidSansMono.ttf"
+# DroidSansMono has a restrictive license. Replace it with DroidSansFallback
+ln -snf ../../../../share/fonts-droid-fallback/truetype/DroidSansFallback.ttf "${BASE_DIR}/usr/lib/koreader/fonts/droid/DroidSansMono.ttf"
- # add debian changelog
- write_changelog "${BASE_DIR}/usr"
+# lintian complains if shared libraries have execute rights.
+find "${BASE_DIR}" -type f -perm /+x -name '*.so*' -print0 | xargs -0 chmod a-x
- fakeroot dpkg-deb -b "${BASE_DIR}" "koreader-${VERSION}-${DEB_ARCH}.deb"
-else
- echo "${COMMAND} not found, unable to build Debian package" 1>&2
- exit 1
-fi
+# add debian changelog
+write_changelog "${BASE_DIR}/usr"
+
+fakeroot dpkg-deb -b "${BASE_DIR}" "koreader-${VERSION}-${DEB_ARCH}.deb"
rm -rf tmp-debian
Building the deb on Arch Linux, and running lintian in docker: How come our freefont files use the wrong extension? |
|
thanks for the diff, @benoit-pierre
No idea. Also no idea why I didn't unvendor them. I can't remember if our freefont is special :p, pinging @NiLuJe |
platform/linux/do_debian_package.sh
Outdated
| @@ -75,7 +75,7 @@ mkdir -p "${BASE_DIR}/DEBIAN" | |||
| cat >"${BASE_DIR}/DEBIAN/control" <<EOF | |||
| Section: graphics | |||
| Priority: optional | |||
| Depends: libsdl2-2.0-0, fonts-noto-hinted, fonts-droid-fallback, libc6 (>= 2.2.3) | |||
| Depends: libsdl2-2.0-0, fonts-noto-hinted, fonts-droid-fallback, fonts-freefont-ttf, libc6 (>= 2.2.3) | |||
There was a problem hiding this comment.
libc6 (>= 2.2.3)
This was true at some point. Not sure what's the glibc version of the current build machine image. Maybe it needs a bump?
There was a problem hiding this comment.
ii libc6:amd64 2.31-0ubuntu9.15 amd64 GNU C Library: Shared libraries
Yes, IIRC, I built then from SVN manually. (Details should be in the commit message for each font build). And I had to mangle the file extension to avoid migration annoyances (but they are built as CFF), as the original upstream build we used was a TrueType one. |
Thanks! I'm assuming all the symbols used across the UI come from nerdfonts. If so and our freefont has the same glyphs as the debian package I'm mostly in for unvendor it. |
|
I don't know about the Free* fonts - and if it is still active upstream. I think we would avoid some bug reports, non-reproducibility, and tough investigations if we were to keep using our shipped fonts. But up to you, this must be a very small user base. |
|
We may very well rely on some obscure SVN-only features of FreeFont somewhere, because the last release is from 2012 (!). (The last SVN commit is from ca. 2 years ago, and is what we're currently using. Obvious typo in commit message: FontForge UI, not FreeFont UI ;)). Soo, yeah, I wouldn't unvendor it either. |
Co-authored-by: Benoit Pierre <benoit.pierre@gmail.com>
|
Rebased on top of latest Makefile changes. Seem to build fine, tested both the x86_64 and the arm64 builds. Cannot test armhf ATM since it isn't available on default arch repos. |
Relates to #9268
Requires koreader/koreader-base#1504
Usage:
kodev release linuxfor native package (same arch as host)LINUX_ARCH=arm release linuxfor armhf on a x86_64 host.LINUX_ARCH=arm64 release linuxfor arm64 on a x86_64 host.It produces a
koreader-linux-$ARCH-$VERSION.tar.xzarchive, where $ARCH followsuname -mconvention {x86_64, armv7l, aarch64}To generate the debian package from the generic archive the following command is required
./platform/linux/do_debian_package.sh path_to.tar.xzThis change is