Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mkosi: Sanitizers #32866

Merged
merged 13 commits into from
May 30, 2024
Merged

mkosi: Sanitizers #32866

merged 13 commits into from
May 30, 2024

Conversation

DaanDeMeyer
Copy link
Contributor

@DaanDeMeyer DaanDeMeyer commented May 16, 2024

No description provided.

@github-actions github-actions bot added tests please-review PR is ready for (re-)review by a maintainer labels May 16, 2024
Copy link

Important

An -rc1 tag has been created and a release is being prepared, so please note that PRs introducing new features and APIs will be held back until the new version has been released.

@bluca bluca added ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR and removed please-review PR is ready for (re-)review by a maintainer labels May 16, 2024
@github-actions github-actions bot added please-review PR is ready for (re-)review by a maintainer and removed ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR labels May 17, 2024
@DaanDeMeyer DaanDeMeyer force-pushed the sanitizers branch 2 times, most recently from f94fda7 to 914e92c Compare May 17, 2024 19:41
@bluca

This comment was marked as off-topic.

@bluca bluca added ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR please-review PR is ready for (re-)review by a maintainer and removed please-review PR is ready for (re-)review by a maintainer ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR labels May 20, 2024
@bluca
Copy link
Member

bluca commented May 21, 2024

Loaded 'libidn2.so.0' via dlopen()
Loaded 'libcryptsetup.so.12' via dlopen()
Loaded 'libpwquality.so.1' via dlopen()
Loaded 'libtss2-esys.so.0' via dlopen()
Loaded 'libtss2-rc.so.0' via dlopen()
Loaded 'libtss2-mu.so.0' via dlopen()
Loaded 'libfido2.so.1' via dlopen()
Loaded 'libbpf.so.1' via dlopen()
Loaded 'libdw.so.1' via dlopen()
Loaded 'libelf.so.1' via dlopen()
Loaded 'libpcre2-8.so.0' via dlopen()
Loaded 'libp11-kit.so.0' via dlopen()
Loaded 'libarchive.so.13' via dlopen()
Loaded 'liblz4.so.1' via dlopen()
Loaded 'libzstd.so.1' via dlopen()
Loaded 'liblzma.so.5' via dlopen()
Loaded 'libkmod.so.2' via dlopen()
Tracer caught signal 11: addr=0x3f000008 pc=0x7f8eb7841968 sp=0x7f8eb07d1d20
==49433==LeakSanitizer has encountered a fatal error.

@DaanDeMeyer DaanDeMeyer force-pushed the sanitizers branch 4 times, most recently from 24096e4 to 7dcfadb Compare May 27, 2024 11:42
@bluca
Copy link
Member

bluca commented May 27, 2024

Without sanitizer the integration test runs takes about 50 minutes, with sanitizers about 90 minutes - that's quite a bump. Also, I am very worried that by making things slower, we'll hide more race conditions (like the ones we found in the past few weeks). Hence, I'd prefer if we had one job that runs with sanitizers, and the rest without. We could maybe to a weekly run with all jobs.

@DaanDeMeyer
Copy link
Contributor Author

Without sanitizer the integration test runs takes about 50 minutes, with sanitizers about 90 minutes - that's quite a bump. Also, I am very worried that by making things slower, we'll hide more race conditions (like the ones we found in the past few weeks). Hence, I'd prefer if we had one job that runs with sanitizers, and the rest without. We could maybe to a weekly run with all jobs.

Isn't it the opposite? I'm pretty sure that the reason we uncovered all those race conditions is because the tests in Github Actions run with drastically fewer resources than they do in CentOS CI.

@bluca
Copy link
Member

bluca commented May 27, 2024

Without sanitizer the integration test runs takes about 50 minutes, with sanitizers about 90 minutes - that's quite a bump. Also, I am very worried that by making things slower, we'll hide more race conditions (like the ones we found in the past few weeks). Hence, I'd prefer if we had one job that runs with sanitizers, and the rest without. We could maybe to a weekly run with all jobs.

Isn't it the opposite? I'm pretty sure that the reason we uncovered all those race conditions is because the tests in Github Actions run with drastically fewer resources than they do in CentOS CI.

I thought they were faster here? But anyway, same thing other way around: a mix of the two would be best, so that we have more variations in the base environment

@DaanDeMeyer DaanDeMeyer force-pushed the sanitizers branch 4 times, most recently from 6ef154e to bdb09f1 Compare May 28, 2024 09:12
@@ -1339,7 +1151,6 @@ static void run_tests(RuntimeScope scope, char **patterns) {
entry(test_exec_ignoresigpipe),
entry(test_exec_inaccessiblepaths),
entry(test_exec_ioschedulingclass),
entry(test_exec_mount_apivfs),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to just skip this test case under sanitizers? I quite like having this as part of the unit test run, rather than the integration test run, as it's much faster feedback

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A unit test really shouldn't be doing what this test is doing, I think this just fits much better as an integration test. I'd much rather have us focus on speeding up the integration tests rather than trying to fit stuff that should be an integration test in a unit test.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An integration test running in a VM will never be able to run in a handful of seconds though, and it also won't be able to run at package build time. Of course I'm not asking to make it work under sanitizers, that would be way too involved, and having the same test case also in the integration test is nice. But I'd really like to keep this, and just have it skipped if sanitizers are enabled, that should be simple enough?

@DaanDeMeyer DaanDeMeyer force-pushed the sanitizers branch 5 times, most recently from 7e630ac to 3cb7239 Compare May 29, 2024 15:12
dbus-broker and dbus-daemon have not been made interchangable on
OpenSUSE so we currently end up with dbus-broker used for the system
bus and dbus-daemon for the session bus. Let's stick to dbus-daemon
on OpenSUSE until they switch to dbus-broker.
When running with sanitizers we need more memory otherwise the unit
gets OOM killed.
The test does not work under sanitizers as strace is used. Until the
test is fixed to not use strace let's skip it when running with
sanitizers.
Required since we run with DynamicUser=1.
Some tests (e.g. test-udev.py) might trigger one of our NSS modules
which means LD_PRELOAD has to be configured properly.
The test fails when running under sanitizers due to missing sanitizer
libraries. For now, let's skip the test until we can make the necessary
changes to run it under sanitizers.
System call filtering is incompatible with sanitizers so let's skip
these tests when we're built with sanitizers.
When DynamicUser= is enabled, we need LD_PRELOAD to be configured
correctly as the tests will load systemd's nss module which will complain
when built with sanitizers if the sanitizer libraries were not loaded
first.
Let's not fail if directories already exist in cp_r().
- Let's set the environment on the kernel command line so it applies
to initrd and main system.
- Let's add the necessary wrappers that are also added in test-functions.
Unlike test-functions we don't use gcc/clang to get the library path as
that requires installing gcc/clang in the initrd.
- Let's drop the hack to get journald writing to the console and have
it write to kmsg instead. We'll get the output either way.
- Stop removing libstdc++ and sanitizer libraries from Arch Linux
initrds and other images as it's required by the sanitizer libraries.
- Add a workaround for specifying extra meson options for opensuse
- Add a leak sanitizer suppression file as a workaround for a false
positive leak in verify_selinuxmnt() in libselinux. We do a soname match
because the stacktrace can't be properly symbolized on Debian.
@DaanDeMeyer DaanDeMeyer merged commit 381918e into systemd:main May 30, 2024
32 of 48 checks passed
@DaanDeMeyer DaanDeMeyer deleted the sanitizers branch May 30, 2024 12:49
@github-actions github-actions bot removed the please-review PR is ready for (re-)review by a maintainer label May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

None yet

2 participants