Lack of systemctl is the least of the problems here - irqbalance is running in the host, not the container. We can add --restart-irqbalance

Other problems:

• some /proc/sys controls may be virtualized, so tuning them in a container won't affect the host
• the network interfaces can be virtualized (can be worked around by using --net host)
• we read from and write to /etc files (e.g. /etc/{default,sysconfig}/irqbalance) which are meaningless in the container

This stuff really should run on the host.

We are aware of it, we run perftune as privileged container inside host network namespace and we trick systemd by chrooting into host filesystem to make sure it restarts service running on the host.

Well, then can't you pick up systemctl from the host?

In theory we can, although it would be more fragile, as it we would have to worry about potential symlinks, non statically linked binaries, ABI incompatiblity.
These may change over time and be different on every deployment.
Whole point of the container image is that it contains every dependency so it can be ported across different machines without issues.

In theory we can, although it would be more fragile, as it we would have to worry about potential symlinks, non statically linked binaries, ABI incompatiblity. These may change over time and be different on every deployment. Whole point of the container image is that it contains every dependency so it can be ported across different machines without issues.

But here you're violating it directly, by chrooting into the host filesystem. If you use systemctl from the image, then you'll have one version of systemctl talking to another version of systemd.

I don't think this is great one way or the other.

Remember, we don't control the host OS.

Effectively, the host may not have python at all, so we can't copy perftune.py to the host (+ python libs would be a mess to copy correctly). So perftune has to be run from the container, which leaves us with:

a) use systemctl from the container and rely on systemctl->systemd working with a skew

The D-bus API is also used by some libs, so this may have some essence of compatibility

b) somehow expose systemctl from the host into the container filesystem

This is problematic on several front as @zimnx already mentioned. Mostly systemctl is a dynamically linked binary and copying all the deps is quite problematic at best.

ldd $( command -v systemctl )
	linux-vdso.so.1 (0x00007ffe87d5d000)
	libsystemd-shared-254.9-1.fc39.so => /usr/lib64/systemd/libsystemd-shared-254.9-1.fc39.so (0x00007f7b6ca22000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7b6c9ed000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f7b6c80b000)
	libacl.so.1 => /lib64/libacl.so.1 (0x00007f7b6c801000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f7b6c7c5000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f7b6c7bb000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f7b6c780000)
	libkmod.so.2 => /lib64/libkmod.so.2 (0x00007f7b6c764000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f7b6c742000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007f7b6c6f1000)
	libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f7b6c218000)
	libpam.so.0 => /lib64/libpam.so.0 (0x00007f7b6c207000)
	libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007f7b6c1e6000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f7b6c1b9000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f7b6c186000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f7b6c0ca000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f7b6bfe9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7b6ce48000)
	libattr.so.1 => /lib64/libattr.so.1 (0x00007f7b6bfdf000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f7b6bfc5000)
	libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f7b6bf96000)
	libeconf.so.0 => /lib64/libeconf.so.0 (0x00007f7b6bf89000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f7b6beee000)
	libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f7b6bee3000)

Then you are left with libc ABI compatibility, which is usually where people give up.

I think we need to build a container with the needed tooling (which should not need systemd btw) to perform the initial configuration. A-la (or just use) NTO - node tuning operator (https://github.com/openshift/cluster-node-tuning-operator/tree/master ) and such.

BTW, why is it trying to use systemd?
From https://github.com/scylladb/seastar/blob/72fe971fe8d8484a210ff487b68a7831b4d0e2f9/scripts/perftune.py#L204 :

if systemd:
        perftune_print("Restarting irqbalance via systemctl...")
        run_one_command(['systemctl', 'try-restart', 'irqbalance'])
    else:
        perftune_print("Restarting irqbalance directly (init.d)...")
        run_one_command(['/etc/init.d/irqbalance', 'restart'])

Maybe it needs to be invoked differently?
(Why are we even running irqbalance in the container?)

irqbalancer is running on every AWS host as systemd service, not within container. We need to change its configuration on host to prevent it from messing around with our IRQ settings, and then initiate restart to pick it up, and we do it from the container.

irqbalancer is running on every AWS host as systemd service, not within container. We need to change its configuration on host to prevent it from messing around with our IRQ settings, and then initiate restart to pick it up, and we do it from the container.

OK. Here's one way I'm aware of doing it - using nsenter - https://github.com/gluster/gluster-containers/blob/master/CentOS/exec-on-host.sh .

I'm unsure why it got broken in 5.4.1 (scylla-5.4.0...scylla-5.4.1 ) - @yaronkaikov ? (btw, we use supervisor, where is systemctl coming from?)

@syuu1228 Can you please have a look ?

@syuu1228 - any updates? I'm mostly concerned how it got broken (do we know how).

ScyllaDB container image doesn't have systemctl command installed, while this command is used by perftune.py script shipped within the same image.

Saving the original irqbalance configuration is in /etc/sysconfig/irqbalance.scylla.orig
Restarting irqbalance: going to ban the following IRQ numbers: 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 45, 46, 47, 48, 49, 50, 51, 52 ...
Restarting irqbalance via systemctl...
ERROR: [Errno 2] No such file or directory: 'systemctl'. Your system can't be tuned until the issue is fixed.

https://github.com/scylladb/scylla-seastar/blob/9d44e5ebfb4c0fe54e2d6812a3968a2f629c34e0/scripts/perftune.py#L206

Well, Scylla Docker image does run perftune.py, since the script does not support container.
On latest version of perftune.py, even if we run it manually in container it will cause following error:

# /opt/scylladb/scripts/perftune.py --tune disks --dir /var/lib/scylla/data --dir /var/lib/scylla/commitlog --dir /var/lib/scylla/hints --dir /var/lib/scylla/view_hints --tune net --nic eth0 --get-cpu-mask
ERROR: Not supported virtual device eth0. Your system can't be tuned until the issue is fixed.

I checked 5.4.1 and 5.4.4 image, it does not executed perftune.py on startup.

Also, about optimizing Host OS using perftune.py, I think user should install perftune.py and run it manually.
To install perftune.py, maybe we don't need new package for this just use existing scylla-server package.

To install perftune.py, maybe we don't need new package for this just use existing scylla-server package.

It needs to be in its own privileged container, with its deps, perhaps, but we can't expect any other format other than a container (for K8S for example)

Also, about optimizing Host OS using perftune.py, I think user should install perftune.py and run it manually.

This is not the case on Kubernetes. There are many nodes with an OS that doesn't have ScyllaDB repositories, not even the same packaging system (if any), or is immutable.

We need to re-ship it in ScyllaDB container image as the change broke the images used with our operator. This isn't a question of whether we should add it but a bug to bring it back as it affects customers on newer versions.

If we want to talk about a future path, I am open to it, but let's first fix the regression.

@syuu1228 , @yaronkaikov - ping - where are we with this? I'd like to see it in a near-future 5.4.x.

I was thinking maybe I need to rewrite perftune.py to support our container image which does not have neither systemd nor sysvinit, but I found that /etc/init.d/irqbalance actually able to use without any errors on our container image.
So the problem is perftune.py has a bug which mistakenly detect systemd on our container even we don't have it.
I sent a PR to fix the bug:
scylladb/seastar#2215

After the fix applied, able to run perftne.py without an error:

# /etc/init.d/irqbalance start
 * Starting SMP IRQ Balancer: irqbalance                                        IRQ 186 was BANNED.
# /opt/scylladb/scripts/perftune.py --mode mq --nic enp0s31f6 --ve
rbose --tune net --tune system
File /etc/default/irqbalance.scylla.orig already exists - not overwriting.
Restarting irqbalance: going to ban the following IRQ numbers: 186 ...
Restarting irqbalance directly (init.d)...
Setting a physical interface enp0s31f6...
Executing: ethtool -L enp0s31f6 rx 24
Executing: ethtool -L enp0s31f6 combined 24
Distributing all IRQs
Setting mask 00000001 in /proc/irq/186/smp_affinity
Setting mask 00ffffff in /sys/class/net/enp0s31f6/queues/rx-0/rps_cpus
Setting net.core.rps_sock_flow_entries to 32768
Setting limit 32768 in /sys/class/net/enp0s31f6/queues/rx-0/rps_flow_cnt
Trying to disable ntuple filtering HW offload for enp0s31f6...ok
Setting mask 00ffffff in /sys/class/net/enp0s31f6/queues/tx-0/xps_cpus: failed to write into /sys/class/net/enp0s31f6/queues/tx-0/xps_cpus: (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7f72b56cb9c0>)
Writing '4096' to /proc/sys/net/core/somaxconn
Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog

Note that the container need to run with privileged mode and host mode network, like:

docker run --privileged --network=host -ti <docker image>

I don't understand why one would run perftune.py in the container.
As @avikivity correctly mentioned/hinted it's supposed to be executed on the host only.

I don't understand why one would run perftune.py in the container. As @avikivity correctly mentioned/hinted it's supposed to be executed on the host only.

K8S ... (however, one would argue even if it runs in a container, it should be running in its own, perhaps privileged, container, with nothing but the perftune scripts need to tune the system)

I don't understand why one would run perftune.py in the container. As @avikivity correctly mentioned/hinted it's supposed to be executed on the host only.

K8S ... (however, one would argue even if it runs in a container, it should be running in its own, perhaps privileged, container, with nothing but the perftune scripts need to tune the system)

It's not supposed to be executed from K8S containers either - for the same reasons.
perftune.py tunes the Host system, there is nothing to tune from inside the container since it's not supposed to be privileged enough.
Unless I'm missing something that's the whole point about containers, they are completely isolated from one another and (should) have no means of tweaking the resources used by other containers, like Host-level resources. And perftune.py is all about tweaking the later.

Why it's not supposed to be executed from container?

Container is just another process spawned in different root with different namespaces. If someone launches perftune with host namespaces then it behaves the same as regular process running on host, being able to tune host system. I don't see why anyone would restrict it.
This is what we were doing for years and it worked well until dependency command was removed from container image.

I was thinking maybe I need to rewrite perftune.py to support our container image which does not have neither systemd nor sysvinit, but I found that /etc/init.d/irqbalance actually able to use without any errors on our container image. So the problem is perftune.py has a bug which mistakenly detect systemd on our container even we don't have it. I sent a PR to fix the bug: scylladb/seastar#2215

After the fix applied, able to run perftne.py without an error:

# /etc/init.d/irqbalance start
 * Starting SMP IRQ Balancer: irqbalance                                        IRQ 186 was BANNED.
# /opt/scylladb/scripts/perftune.py --mode mq --nic enp0s31f6 --ve
rbose --tune net --tune system
File /etc/default/irqbalance.scylla.orig already exists - not overwriting.
Restarting irqbalance: going to ban the following IRQ numbers: 186 ...
Restarting irqbalance directly (init.d)...
Setting a physical interface enp0s31f6...
Executing: ethtool -L enp0s31f6 rx 24
Executing: ethtool -L enp0s31f6 combined 24
Distributing all IRQs
Setting mask 00000001 in /proc/irq/186/smp_affinity
Setting mask 00ffffff in /sys/class/net/enp0s31f6/queues/rx-0/rps_cpus
Setting net.core.rps_sock_flow_entries to 32768
Setting limit 32768 in /sys/class/net/enp0s31f6/queues/rx-0/rps_flow_cnt
Trying to disable ntuple filtering HW offload for enp0s31f6...ok
Setting mask 00ffffff in /sys/class/net/enp0s31f6/queues/tx-0/xps_cpus: failed to write into /sys/class/net/enp0s31f6/queues/tx-0/xps_cpus: (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7f72b56cb9c0>)
Writing '4096' to /proc/sys/net/core/somaxconn
Writing '4096' to /proc/sys/net/ipv4/tcp_max_syn_backlog

Note that the container need to run with privileged mode and host mode network, like:

docker run --privileged --network=host -ti <docker image>

Ah, this was wrong.
I thought it can work even probably wrong way to use, but I realized it may not work.
When we pass --privileged --network=host we can have enough privilege to access HW information on /sys and /proc, but Host OS likely already running irqbalance on Host OS side.
If we modify IRQ configurations from both side, it may cause unexpected behavior, so we should not do that.

So as we expected, user should install & run pertune.py on HostOS not the container.

If we modify IRQ configurations from both side, it may cause unexpected behavior, so we should not do that.

That's why you should mount host irqbalancer configuration in Bidirectional mode to change irqbalancer on host, not within the container.

Again, perftune can be run from the container, but it needs special configuration. We know what we need to change, all we need is image to contain all dependency commands that perftune uses.

So as we expected, user should install & run pertune.py on HostOS not the container.

1. This is unacceptable for K8S
2. I think you should use 'nsenter' (I think!) to run it on the host, and not within the container. See https://zmalik.dev/posts/packet-drop
3. Others can configure all of this, on K8S. We may want to look at NTO (https://github.com/openshift/cluster-node-tuning-operator ) and such.

Affected versions are 5.4.1 onwards - 5.4.0 is not affected.

I investigated what is the difference between 5.4.0 and 5.4.1 Docker image, since I have no idea how does it work in 5.4.0 and how it's break in 5.4.1.

Both versions are using exactly same revision of perftune.py, only difference is installed package.
Seems like 5.4.0 has /usr/bin/systemctl which is provided by systemd package, and 5.4.1 doesn't.
And it's looks like the difference comes from removing ssh-server:
7ce6962

I tested to run perftune.py with irqbalance on both version, both are causing error but 5.4.0 does not cause "No such file or directory 'systemctl'", since it has systemctl.
It fails to connect systemd instead, since systemd does not running in the container.

• 5.4.0

root@monster:/# /etc/init.d/irqbalance start
 * Starting SMP IRQ Balancer: irqbalance                                 [ OK ] 
root@monster:/# /opt/scylladb/scripts/perftune.py --nic enp0s31f6 --tune net --tune system
Saving the original irqbalance configuration is in /etc/default/irqbalance.scylla.orig
Restarting irqbalance: going to ban the following IRQ numbers: 186 ...
Restarting irqbalance via systemctl...
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
ERROR: Command 'systemctl try-restart irqbalance' returned non-zero exit status 1.. Your system can't be tuned until the issue is fixed

• 5.4.1

root@monster:/# /etc/init.d/irqbalance start
 * Starting SMP IRQ Balancer: irqbalance                                 [ OK ]
root@monster:/# /opt/scylladb/scripts/perftune.py --nic enp0s31f6 --tune net --tune system
Saving the original irqbalance configuration is in /etc/default/irqbalance.scylla.orig
Restarting irqbalance: going to ban the following IRQ numbers: 186 ...
Restarting irqbalance via systemctl...
ERROR: [Errno 2] No such file or directory: 'systemctl'. Your system can't be tuned until the issue is fixed.

I am not sure this really can fix the problem (since it causing another error), but we can revert to the previous behavior if we add systemd package to the Docker image again.

Also note that perftune.py will not try to restart irqbalance if it's not currently running:

    # return early if irqbalance is not running
    if not is_process_running('irqbalance'):
        perftune_print("irqbalance is not running")
        return

Good findings @syuu1228 - but let's fix the real issue - we need perftune to work well here. If we are (re)adding systemd and just masking the issue, we are not solving anything here.

Have you mounted host dbus inside the container so that it connects to host systemd?

I think this should mimic what we do in Operator:

sudo podman run --privileged --network=host --pid=host --rm -e "SYSTEMD_IGNORE_CHROOT=1"  -v "/etc/systemd:/etc/systemd" -v "/sys/class:/sys/class" -v "/sys/devices:/sys/devices" -v "/lib/systemd/system:/lib/systemd/system" -v "/var/run/dbus:/var/run/dbus" -v "/run/systemd/system:/run/systemd/system" --entrypoint=/opt/scylladb/scripts/perftune.py  docker.io/scylladb/scylla:5.4.0 --nic eno2 --tune net --tune system 

When irqbalancer is running on the host, additional volume is needed to change the right configuration file:

sudo podman run --privileged --network=host --pid=host --rm -e "SYSTEMD_IGNORE_CHROOT=1" -v "/etc/sysconfig/irqbalance:/etc/sysconfig/irqbalance" -v "/etc/systemd:/etc/systemd" -v "/sys/class:/sys/class" -v "/sys/devices:/sys/devices" -v "/lib/systemd/system:/lib/systemd/system" -v "/var/run/dbus:/var/run/dbus" -v "/run/systemd/system:/run/systemd/system" --entrypoint=/opt/scylladb/scripts/perftune.py  docker.io/scylladb/scylla:5.4.0 --nic eno2 --tune net --tune system

Why it's not supposed to be executed from container?

Container is just another process spawned in different root with different namespaces. If someone launches perftune with host namespaces then it behaves the same as regular process running on host, being able to tune host system. I don't see why anyone would restrict it. This is what we were doing for years and it worked well until dependency command was removed from container image.

Why would we package systemctl in the container image? There isn't a systemd here.

If you want a systemctl to talk to an outside systemd, bind-mount it (or some script to forward the command to the outside systemd).

Why it's not supposed to be executed from container?
Container is just another process spawned in different root with different namespaces. If someone launches perftune with host namespaces then it behaves the same as regular process running on host, being able to tune host system. I don't see why anyone would restrict it. This is what we were doing for years and it worked well until dependency command was removed from container image.

Why would we package systemctl in the container image? There isn't a systemd here.

If you want a systemctl to talk to an outside systemd, bind-mount it (or some script to forward the command to the outside systemd).

This was already covered in #17787 (comment)

Can we please focus on distinguishing:

1. Fixing the regression ASAP (bringing back systemctl)
2. Figuring a different approach for the future, if needs be

But those are separate tasks and I feel like the timeline for 1. is quite beyond what it should be.

I wrote a scripts to implement this without changing perftune.py since this will very specific to Scylla Operator not generic usage of the container, here are the code:

These two files are dummy irqbalance service which run in the container, let perftune.py to detect irqbalance is running:

• /etc/supervisord.conf.d/dummy-irqbalance.conf

[program:dummy-irqbalance]
command=/usr/local/bin/dummy_irqbalance
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

• /usr/local/bin/dummy_irqbalance

#!/usr/bin/env python3
import os
import sys
import signal

if __name__ == '__main__':
    with open('/proc/self/comm', 'w') as f:
        f.write('irqbalance')
    signal.pause()

And here is dummy /etc/init.d/irqbalance which perftune.py will invoke:

• /etc/init.d/irqbalance

#!/bin/bash

HOST_ROOTFS="/rootfs"
HOST_ESCAPE="nsenter --root=${HOST_ROOTFS} --mount=${HOST_ROOTFS}/proc/1/ns/mnt --ipc=${HOST_ROOTFS}/proc/1/ns/ipc --net=${HOST_ROOTFS}/proc/1/ns/net --uts=${HOST_ROOTFS}/proc/1/ns/uts"

if [[ ! -d "${HOST_ROOTFS}" ]]; then
    echo "Host rootfs not found at ${HOST_ROOTFS}"
fi

if ! ${HOST_ESCAPE} /bin/true ; then
    echo "Could not run a command on the host"
fi

case "$1" in
    restart)
	if [[ "$(cat ${HOST_ROOTFS}/proc/1/comm)" = "systemd" ]]; then
            ${HOST_ESCAPE} /usr/bin/systemctl restart irqbalance
	else
            ${HOST_ESCAPE} /etc/init.d/irqbalance restart
	fi
	if [[ $? ]]; then
	    exit 1
	fi
        ;;
    *)
        N=/etc/init.d/irqbalance
        echo "Usage: $N {restart}" >&2
        exit 1
        ;;
esac

exit 0

However, I found that nsenter actually cannot able to run HostOS's systemctl:

# nsenter --root=/rootfs --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /usr/bin/systemctl restart irqbalance
Failed to connect to bus: No data available

It can run basic command like ls in HostOS, maybe dbus is not working correctly on nsenter.

• Fixing the regression ASAP (bringing back systemctl)

As I described at #17787 (comment), the difference from 5.4.0 is dropping systemd package, so let's just add it again.
I sent PR at: #18643

And then we can continue disccuss how to perform IRQ tunup on Scylla Operator on differeunt issue, since it will not be urgent issue.

It can run basic command like ls in HostOS, maybe dbus is not working correctly on nsenter.

I assume you still need to run it privileged. Which I keep re-iterating should be our long term solution - we run perftune and whatnot setup for the host as a separate, privileged container.

I agree that if it's a regression we need to fix it first and come up with a better solution later.