perftune.yaml is not created on AL2 and cause to Scylla Doctor collector failure

[juliayakovlev]

Packages

Scylla version: 5.5.0~dev-20240509.28791aa2c1d3 with build-id 9e34b204407d7acc91d3980f060cbdfb6a927bdf
Kernel Version: 4.14.165-131.185.amzn2.x86_64

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.
• It is not a regression

Recently Scylla Doctor was added to all artifact tests.

Scylla Doctor failed in all amazon2 tests with error:

perftune.yaml is not found

perftune.yaml is really not created.

I tried to run the test with latest amazon2 image (ami-0c9e7cbc8f9f6446b, eu-west-1)

,     #_
   ~\_  ####_        Amazon Linux 2
  ~~  \_#####\
  ~~     \###|       AL2 End of Life is 2025-06-30.
  ~~       \#/ ___
   ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
      ~~._.   _/
         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

[ec2-user@artifacts-amazon2-jenkins-db-node-127dc754-1 ~]$ systemctl --version
systemd 219

The problem still persist - perftune.yaml does not exists.

Looking into journalctl, found:

May 12 13:54:16 artifacts-amazon2-jenkins-db-node-127dc754-1 systemd[1]: [/usr/lib/systemd/system/scylla-server.service:18] Executable path is not absolute, ignoring: +/opt/scylladb/scripts/scylla_prepare
May 12 13:54:16 artifacts-amazon2-jenkins-db-node-127dc754-1 systemd[1]: [/usr/lib/systemd/system/scylla-server.service:20] Executable path is not absolute, ignoring: +/opt/scylladb/scripts/scylla_stop

There is commit that made the change in this part. I do not know if it has the impact here or not.

@roydahan @tzach
Should we keep AL2 support or switch to AL3, or keep both, or stop support of Amazon Linux completely?

Installation details

Cluster size: 1 nodes (i4i.large)

Scylla Nodes used in this run:

• artifacts-amazon2-jenkins-db-node-dc92ef21-1 (3.250.156.13 | 10.4.2.101) (shards: 2)

OS / Image: ami-099a8245f5daa82bf (aws: undefined_region)

Test: artifacts-amazon2-test
Test id: dc92ef21-b7b9-4433-afae-130a9800a65d
Test name: scylla-master/artifacts/artifacts-amazon2-test
Test config file(s):

• amazon2.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor dc92ef21-b7b9-4433-afae-130a9800a65d
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs dc92ef21-b7b9-4433-afae-130a9800a65d

Logs:

• db-cluster-dc92ef21.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/dc92ef21-b7b9-4433-afae-130a9800a65d/20240509_232345/db-cluster-dc92ef21.tar.gz
• sct-runner-events-dc92ef21.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/dc92ef21-b7b9-4433-afae-130a9800a65d/20240509_232345/sct-runner-events-dc92ef21.tar.gz
• sct-dc92ef21.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/dc92ef21-b7b9-4433-afae-130a9800a65d/20240509_232345/sct-dc92ef21.log.tar.gz

Jenkins job URL
Argus

[juliayakovlev]

@fruch @vladzcloudius FYI

[fruch]

@roydahan cc90ff1 is incompatible with AL2, which has systemd 219.
and to your question, there is Amazon Linux 3, they call it Amazon linx 2023 - https://aws.amazon.com/linux/amazon-linux-2023/

so @tzach, we need a call here, do we keep AL2 support (it's EOL in 14 months) ? if yes cc90ff1 need to be reverted

[mykaul]

Not sure there's a lot to triage here. I'd drop support for it.