system_auth keyspace not found in snapshot in test_snapshot_defaults_to_all_keyspaces

[Deexie]

snapshot_test.TestSnapshotOptions.test_snapshot_defaults_to_all_keyspaces fails with:

AssertionError: assert {'system_schema', 'system_distributed_everywhere', 'system_distributed', 'ks2', 'system_auth_v2', 'ks0', 'system_traces', 'system', 'ks1'} == {'system_auth', 'system_schema', 'system_distributed_everywhere', 'ks2', 'system_distributed', 'system_auth_v2', 'ks0', 'system_traces', 'system', 'ks1'}
  Extra items in the right set:
  'system_auth'
  Full diff:
    {
     'ks0',
     'ks1',
     'ks2',
     'system',
  -  'system_auth',
     'system_auth_v2',
     'system_distributed',
     'system_distributed_everywhere',
     'system_schema',
     'system_traces',
    }

https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-consistent-topology-changes/1459/testReport/junit/snapshot_test/TestSnapshotOptions/FullDtest___full_split000___test_snapshot_defaults_to_all_keyspaces/

[nuivall]

@Deexie consistently or randomly?

[Deexie]

@Deexie consistently or randomly?

Randomly

[nuivall]

@Deexie something odd has happened with this test run, why there are 87 other tests failures there?

[Deexie]

Most of the tests failed because of timeout waiting for logs. I assumed it happened due to some external conditions. I opened issues for the failures that stood out and seemed to be specific.

[nuivall]

But this may also is kind of timeout scenario. If scylla process was starved somehow, and creation of system_auth happens asynchronously on startup, this check will simply fail.

[nuivall]

I think we can close this issue, and reopen if this occurs on non starved run.