[CI] DockerTests test500Readiness failing

[joegallo]

Build scan:
https://gradle-enterprise.elastic.co/s/jfhv4jdyalmjm/tests/:qa:packaging:destructiveDistroTest.default-docker/org.elasticsearch.packaging.test.DockerTests/test500Readiness

Reproduction line:

null

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.packaging.test.DockerTests#test500Readiness

Failure excerpt:

java.lang.AssertionError: (No message provided)

  at __randomizedtesting.SeedInfo.seed([C08005FCAB509171:10D51055CE402A98]:0)
  at org.junit.Assert.fail(Assert.java:87)
  at org.junit.Assert.assertTrue(Assert.java:42)
  at org.junit.Assert.assertTrue(Assert.java:53)
  at org.elasticsearch.packaging.test.DockerTests.test500Readiness(DockerTests.java:1222)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[joegallo]

Some other failures

[joegallo]

It got me on #108518 and @benwtrent on #108522 and #107047. And @ldematte on #108521.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

This test is related to the readiness probe so adding the core/infra label to this. Not sure if any changes have been made in that area lately. Also muting for now.

[mark-vieira]

@rjernst As far as I can tell the test waits for ES to report green, then attempts to connect on the readiness port and fails. Is there any scenario in which the node would report as green but the readiness port would not be listening? Is there some kind of race condition here?

[rjernst]

Readiness is orthogonal to health. It waits for 2 conditions to be met:

• A master node is elected
• File settings have been applied

If the cluster is green, a master node should be there. So that leaves file settings, but this test doesn't use file settings, so there shouldn't be any waiting.

I'm attempting to gather more information but dumping the ES log file before we fail the test:
#108587