[CI] elasticsearch-ci/7.17.22 / bwc-snapshots-windows fails

[ldematte]

CI Link

https://gradle-enterprise.elastic.co/s/au3yen3ihluxs

Repro line

N/A

Does it reproduce?

Didn't try locally, but it seems to fail pretty reliably (at least on my PR).

Applicable branches

7.17

Failure history

No response

Failure excerpt

The failure message is:

Execution failed for task ':qa:ccs-rolling-upgrade-remote-cluster:v7.17.22#oldClusterTest'.
> process was found dead while waiting for ports files, node{:qa:ccs-rolling-upgrade-remote-cluster:v7.17.22-local-0}

The root cause seems this:

[2024-05-09T16:15:33.909848600Z] [BUILD] Starting Elasticsearch process	
»  May 09, 2024 4:15:40 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>	
»  WARNING: COMPAT locale provider will be removed in a future release	
»   ↑ repeated 2 times ↑	
» [2024-05-09T16:15:58,077][ERROR][o.e.b.Elasticsearch      ] [v7.17.22-local-0] fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchException: Failed to bind service	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:283)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.Node.<init>(Node.java:192)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)	
»  Caused by: org.elasticsearch.gateway.CorruptStateException: Format version is not supported. Upgrading to [8.15.0] is only supported from version [7.17.0].	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.env.NodeEnvironment.checkForIndexCompatibility(NodeEnvironment.java:517)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.env.NodeEnvironment.upgradeLegacyNodeFolders(NodeEnvironment.java:416)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:309)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:511)	
»  	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:258)	
»  	... 4 more	
»  	
»  ERROR: Elasticsearch did not exit normally - check the logs at C:\bk\qa\ccs-rolling-upgrade-remote-cluster\build\testclusters\v7.17.22-local-0\logs\v7.17.22-local.log

Tagging both Core/Infra and Distributed, as it could be a version compatibility issue or a persisted cluster state issue - the comment line above the error says:

// We are upgrading the cluster, but we didn't find any previous metadata. Corrupted state or incompatible version.

Curiously, this seems to happen on Windows only?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ldematte]

Lacking a better alternative, here is an "empty" PR on main that shows the issue: #108490