[ML] IllegalStateException from duplicate keys when reporting machine learning usage

[jonathan-buttner]

Description

IllegalStateException has been observed when reporting ML telemetry.

java.lang.IllegalStateException: Duplicate key .elser_model_2_linux-x86_64 (attempted merging values org.elasticsearch.xpack.core.ml.action.GetTrainedModelsStatsAction$Response$TrainedModelStats@839b3935 and org.elasticsearch.xpack.core.ml.action.GetTrainedModelsStatsAction$Response$TrainedModelStats@2193bfad)
	at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.elasticsearch.ml@8.15.0/org.elasticsearch.xpack.ml.MachineLearningUsageTransportAction.addTrainedModelStats(MachineLearningUsageTransportAction.java:515)
	at org.elasticsearch.ml@8.15.0/org.elasticsearch.xpack.ml.MachineLearningUsageTransportAction.lambda$addInferenceUsage$22(MachineLearningUsageTransportAction.java:441)
	at org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
	at org.elasticsearch.server@8.15.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.server@8.15.0/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:202)
	at org.elasticsearch.server@8.15.0/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:196)
	at org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307)

The issue is likely due to trying to map the stats for each deployment of a unique model id.

The failure originates here: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MachineLearningUsageTransportAction.java#L515

The deployments are added here using the same model id: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/action/GetTrainedModelsStatsAction.java#L313

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[jonathan-buttner]

Same thing happens when trying to use GET _cat/ml/trained_models

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Duplicate key .elser_model_2 (attempted merging values org.elasticsearch.xpack.core.ml.action.GetTrainedModelsStatsAction$Response$TrainedModelStats@d89b6021 and org.elasticsearch.xpack.core.ml.action.GetTrainedModelsStatsAction$Response$TrainedModelStats@88253226)"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "Duplicate key .elser_model_2 (attempted merging values org.elasticsearch.xpack.core.ml.action.GetTrainedModelsStatsAction$Response$TrainedModelStats@d89b6021 and org.elasticsearch.xpack.core.ml.action.GetTrainedModelsStatsAction$Response$TrainedModelStats@88253226)"
  },
  "status": 500
}