TRT8 -> TRT10, severe performance degradation.

[Mr-Nineteen]

• Verified model: Search and recommendation model.

• Inference time for TRT8 model：

latency	milliseconds
50%	12.860
60%	13.192
70%	13.513
80%	14.229
90%	15.736
95%	16.618
99%	19.204

• Inference time for TRT10 model：

latency	milliseconds
50%	37.349
60%	39.082
70%	41.100
80%	43.191
90%	46.483
95%	49.919
99%	56.780

• Analyze the main time-consuming part:

    // Prepare inputs
    for (size_t i = 0; i < inputs.size(); ++i) {
      const TFTensor& input = inputs[i];
      const std::string& input_name = input_names_[i];
      if (!context_->setTensorAddress(input_name.c_str(), input.data())) {
        return tf::errors::Internal(
            carbon::Printf("Failed to `setTensorAddress` for input name [%s]",
                           input_name.c_str()));
      }
      nvinfer1::Dims dims;
      TensorShapeToDims(input.shape(), &dims);
      if (!context_->setInputShape(input_name.c_str(), dims)) {
        return tf::errors::Internal(carbon::Printf(
            "Failed to `setInputShape` for name [%s]", input_name.c_str()));
      }
    }

It is this setInputShape API that causes the increase in time consumption. If there are nearly a thousand inputs, the time consumption increases by tens of milliseconds.

[lix19937]

Why not use trtexec for benchmark ?

[Mr-Nineteen]

@lix19937

TRT is integrated into a self-developed framework, with its own set of benchmark tests and verification processes.

This issue is solely due to the upgrade to TRT10.

[zerollzeng]

Could you please provide a reproduce? @Mr-Nineteen