Inference using TensorRT

[mon28]

Hi,

I have been exploring models that I can fine tune with my own data to provide embeddings for the task of pair wise similarity calculation.
My data looks like: [title][space][url]. I do not have domain specific information. There are two questions that I have:

1. What would the instruction look like in this case for training and inference scenarios?
2. I wish to deploy this in production and use TensorRT for inference. Could you help me with an example of how that would work out?

Thanks,
Mon

[hongjin-su]

Hi, Thanks a lot for your interest in the INSTRUCTOR!

1. In the section 2.3 of our paper, we provide the template to write instructions.
2. There are several good tutorials that talk about TensorRT:
   https://developer.nvidia.com/tensorrt-getting-started
   https://medium.com/ching-i/tensorrt-%E4%BB%8B%E7%B4%B9%E8%88%87%E5%AE%89%E8%A3%9D%E6%95%99%E5%AD%B8-45e44f73b25e
   In general, the technique that applies to transformer models may also be applicable for the INSTRUCTOR model, as it has very similar architecture to T5-encoder.

Hope this helps!