nemo2riva export doesn't work with FastPitch

[fr0zenshard]

Description

When I try to export a model to Riva (using nemo2riva==2.14.0), the FastPitch model export doesn't work (meanwhile, the HiFiGAN export works wonderfully).
I've tried various combinations of onnx and onnxruntime, combinations of NeMo and nemo2riva versions, but in the end, nothing works and the following error always pops up:

onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Failed to load model with error: Invalid tensor data type 0.

Steps/Code to reproduce bug

Using NeMo image (1.21.0, 1.22.0, 1.23.0) + installed nemo2riva==2.14.0:

nemo2riva --out $(OUT_RIVA).riva $(NEMO_MODEL) --key tlt_encode --runtime-check

Additional info

• Without --runtime-check you can get exported .riva model, but it doesn't work by the end (and the model size is huge, also)
• Connected issue without provided solution (was automatically closed) - Unable to convert any nemo file using nemo2riva #8569
• Connected issues from nemo2riva - Unable to convert any Model from .nemo to .riva nvidia-riva/nemo2riva#38, Conformer CTC converted with nemo2riva 2.13.1 deployed on Riva 2.13.1 fails to load nvidia-riva/nemo2riva#36

The failed results mentioned by the authors of the issue above can be obtained by:

1. Using exported .riva FastPitch from nemo2riva without a runtime_check
2. Downgrading onnx_graphsurgeon to version 0.3.27

In such an environment, the build and deploy of rmir will occur, but at the start of the Riva server, there will be either an:

• IR compatibility issue (version 9, not 8)
• OR -UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test/1/model.onnx failed:Invalid tensor data type 0.

[fr0zenshard]

I've tried trained models from previous version of NeMo and newly trained within version 1.23.0 and etc.

[fr0zenshard]

As I understand, problem is in: nemo2riva/nemo2riva/patches/tts/fastpitch.py at r2.15.0 · nvidia-riva/nemo2riva. Runtime didn't catch this additional params and runtime-check failing

[fr0zenshard]

hey?