You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have converted a norma t5 small model to Onnx using onnxruntime 1.15.1, python =3.10.12 in Intel Processor and AMD processor but received different response! Please let me know how to use the same for AMD processors
Input Seq: translate English to German: The house is wonderful.
Results from Intel based processors:
###T5 Small Output
###Actual: Das Haus ist wunderbar.
ONNX Result: Das Haus ist wunderbar.
###Quantized: Das Haus ist wunderbar.
###Neural Compressor: Intel: Das Haus ist wunderbar.
Quantization code with Calibration Data Loader/ Data Set
from neural_compressor import quantization
from neural_compressor.config import PostTrainingQuantConfig,TuningCriterion
from transformers import AutoModelForSeq2SeqLM
model="/t5-smallonnx"
conf = (
PostTrainingQuantConfig(approach="dynamic"
)) # default approach is "auto", you can set "dynamic":PostTrainingQuantConfig(approach="dynamic")
q_model = quantization.fit(
model=model,
conf=conf
)
q_model.save("/t5-small-neural-compressor.onnx")
Neural compressor version: 2.4.1
The text was updated successfully, but these errors were encountered:
Hi chensuyue,
Thank you for your response!
It is been quantized in Intel based HARDWARE only and inferred in AMD processor! It is working but generation is not as expected!
Oh, I see, thanks for your reply, I will track this issue. But fix the inference issue on AMD processor is not a high priority task for my team, we welcome the contribution from community to fix this issue.
Hi Team,
I have converted a norma t5 small model to Onnx using onnxruntime 1.15.1, python =3.10.12 in Intel Processor and AMD processor but received different response! Please let me know how to use the same for AMD processors
Input Seq: translate English to German: The house is wonderful.
Results from Intel based processors:
###T5 Small Output
###Actual: Das Haus ist wunderbar.
ONNX Result: Das Haus ist wunderbar.
###Quantized: Das Haus ist wunderbar.
###Neural Compressor: Intel: Das Haus ist wunderbar.
Results from AMD based processors:
##AMD
###ONNNX: Das Haus ist wunderbar.
###Quantized: Das Haus ist wunderbar.
###Neural Compressor: DOMIEIE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE IE
Code used for conversion:
Quantization code with Calibration Data Loader/ Data Set
from neural_compressor import quantization
from neural_compressor.config import PostTrainingQuantConfig,TuningCriterion
from transformers import AutoModelForSeq2SeqLM
model="/t5-smallonnx"
conf = (
PostTrainingQuantConfig(approach="dynamic"
)) # default approach is "auto", you can set "dynamic":PostTrainingQuantConfig(approach="dynamic")
q_model = quantization.fit(
model=model,
conf=conf
)
q_model.save("/t5-small-neural-compressor.onnx")
Neural compressor version: 2.4.1
The text was updated successfully, but these errors were encountered: