You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I noticed that when supplying a target_prefix to the translate_batch or generate_tokens method, the latencies for generating the supplied tokens is equivalent to the situation where they are not provided, while I would expect negligible latency because those tokens don't require any generation steps. I'm expecting the first step to be the generation of the token after the prefix tokens.
Am I missing something, or is this due to an inefficiency in ctranslate2's generation logic ?
Thanks,
Simon
The text was updated successfully, but these errors were encountered:
If you specified the target_prefix, it would decode once in a step then generate one by one with the next steps. Without target_prefix, it would generate one by one token. In theory, it have to run faster in case of using target_prefix. Could you test with a long prefix ?
Hello,
I noticed that when supplying a target_prefix to the translate_batch or generate_tokens method, the latencies for generating the supplied tokens is equivalent to the situation where they are not provided, while I would expect negligible latency because those tokens don't require any generation steps. I'm expecting the first step to be the generation of the token after the prefix tokens.
Am I missing something, or is this due to an inefficiency in ctranslate2's generation logic ?
Thanks,
Simon
The text was updated successfully, but these errors were encountered: