New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Experiments for speculative_decoding #680
Comments
Will work on this ASAP🏃♂️. Just for clarification, you want to test spec inference on that specific setting (first 100 inputs from alpaca test dataset as prompts) when model=your_model and draft_model=your_draft_model, or some other settings, or something else? |
Thanks for the reply. |
I am wondering if the problem has been resolved. If you need anything, please feel free to let us know 😄 |
"We tested the speculative inference using the first 100 inputs from alpaca test dataset as prompts. When model=gpt2-xl, draft_model=gpt2".
I want to test speedup for my own model and draft_model. Where can I found the scripts for this?
Thank you in advance.
The text was updated successfully, but these errors were encountered: