Add instructions to convert Hugging Face models to PyTorch #3523
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3523
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 12f514e with merge base 1b73db4 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
examples/models/llama2/README.md
Outdated
| sd = convert_weights.tune_to_meta(sd['model']) | ||
|
|
||
| print("saving checkpoint") | ||
| torch.save(sd, "/destination/dir/checkpoint.pth") |
There was a problem hiding this comment.
np: /the/destination/dir/checkpoint.pth
examples/models/llama2/README.md
Outdated
| @@ -117,6 +117,33 @@ You can export and run the original Llama3 8B model. | |||
|
|
|||
| Due to the larger vocabulary size of Llama3, we recommend quantizing the embeddings with `--embedding-quantize 4,32` to further reduce the model size. | |||
|
|
|||
| ### Option D: Download models from Hugging Face and convert | |||
There was a problem hiding this comment.
and convert from safetensor format to state dict
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
As titled. It's pretty common that users download the LLM models in safetensor format. Add instructions and example script to convert them to PyTorch format so that export_llama script can accept. It leverages the utils from TorchTune.
Thanks @l3utterfly and @kartikayk for the discussions and suggestions!
More context in #3303