-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add safe tensor support to convert-llama.py #52
Add safe tensor support to convert-llama.py #52
Conversation
Please update also |
converter/convert-llama.py
Outdated
if '/' in modelPath: | ||
modelName = modelPath.split('/')[-1] | ||
else: | ||
modelName = modelPath.split('\\')[-1] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the os.path.basename
function would be better to extract the filename.
I updated the usage a bit, though could probably mention that it would work with the hugging face repo for Llama as well. |
@DifferentialityDevelopment I'm wondering about this part: with safetensors.safe_open(model_file, framework="pt") as f:
for layer in f.keys():
layers.append({
"name" : layer,
"file" : model_file
}) Are you sure that the source model has all layers in the correct order that is expected by Distributed Llama? |
DId not check yet, will do a full convert on llama-3 8B Instruct, do a test with distributed llama and report back. |
The convert process itself does seem to work fine, but will test once it finishes python converter/convert-llama.py J:\Llama-3\Meta-Llama-3-8B-Instruct J:\Llama-3\Meta-Llama-3-8B-Instruct-Distributed q40 |
Please consider also that, some models may have a different layers order by some reason. |
I would think the order of the keys when loading a .safetensor model is the same as from the .pth file but I could be wrong, will do a bit of research. |
Your absolutely right, the layers are not necessarily in the right order, see below output of their keys, I noticed that layer 9 only appears after layer 20. Additionally they have a different name convention, so I had to change a few more things. Keys: model.embed_tokens.weight => 128256 |
I reccomend to use the same appraoch as you can see in the |
@DifferentialityDevelopment I'm closing this pull request. The |
I haven't yet updated the other model conversion scripts yet, but this allows you to convert any llama model that uses safetensor.