Add BPE pre-tokenization for Command-R. #7033
Closed
+23
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for Expand details for performance related PR only
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 550 iterations"
y-axis "llamacpp:prompt_tokens_seconds"
x-axis "llamacpp:prompt_tokens_seconds" 1714650185 --> 1714650811
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 570.5, 570.5, 570.5, 570.5, 570.5, 772.11, 772.11, 772.11, 772.11, 772.11, 666.45, 666.45, 666.45, 666.45, 666.45, 712.62, 712.62, 712.62, 712.62, 712.62, 767.81, 767.81, 767.81, 767.81, 767.81, 778.27, 778.27, 778.27, 778.27, 778.27, 778.55, 778.55, 778.55, 778.55, 778.55, 811.87, 811.87, 811.87, 811.87, 811.87, 810.2, 810.2, 810.2, 810.2, 810.2, 812.02, 812.02, 812.02, 812.02, 812.02, 838.12, 838.12, 838.12, 838.12, 838.12, 868.27, 868.27, 868.27, 868.27, 868.27, 894.9, 894.9, 894.9, 894.9, 894.9, 835.57, 835.57, 835.57, 835.57, 835.57, 842.47, 842.47, 842.47, 842.47, 842.47, 842.57, 842.57, 842.57, 842.57, 842.57, 840.07, 840.07, 840.07, 840.07, 840.07, 813.54, 813.54, 813.54, 813.54, 813.54, 812.52, 812.52, 812.52, 812.52, 812.52, 812.27, 812.27, 812.27, 812.27, 812.27, 820.03, 820.03, 820.03, 820.03, 820.03, 816.47, 816.47, 816.47, 816.47, 816.47, 828.49, 828.49, 828.49, 828.49, 828.49, 833.13, 833.13, 833.13, 833.13, 833.13, 828.21, 828.21, 828.21, 828.21, 828.21, 827.08, 827.08, 827.08, 827.08, 827.08, 824.19, 824.19, 824.19, 824.19, 824.19, 823.88, 823.88, 823.88, 823.88, 823.88, 823.42, 823.42, 823.42, 823.42, 823.42, 828.42, 828.42, 828.42, 828.42, 828.42, 829.83, 829.83, 829.83, 829.83, 829.83, 827.81, 827.81, 827.81, 827.81, 827.81, 832.7, 832.7, 832.7, 832.7, 832.7, 844.49, 844.49, 844.49, 844.49, 844.49, 849.91, 849.91, 849.91, 849.91, 849.91, 858.9, 858.9, 858.9, 858.9, 858.9, 858.45, 858.45, 858.45, 858.45, 858.45, 855.88, 855.88, 855.88, 855.88, 855.88, 858.04, 858.04, 858.04, 858.04, 858.04, 859.34, 859.34, 859.34, 859.34, 859.34, 858.77, 858.77, 858.77, 858.77, 858.77, 865.77, 865.77, 865.77, 865.77, 865.77, 847.64, 847.64, 847.64, 847.64, 847.64, 846.71, 846.71, 846.71, 846.71, 846.71, 845.25, 845.25, 845.25, 845.25, 845.25, 845.64, 845.64, 845.64, 845.64, 845.64, 852.07, 852.07, 852.07, 852.07, 852.07, 851.57, 851.57, 851.57, 851.57, 851.57, 857.48, 857.48, 857.48, 857.48, 857.48, 856.26, 856.26, 856.26, 856.26, 856.26, 858.65, 858.65, 858.65, 858.65, 858.65, 862.37, 862.37, 862.37, 862.37, 862.37, 861.33, 861.33, 861.33, 861.33, 861.33, 866.91, 866.91, 866.91, 866.91, 866.91, 868.21, 868.21, 868.21, 868.21, 868.21, 867.7, 867.7, 867.7, 867.7, 867.7, 868.47, 868.47, 868.47, 868.47, 868.47, 867.71, 867.71, 867.71, 867.71, 867.71, 870.35, 870.35, 870.35, 870.35, 870.35, 872.6, 872.6, 872.6, 872.6, 872.6, 871.26, 871.26, 871.26, 871.26]
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 550 iterations"
y-axis "llamacpp:predicted_tokens_seconds"
x-axis "llamacpp:predicted_tokens_seconds" 1714650185 --> 1714650811
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.05, 36.05, 36.05, 36.05, 36.05, 37.32, 37.32, 37.32, 37.32, 37.32, 31.76, 31.76, 31.76, 31.76, 31.76, 29.58, 29.58, 29.58, 29.58, 29.58, 30.78, 30.78, 30.78, 30.78, 30.78, 31.33, 31.33, 31.33, 31.33, 31.33, 32.93, 32.93, 32.93, 32.93, 32.93, 34.07, 34.07, 34.07, 34.07, 34.07, 34.38, 34.38, 34.38, 34.38, 34.38, 34.76, 34.76, 34.76, 34.76, 34.76, 34.67, 34.67, 34.67, 34.67, 34.67, 34.46, 34.46, 34.46, 34.46, 34.46, 33.72, 33.72, 33.72, 33.72, 33.72, 32.94, 32.94, 32.94, 32.94, 32.94, 32.75, 32.75, 32.75, 32.75, 32.75, 32.54, 32.54, 32.54, 32.54, 32.54, 32.52, 32.52, 32.52, 32.52, 32.52, 32.53, 32.53, 32.53, 32.53, 32.53, 31.94, 31.94, 31.94, 31.94, 31.94, 31.55, 31.55, 31.55, 31.55, 31.55, 31.46, 31.46, 31.46, 31.46, 31.46, 31.57, 31.57, 31.57, 31.57, 31.57, 31.69, 31.69, 31.69, 31.69, 31.69, 31.33, 31.33, 31.33, 31.33, 31.33, 31.53, 31.53, 31.53, 31.53, 31.53, 31.76, 31.76, 31.76, 31.76, 31.76, 31.65, 31.65, 31.65, 31.65, 31.65, 31.11, 31.11, 31.11, 31.11, 31.11, 31.18, 31.18, 31.18, 31.18, 31.18, 31.37, 31.37, 31.37, 31.37, 31.37, 31.45, 31.45, 31.45, 31.45, 31.45, 31.59, 31.59, 31.59, 31.59, 31.59, 31.72, 31.72, 31.72, 31.72, 31.72, 31.58, 31.58, 31.58, 31.58, 31.58, 31.57, 31.57, 31.57, 31.57, 31.57, 31.35, 31.35, 31.35, 31.35, 31.35, 30.99, 30.99, 30.99, 30.99, 30.99, 31.11, 31.11, 31.11, 31.11, 31.11, 31.18, 31.18, 31.18, 31.18, 31.18, 31.32, 31.32, 31.32, 31.32, 31.32, 31.42, 31.42, 31.42, 31.42, 31.42, 31.31, 31.31, 31.31, 31.31, 31.31, 31.18, 31.18, 31.18, 31.18, 31.18, 30.64, 30.64, 30.64, 30.64, 30.64, 30.2, 30.2, 30.2, 30.2, 30.2, 29.78, 29.78, 29.78, 29.78, 29.78, 29.7, 29.7, 29.7, 29.7, 29.7, 29.78, 29.78, 29.78, 29.78, 29.78, 29.87, 29.87, 29.87, 29.87, 29.87, 29.91, 29.91, 29.91, 29.91, 29.91, 30.01, 30.01, 30.01, 30.01, 30.01, 30.04, 30.04, 30.04, 30.04, 30.04, 29.86, 29.86, 29.86, 29.86, 29.86, 29.7, 29.7, 29.7, 29.7, 29.7, 29.83, 29.83, 29.83, 29.83, 29.83, 29.93, 29.93, 29.93, 29.93, 29.93, 30.06, 30.06, 30.06, 30.06, 30.06, 30.2, 30.2, 30.2, 30.2, 30.2, 30.25, 30.25, 30.25, 30.25, 30.25, 30.27, 30.27, 30.27, 30.27, 30.27, 30.31, 30.31, 30.31, 30.31]
Details
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 550 iterations"
y-axis "llamacpp:kv_cache_usage_ratio"
x-axis "llamacpp:kv_cache_usage_ratio" 1714650185 --> 1714650811
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.19, 0.19, 0.19, 0.19, 0.19, 0.41, 0.41, 0.41, 0.41, 0.41, 0.23, 0.23, 0.23, 0.23, 0.23, 0.13, 0.13, 0.13, 0.13, 0.13, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.11, 0.11, 0.11, 0.11, 0.11, 0.17, 0.17, 0.17, 0.17, 0.17, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.11, 0.11, 0.11, 0.11, 0.11, 0.23, 0.23, 0.23, 0.23, 0.23, 0.24, 0.24, 0.24, 0.24, 0.24, 0.23, 0.23, 0.23, 0.23, 0.23, 0.21, 0.21, 0.21, 0.21, 0.21, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.33, 0.33, 0.33, 0.33, 0.33, 0.27, 0.27, 0.27, 0.27, 0.27, 0.14, 0.14, 0.14, 0.14, 0.14, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.25, 0.25, 0.25, 0.25, 0.25, 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.14, 0.14, 0.14, 0.14, 0.28, 0.28, 0.28, 0.28, 0.28, 0.33, 0.33, 0.33, 0.33, 0.33, 0.24, 0.24, 0.24, 0.24, 0.24, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.23, 0.23, 0.23, 0.23, 0.23, 0.21, 0.21, 0.21, 0.21, 0.21, 0.3, 0.3, 0.3, 0.3, 0.3, 0.2, 0.2, 0.2, 0.2, 0.2, 0.18, 0.18, 0.18, 0.18, 0.18, 0.17, 0.17, 0.17, 0.17, 0.17, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.45, 0.45, 0.45, 0.45, 0.45, 0.5, 0.5, 0.5, 0.5, 0.5, 0.45, 0.45, 0.45, 0.45, 0.45, 0.41, 0.41, 0.41, 0.41, 0.41, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.17, 0.17, 0.17, 0.17, 0.17, 0.21, 0.21, 0.21, 0.21, 0.21, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.15, 0.15, 0.15, 0.15, 0.24, 0.24, 0.24, 0.24, 0.24, 0.3, 0.3, 0.3, 0.3, 0.3, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23]
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 550 iterations"
y-axis "llamacpp:requests_processing"
x-axis "llamacpp:requests_processing" 1714650185 --> 1714650811
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0]
|
|
I tried to run the diff --git a/convert-hf-to-gguf-update.py b/convert-hf-to-gguf-update.py
index f4774003..4ad4d867 100644
--- a/convert-hf-to-gguf-update.py
+++ b/convert-hf-to-gguf-update.py
@@ -95,6 +95,14 @@ for model in models:
save_path = f"models/tokenizers/{name}/tokenizer.json"
download_file_with_auth(url, token, save_path)
+ # if downloaded file is less than 1KB, we likely need to download an LFS instead
+ if os.path.getsize(save_path) < 1024:
+ # remove the file
+ os.remove(save_path)
+ url = f"{repo}/resolve/main/tokenizer.json"
+ save_path = f"models/tokenizers/{name}/tokenizer.json"
+ download_file_with_auth(url, token, save_path)
+
if tokt == TOKENIZER_TYPE.SPM:
url = f"{repo}/resolve/main/tokenizer.model"
save_path = f"models/tokenizers/{name}/tokenizer.model" |
|
Lets goooooO! |
sealad886
suggested changes
May 2, 2024
There was a problem hiding this comment.
I've had an a-ha moment, and these look good. I've run it and got the same.
Optional: add in command-r-plus with nearly identical footprint, since the pre-tokenizers are the same (couple ways to make that happen).
One note: tokenizer.config specifies the Digits pre-tokenizer is used and individual_digits=True. There's not a regex pattern in llama.cpp explicitly matching individual digits. EVen though the tests pass, would that be a possibilty to add in for completeness sake?
|
I get why it's changed, but does it mean the old quants are broken even when using the older versions of llamacpp? I'm not sure I can face downloading and quantizing it myself. |
My understanding of the overall issue (which admittedly is not as nuanced as others') would suggest that, for the most part, yes you should re-download and re-convert and re-quantize your models. The longer version is that newer models are more likely to use newer pre-tokenizer splits. Luckily, command-r and command-r-plus appear to be nearly identical to the default pre-tokenizer's splits, so for these models it's probably okay? |
I just remembered that the imat quants will definitely be incorrect for the new version of llamacpp. I have a non-imat q6k that I was hoping to use until some kind person makes a new q8 for huggingface ;) |
Yeah exactly...lots of componding issues. I've ended up moving some of these over to Ollama as well, so now I have to untangle what came from me and what I pulled from them. But I wouldn't be surprised if the majority of Ollama models have this same issue and will need to be re-built. |
sealad886
approved these changes
May 2, 2024
|
This PR works for both Command-R version, since they have the same hash. |
Thanks, I added this update to the PR. |
had to run this several times (it'll create the folder and then error out it seems) i think cmd-r was already done at this point but i just wanted to point this out. blocked, but cmd-r is done anyway Got to convert Command R 35B into a f16 GGUF. I'm currently quantizing them and I'll test out Q5KM soon. |
|
@drummerv I am not getting any errors. convert-hf-to-gguf-update.py |
|
Superseded by PR #7063. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I read #6920 and 120cf37 and attempting to add support Command-R support.
Closes #7030 and #7040.
EDIT: I also tested Command-R+ successfully using this PR.