Benchmarks

[hasanar1f]

Do we have a script to run the benchmarks mentioned in the paper? Especially for llama 7B, 13B models. Also, I can not run the llama-bench and I am getting a model loading error. Any lead?

[hodlen]

We benchmarked the performance simply via the main binary specifying 8 threads and expected output with the same set of prompts. You can use this command and check the printed performance stats on your end:

./build/bin/main -m "$model" -n "$output_len" -p "$prompt" -t "8" --ignore-eos

However, please be aware that the current open-sourced version of PowerInfer may have some differences from the internal version evaluated in our paper, particularly in terms of performance. For instance, we have simplified certain features, such as fine-grained tensor offloading and comprehensive FFN neuron offloading policy, to facilitate easier use for a broader range of users.

[hodlen]

Also, I can not run the llama-bench and I am getting a model loading error. Any lead?

We haven't tested with llama-bench yet, and there can be incompatibilities. Would mind sharing your error log so we can better pinpointing any bug?