New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
show benchmarks #5
Comments
Hi @laralove143, please give https://www.takeargmax.com/blog/whisperkit a read for the value proposition of WhisperKit. |
that blog is very useful maybe it could be shown more clearly in the readme, for example talking about its contents alternatively, some stuff from the blog could be included in the readme as well, like that demo video is very useful |
Thanks for the feedback! We will think about a better way to organize information about WhisperKit that is more accessible. We will definitely flesh out the README and docs more before stable release. |
Tracking this here: #28 |
My understanding from running llama.cpp on iOS/macOS (via Swift, including streaming) is that Metal is faster than CoreML or Metal+CoreML. There may be some other benefits to using CoreML. Maybe battery? I don't know myself |
This is certainly possible in specific cases but can not be a generally true statement. For context, WhisperKit is currently tuned for mobile and lower-end Macs where the Neural Engine is much more powerful with respect to the GPU (that Metal can harness) and Core ML is the primary framework for deploying to the Neural Engine. That being said, we are actively working on a Metal backend to complement the Core ML backend. |
Thanks for the context. Would be great to see a benchmark later. I recall the Metal-only whispercpp being faster even on lower spec devices such as iPhone but can't find the numbers at the moment. |
Here are some numbers I hadn't seen before showing far better Metal performance on an iPhone using Metal instead of CoreML. So it looks like it holds true for mobile... https://www.bjnortier.com/2023/11/17/Hello-Transcribe-3.2.html It mentions some other downsides to CoreML such as the slow caching step and unpredictable cache ejection by the OS |
the advantage of this project is that it uses CoreML for a performance gain, so showing benchmarks would solidify how much this advantage is
The text was updated successfully, but these errors were encountered: