Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement test data-driven unsupportedModelDeviceCombination at init #118

Open
atiorh opened this issue Apr 15, 2024 · 2 comments
Open

Implement test data-driven unsupportedModelDeviceCombination at init #118

atiorh opened this issue Apr 15, 2024 · 2 comments

Comments

@atiorh
Copy link
Contributor

atiorh commented Apr 15, 2024

After specifying a minimum OS version of macOS13 and iOS16, there is still a large matrix of possible model-device configurations for deployment:

Devices have varying capabilities across:

  • Available RAM: Ranging from 4GB on iPhone 12 mini to 192GB on M2 Ultra Mac Studio
  • Chip generation: A12 (iPhone XS) to A17 Pro (iPhone 15 Pro), M1-M3 for Mac and iPads etc. , S9 SiP for Apple Watch
  • Apple Silicon vs legacy Intel Macs (still supported by macOS 14)

Model versions that have varying requirements:

  • Original variants: tiny, base, small, large distil-large
  • Argmax variants: compressed, turbo and compressed+turbo

Not all model versions will successfully run on all Apple devices. Some combinations may be enabled in the future by Argmax work. Some combinations may be disabled due to OS regressions. Hence, we would like to automatically test all combinations before each release using the testing infrastructure being laid out in #99 by @Abhinay1997 and throw unsupportedModelDeviceCombination for combinations that fail to pass all the tests during WhisperKit pipeline init time.

Taking it a step further, the WhisperKit pipeline object should be able to take an optional "peakMemoryAllowed" argument at init time and compare that against the peak memory recorded during test on the particular model-device combination and throw unsupportedModelDeviceCombination.

For the time being, we will disable all Intel Macs. Please reach out if you need Intel Macs support for your product.

@iandundas
Copy link

That sounds great! Additionally it would be great if there was maybe a function that took an enum parameter indicating what the priority was (e.g. speed, quality, low-memory), and WhisperKit would return a recommended model / a short short list of recommended models that would be the best fit for the current system.

@atiorh
Copy link
Contributor Author

atiorh commented May 22, 2024

Sounds great, thanks for the input @iandundas! We are going to start working on this soon :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants