Combination of Core ML and Metal

[LVCSRer]

Hello. Big thanks for community effort!

I'd like to know if my understanding is right.

With Core ML & Metal, we can have 4 combination. My understanding is

• No Core ML / No Metal : all operation on CPU
• No Core ML / Use Metal : all operation on GPU
• Use Core ML / No Metal : Encoding on ANE, others on CPU
• Use Core ML / Use Metal : Encoding on ANE, others on GPU

But not sure when I see below post. How can I control Core ML for ANE, CPU, and GPU?
#1270 (comment)

[ggerganov]

Yes, this is correct - there are 4 options as you listed them.

• No Core ML / No Metal: build with make -j, run with -ng
• No Core ML / Use Metal: build with make -j, run without -ng
• Use Core ML / No Metal: build with WHISPER_COREML=1 make -j, run with -ng
• Use Core ML / Use Metal: build with WHISPER_COREML=1 make -j, run without -ng

To control what hardware Core ML uses, you will need to edit the following source file:

whisper.cpp/coreml/whisper-encoder.mm

Lines 24 to 30 in a3d0aa7

	// select which device to run the Core ML model on
	MLModelConfiguration *config = [[MLModelConfiguration alloc] init];
	config.computeUnits = MLComputeUnitsCPUAndGPU;
	//config.computeUnits = MLComputeUnitsCPUAndNeuralEngine;
	//config.computeUnits = MLComputeUnitsAll;

Notice that Core ML can actually be configured to use the GPU instead of the ANE.

[LVCSRer]

Wow! Thanks for the answer!

[LVCSRer]

https://developer.apple.com/documentation/coreml/mlcomputeunits?changes=_8&language=objc