C++ inference with DirectML:How to use DirectML in C++ library?

[CHNVigny]

Is your feature request related to a problem? Please describe.
I have to compile my C++ code under x86 mode, I want to accelerate my code by using DirectML but not found DirectML interface in header "onnxruntime_cxx_api.h". How to use DirectML to accelerate inference with C++ API?

System information

• ONNX Runtime version :1.11.0(x86)

Describe the solution you'd like
I want to call DirectML interface in C++ library but not found how to append DML execution provider.

Additional context
give me a sample please, THANKS!!!!!

[deischi]

Take a look at OrtSessionOptionsAppendExecutionProvider_DML or SessionOptionsAppendExecutionProvider_DML in dml_provider_factory.h.

see also https://github.com/microsoft/onnxruntime-openenclave/blob/openenclave-public/docs/execution_providers/DirectML-ExecutionProvider.md

[BengtGustafsson]

I have a C++ based implementation that I want to run with DML. I call OrtSessionOptionsAppendExecutionProvider_DML with my Ort::SessionOptions object like this, but later, in the actual inference, it just crashed with a Microsoft com_error exception. I don't know if this is connected to me using C++ API apart from this snippet, but I can see that your demo programs for DML are all C through and through, and these programs work. Our program works fine with CPU inference by commenting this part out:

    const OrtDmlApi* ort_dml_api{ nullptr };
    CheckStatus(g_ort->GetExecutionProviderApi("DML", ORT_API_VERSION, reinterpret_cast<const void**>(&ort_dml_api)));
    m_sessionOptions.DisableMemPattern();
    m_sessionOptions.SetExecutionMode(ExecutionMode::ORT_SEQUENTIAL);

    CheckStatus(ort_dml_api->SessionOptionsAppendExecutionProvider_DML(m_sessionOptions, deviceId));
    
  So the question is if this is supposed to work or if it is impossible to use C++ at all with the DirectML provider.

[deischi]

I am using GPU/DirectML inferencing form C++ al the time. Typically that is working fine.

Some things come to my mind immediately:

• Did you call CoInititalizeEx
• DirectML internally throws some _com_error exceptions - but they are handled. If you are using Visual Studio - are you Exception Settings too strict?
• Catch the com_error. What is the error message?

[BengtGustafsson]

Yes, my bad. I was first using the dirctml.dll distributed to my Windows 11 but this does not seem to be updated wirth Windows update at all, and was too old (1.6). When I use 1.10 it works, I just didn't skip through all those exceptions in VS.

The question remains: Why isn't there C++ methods in SessionOptions for DML as for the other providers?

I also wanted to ask why the onnxruntime.dll is not the same irrespective of which providers were enabled when it was built. As it stands there is no pre-built binary which we can use if we want to providse both CUDA and DML possibilities. To me it seems that the onnxruntime.dll should be the same but I can see that it gets 1 MB larger when you enable DML, which is logical as DML interface is not a separate provider dll (which would have been logical). So I tried using the DML-enabled onnxruntime.dll with the cuda provider DLL from another pre-built binary but that didn't work. I don't get link errors so the AppendExecutionProvider_CUDA method is there, but if I call it it doesn't work.

[ksasso1028]

Hey @BengtGustafsson , take a look at the perf tests for all execution providers, including dml. if you search for dml here youll see the headers you need to include along with how to pass the session options. ->

onnxruntime/onnxruntime/test/perftest/ort_test_session.cc

Line 564 in edffa2a

} else if (provider_name_ == onnxruntime::kDmlExecutionProvider) {

After setting the desired options you just do
session_options.AppendExecutionProvider("DML", dml_options);

This file will also outline any other execution providers you may be curious about in C++

best,
Kevin