ONNX Runtime Thread Safety

[genbattle]

I'm looking at using ONNX from Rust using some community-developed bindings. One of the things that Rust offers is compile-time type checking around thread safety. The author of these bindings has been extremely conservative in how they've exposed ONNX to Rust with respect to thread safety; Environment and Session types have not been marked as being able to be shared or even sent between threads.

In practice this makes using ONNX fairly restrictive in Rust, but I wasn't sure how the original author of the bindings had arrived at this position; whether it was a pre-emptive decision to try and head off any concurrency issues, or possibly they just used the default provided (Rust will assume C pointers are not thread safe in any respect by default and have to explicitly be marked otherwise). My hope is that with more information I can modify these bindings to be less restrictive and make working with ONNX a lot easier.

So my question is whether there's any definitive documentation around thread safety in ONNX? Can Environment or Sessions be accessed from multiple threads sequentially without issues? Which operations aren't safe to perform concurrently?

I've seen this mentioned explicitly a few times:

Multiple threads can invoke the Run() method on the same inference session object. See API doc for more details.

And the other methods of Session seem to be read-only, so my intuition tells me that Sessions are safe to send between and execute from multiple threads concurrently.

[Craigacp]

In the Java API (which uses the C API) we have a singleton environment shared between all threads, and sessions themselves are thread safe after construction. We test the thread safety of Session.Run in our unit tests - https://github.com/microsoft/onnxruntime/blob/master/java/src/test/java/ai/onnxruntime/InferenceTest.java#L614. I'm not sure how this interacts with some of the more complicated bits of the API around custom allocators, but we haven't exposed those in Java yet.

[kashev]

FYI for folks finding this a few years later - the line number for the referenced test has changed, but the multi-threading test is still here: https://github.com/microsoft/onnxruntime/blob/main/java/src/test/java/ai/onnxruntime/InferenceTest.java#L533