Tried to specify the thread pool when creating an OrtEnvironment, but one already exists

[brucewkz]

Describe the issue

I built an embedding serving web app using Spring AI + OnnxRuntime, the app is started normally，but when I request the http://127.0.0.1/embedding I get the following error：

(base) ubuntu@ubuntu:~/onnxruntimeDemo$ java -jar onnxruntimeDemo-1.0-SNAPSHOT.jar

. ____ _ __ _ _
/\ / ' __ _ () __ __ _ \ \ \
( ( )__ | '_ | '| | ' / ` | \ \ \
\/ )| |)| | | | | || (| | ) ) ) )
' || .__|| ||| |_, | / / / /
=========||==============|/=////

:: Spring Boot :: (v3.3.1)

2024-07-09T03:56:16.098Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] c.h.c.d.OnnxRuntimeApplication : Starting OnnxRuntimeApplication using Java 17.0.9 with PID 1003336 (/home/ubuntu/onnxruntimeDemo/onnxruntimeDemo-1.0-SNAPSHOT.jar started by ubuntu in /home/ubuntu/onnxruntimeDemo)
2024-07-09T03:56:16.104Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] c.h.c.d.OnnxRuntimeApplication : No active profile set, falling back to 1 default profile: "default"
2024-07-09T03:56:16.819Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat initialized with port 8080 (http)
2024-07-09T03:56:16.830Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.a.c.c.StandardService : Starting service [Tomcat]
2024-07-09T03:56:16.830Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.25]
2024-07-09T03:56:16.859Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.a.c.c.C.[.[.[/] : Initializing Spring embedded WebApplicationContext
2024-07-09T03:56:16.861Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 727 ms
2024-07-09T03:56:21.022Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] a.d.u.Platform : Found matching platform from: jar:file:/home/ubuntu/onnxruntimeDemo/lib/tokenizers-0.26.0.jar!/native/lib/tokenizers.properties
2024-07-09T03:56:22.829Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model input names: input_ids, attention_mask
2024-07-09T03:56:22.831Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model output names: token_embeddings, sentence_embedding
2024-07-09T03:56:24.361Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model input names: input_ids, attention_mask
2024-07-09T03:56:24.361Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.a.t.TransformersEmbeddingClient : Model output names: token_embeddings, sentence_embedding
2024-07-09T03:56:24.656Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat started on port 8080 (http) with context path '/'
2024-07-09T03:56:24.666Z INFO 1003336 --- [OnnxRuntimeApplication] [ main] c.h.c.d.OnnxRuntimeApplication : Started OnnxRuntimeApplication in 8.922 seconds (process running for 9.46)
2024-07-09T03:56:45.723Z INFO 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.a.c.c.C.[.[.[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-07-09T03:56:45.724Z INFO 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.s.w.s.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2024-07-09T03:56:45.727Z INFO 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.s.w.s.DispatcherServlet : Completed initialization in 3 ms
2024-07-09T03:56:45.865Z ERROR 1003336 --- [OnnxRuntimeApplication] [nio-8080-exec-1] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.] with root cause

java.lang.IllegalStateException: Tried to specify the thread pool when creating an OrtEnvironment, but one already exists.
at ai.onnxruntime.OrtEnvironment.getEnvironment(OrtEnvironment.java:136) ~[onnxruntime_gpu-1.14.0.jar:1.14.0]
at ai.djl.onnxruntime.engine.OrtEngine.(OrtEngine.java:56) ~[onnxruntime-engine-0.26.0.jar:?]
at ai.djl.onnxruntime.engine.OrtEngine.newInstance(OrtEngine.java:64) ~[onnxruntime-engine-0.26.0.jar:?]
at ai.djl.onnxruntime.engine.OrtEngineProvider.getEngine(OrtEngineProvider.java:43) ~[onnxruntime-engine-0.26.0.jar:?]
at ai.djl.engine.Engine.getEngine(Engine.java:216) ~[api-0.26.0.jar:?]
at ai.djl.engine.Engine.getInstance(Engine.java:149) ~[api-0.26.0.jar:?]
at ai.djl.ndarray.NDManager.newBaseManager(NDManager.java:120) ~[api-0.26.0.jar:?]
at org.springframework.ai.transformers.TransformersEmbeddingClient.call(TransformersEmbeddingClient.java:280) ~[spring-ai-transformers-1.0.0.jar:1.0.0]
at org.springframework.ai.transformers.TransformersEmbeddingClient.embed(TransformersEmbeddingClient.java:232) ~[spring-ai-transformers-1.0.0.jar:1.0.0]

To reproduce

@configuration
public class TransformerConf {

@Bean("transformersEmbeddingClient")
public TransformersEmbeddingClient embeddingClient() throws Exception {
    TransformersEmbeddingClient embeddingClient = new TransformersEmbeddingClient();
    embeddingClient.setTokenizerResource("classpath:bin/tokenizer.json");
    embeddingClient.setModelResource("classpath:bin/model.onnx");
    embeddingClient.setModelOutputName("token_embeddings");
    embeddingClient.setDisableCaching(false);
    embeddingClient.afterPropertiesSet();
    return embeddingClient;
}

}

@GetMapping("/embedding")
public List<List> getResult() {
return embeddings = embeddingClient.embed(List.of("xxxxxx"));
}

Urgency

No response

Platform

Linux

OS Version

Ubuntu 9.4.0-1ubuntu1~20.04.2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

0.26.0

ONNX Runtime API

Java

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[Craigacp]

I think this is a bug in the interaction of Spring AI and DJL as both of them shouldn't initialize the ORT environment. It's initialized here in Spring AI (https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-transformers/src/main/java/org/springframework/ai/transformers/TransformersEmbeddingModel.java#L186) and then re-inited with a thread pool in DJL (https://github.com/deepjavalibrary/djl/blob/master/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtEngine.java#L61). The error comes out of ORT as we can't recreate the environment with a different thread pool after it's been initially created due to constraints in the ORT native library.

[brucewkz]

I think this is a bug in the interaction of Spring AI and DJL as both of them shouldn't initialize the ORT environment. It's initialized here in Spring AI (https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-transformers/src/main/java/org/springframework/ai/transformers/TransformersEmbeddingModel.java#L186) and then re-inited with a thread pool in DJL (https://github.com/deepjavalibrary/djl/blob/master/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtEngine.java#L61). The error comes out of ORT as we can't recreate the environment with a different thread pool after it's been initially created due to constraints in the ORT native library.

Yes, you're right. I also know that it was initialized twice. Now it's unavoidable when the app starts，Do you have any good idea? @snnn Can you provide some help？thx.

[Craigacp]

This is documented behaviour for ORT, our options are either to return an incorrect object with a warning, or throw. In the case where the user wanted a different log name we warn, but incorrect threading is a problem which could break the application so we throw. It's been like that for 3 years - https://github.com/microsoft/onnxruntime/blame/main/java/src/main/java/ai/onnxruntime/OrtEnvironment.java#L119.

Spring AI should check to see if there is an environment created in DJL and use that, given they use DJL for the post processing (though that's not necessary and will involve converting the ONNX output into a Java object and then into a Pytorch tensor before performing the computation - https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-transformers/src/main/java/org/springframework/ai/transformers/TransformersEmbeddingModel.java#L327). I've got an example of doing this in plain Java & ONNX from 3 years ago in Tribuo, and we have a better example with masking & padding support internally but haven't got around to releasing it yet.

[brucewkz]

@Craigacp Thank you so much. I have submitted an issue: spring-projects/spring-ai#1020