OpenVino EP high memory usage

[simonesneault]

Hello

We are trying to use openvino EP here to perform inferences on a unet3D model trained with pytorch, then converted to onnx

The inference time becomes much better when we use the openvino backend, but the memory usage becomes huge

Here is a comparison of the timing and memory consumption we have between for a tensor of size 1 1 240 240 240

onnx 1.21 CPU time : 2.7s
onnx 1.21 CPU memory : 371 MB
onnx 1.21 OpenVino 2024.6 EP time : 0.7s
onnx 1.21 OpenVIno 2024.6 EP memory : 1942 MB
libtorch 2.2.2 time : 2.1s
libtorch 2.2.2 memory : 1280 MB

// prepare the session and everything
Ort::Env l_env( ORT_LOGGING_LEVEL_WARNING, "Onnx" );
Ort::SessionOptions l_session_options;
l_session_options.DisableProfiling();
l_session_options.DisableCpuMemArena();
l_session_options.DisableMemPattern();

// enable some backend if asked to
if( m_enable_openvino ) {
    l_session_options.SetGraphOptimizationLevel( ORT_DISABLE_ALL );
    l_session_options.AppendExecutionProvider_OpenVINO_V2( { { "device_type", "CPU" } } );
}

Ort::Session l_session( l_env, l_model_content.data(), l_model_content.size(), l_session_options );

// prepare the tensor
std::vector< int64_t > l_input_dims = { 1, 1, 240, 240, 240 };
Ort::MemoryInfo l_memory_info = Ort::MemoryInfo::CreateCpu( OrtArenaAllocator, OrtMemTypeCPU );
Ort::Value l_input_tensor( Ort::Value::CreateTensor< float >( l_memory_info, l_data_ptr,lnb_element, l_input_dims.data(), l_input_dims.size() ) );

// Run the inference
std::vector< Ort::Value > l_output_tensors = l_session.Run( Ort::RunOptions{ nullptr }, std::vector< const char* >( { "input" } ).data(), &l_input_tensor, 1, std::vector< const char* >( { "output" } ).data(), 1 );

Do you know how to reduce the memory usage at runtime ? We've noticed that the line l_session_options.DisableCpuMemArena(); helped a lot for the classical CPU backend but it has no effect for openvino ...
Thanks