[Mobile] Issue with Importing YOLOv8 Pose Model into Unity using Sentis

[aakash94]

Describe the issue

I followed the pose estimation tutorial to obtain the yolov8n-pose.with_pre_post_processing.onnx model. The model passes validation using ONNX's checker and is confirmed to be valid.

However, when importing the model into Unity, I encounter the following error from Unity's Sentis:

Asset import failed, "Assets/Models/pose_pre_post.onnx" > KeyNotFoundException: The given key '_ppp0_bgr_data' was not present in the dictionary.

Upon inspecting the model in Netron, I discovered that _ppp0_bgr_data is located between the pre_process_0 and pre_process_1 nodes.

Do note that the model that is generated without the pre and post processing steps baked in works just fine.

How can I resolve this issue? If this is not the appropriate place to seek a solution, could you please direct me to the correct platform or resource where I can get assistance?

Thank you.

To reproduce

Follow the pose estimation tutorial to obtain the yolov8n-pose.with_pre_post_processing.onnx model.

Drag and drop the model into Unity(6, with Sentis 1.5) as an asset to observe an error in importing the model.

Urgency

No response

Platform

Other / Unknown

OS Version

Sentis 1.5, Unity 6,

ONNX Runtime Installation

Released Package

Compiler Version (if 'Built from Source')

No response

Package Name (if 'Released Package')

None

ONNX Runtime Version or Commit ID

onnxruntime-1.18.1-cp310

ONNX Runtime API

C++/C

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[skottmckay]

This seems like an issue for the Unity/Sentis folks.

_ppp0_bgr_data is an internal value of the model. I don't know what dictionary they're checking values against or why that is not in it, but given it's an internal value a user would never directly touch it.

I do note this: https://discussions.unity.com/t/error-importing-onnx-model/370378

When the input is an image we use a custom operator called DecodeImage to convert it from jpg/png to RGB. That custom operator is implemented in the onnxruntime_extensions library as you would have seen in the tutorial. Maybe Unity/Sentis doesn't handle custom operators or requires some extra configuration to do so.

When the input is RGB we do not need to use the custom operator.