Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Web] Upgrading from 1.20.1 to 1.21.* breaks Segment Anything models on WebGPU #23183

Open
xenova opened this issue Dec 23, 2024 · 8 comments · May be fixed by #23311
Open

[Web] Upgrading from 1.20.1 to 1.21.* breaks Segment Anything models on WebGPU #23183

xenova opened this issue Dec 23, 2024 · 8 comments · May be fixed by #23311
Labels
ep:WebGPU ort-web webgpu provider model:transformer issues related to a transformer model: BERT, GPT2, Hugging Face, Longformer, T5, etc. .NET Pull requests that update .net code platform:web issues related to ONNX Runtime web; typically submitted using template

Comments

@xenova
Copy link

xenova commented Dec 23, 2024

Describe the issue

After upgrading to onnxruntime-web version 1.21.0-dev.20241205-d27fecd3d3 in this commit, Segment Anything models no longer work on WebGPU. WASM is correct (albeit slow). You can use https://huggingface.co/Xenova/slimsam-77-uniform for testing, as shown below.

Correct behaviour:
Image

Incorrect behaviour:
Image

To reproduce

  1. Check out the demo source code: https://github.com/huggingface/transformers.js-examples/blob/main/segment-anything-webgpu/index.js
  2. Update the import to https://cdn.jsdelivr.net/npm/@huggingface/[email protected] and see that everything works correctly.
  3. Update the import to https://cdn.jsdelivr.net/npm/@huggingface/[email protected] and see that the output is incorrect.

Urgency

Since Segment Anything is quite a popular Transformers.js model, this is pretty important.

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.21.0-dev.20241205-d27fecd3d3

Execution Provider

'webgpu' (WebGPU)
@xenova xenova added the platform:web issues related to ONNX Runtime web; typically submitted using template label Dec 23, 2024
@xenova
Copy link
Author

xenova commented Dec 23, 2024

cc @fs-eire @guschmue

@github-actions github-actions bot added .NET Pull requests that update .net code ep:WebGPU ort-web webgpu provider model:transformer issues related to a transformer model: BERT, GPT2, Hugging Face, Longformer, T5, etc. labels Dec 23, 2024
@fs-eire
Copy link
Contributor

fs-eire commented Dec 30, 2024 •
edited
Loading

After some investigation, I found the root cause of the issue is #22755. If I revert the change and rebuild, the SAM demo will work.

There should be some problem of the ScatterND WebGPU implementation. Need further investigation.

@xenova
Copy link
Author

xenova commented Dec 31, 2024

Thanks for the update! Please let me know if you put out a build with the change reverted (or whether you can put out a build that has it fixed).

@xenova xenova mentioned this issue Jan 4, 2025
Open
5 tasks
@prathikr
Copy link
Contributor

prathikr commented Jan 6, 2025

@xenova will look into this in the next day or two. If I cannot fix by then I will revert the PR to mitigate the issue while continuing to investigate. I appreciate your patience.

@xenova
Copy link
Author

xenova commented Jan 7, 2025 •
edited
Loading

Thanks @prathikr! If you could publish a dev build with the PR reverted, that could work too! Hopefully it can also include #23026 too.

@xenova
Copy link
Author

xenova commented Jan 7, 2025

Also, if it helps debug, it appears to be an issue w/ fp16 or q8 (not fp32): huggingface/transformers.js#1134 (comment).

@prathikr prathikr mentioned this issue Jan 7, 2025
Merged
prathikr added a commit that referenced this issue Jan 8, 2025
@prathikr
disable scatternd op for jsep (#23277)
d8e8d4f 
mitigates #23183 while we
investigate final solution
@fs-eire
Copy link
Contributor

fs-eire commented Jan 8, 2025

@xenova please try the latest dev package: [email protected]

snnn pushed a commit that referenced this issue Jan 8, 2025
@prathikr @snnn
disable scatternd op for jsep (#23277)
03afc08 
mitigates #23183 while we
investigate final solution
@xenova
Copy link
Author

xenova commented Jan 9, 2025

Thanks @fs-eire - will try!

@prathikr prathikr linked a pull request Jan 9, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ep:WebGPU ort-web webgpu provider model:transformer issues related to a transformer model: BERT, GPT2, Hugging Face, Longformer, T5, etc. .NET Pull requests that update .net code platform:web issues related to ONNX Runtime web; typically submitted using template
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fixes bug and enables scatternd for jsep microsoft/onnxruntime
3 participants
@fs-eire @xenova @prathikr