Mistral 7B DirectML model fails on 2nd turn of conversation in Playground

[SteveJSteiner]

***Note an earlier bug was logged on this but closed because the laptop was not plugged in. This laptop was on wall power the entire time.

************>systeminfo
OS Name: Microsoft Windows 11 Enterprise
OS Version: 10.0.22621 N/A Build 22621
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: N/A
Registered Organization: N/A
Product ID: 00330-80000-00000-AA305
Original Install Date: 2/2/2024, 10:08:49 AM
System Boot Time: 12/10/2024, 1:39:27 PM
System Manufacturer: Microsoft Corporation
System Model: Surface Book 3
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 126 Stepping 5 GenuineIntel ~1298 Mhz
BIOS Version: Microsoft Corporation 23.101.140, 10/10/2024

Total Physical Memory: 32,354 MB
Available Physical Memory: 14,984 MB
Virtual Memory: Max Size: 37,218 MB
Virtual Memory: Available: 9,386 MB
Virtual Memory: In Use: 27,832 MB

***************>wmic path win32_videocontroller get name
Name
Intel(R) Iris(R) Plus Graphics
NVIDIA GeForce GTX 1660 Ti with Max-Q Design

-----Repro instructions-------------

1. Download the Mistral 7B DirectML model.
2. Got to Playground tools
3. Change inference parameter settings (* this might not be necessary): Inference Parameter Settings: Max response length (tokens) 1024, temperature 0.7
4. Prompt with: Please give three import ideas from Deleuze.
5. After reply prompt with: Please give 3 more.
   Expected: a response
   Actual: no text response
6. Prompt a 3rd time with: ?
   This appears to have produced the exception below, however the failure happened before at the 2nd prompt.

-- Note -- The ui indicates this produced 414 out of 32,768 tokens.

---Contents of the Ai Toolkit 'Output' window: ---
[2024-12-17T18:31:11.621Z] [INFO] Command registration.
[2024-12-17T18:31:30.267Z] [INFO] telemetry event:activate_extension sent
Connected to agent:Inference.Service.Agent pipe after retries:0
Finished agent startup...
Agent unlocked
Finished agent startup...
Agent unlocked
Information: Microsoft.Neutron.Rpc.Service.JsonRpcService [2306] 2024-12-17T10:31:31.2378579-08:00 Accepting pipe incoming pipeName:ai.21a1427d2260ea465c071c8ed8ae00e2 numOfSession:2
[2024-12-17T18:31:44.320Z] [INFO] Loading View: catalogModels
[2024-12-17T18:31:48.668Z] [INFO] Loading View: modelPlayground
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [1400] 2024-12-17T10:31:52.2456226-08:00 Loading model:mistral-7b-v02-int4-directml
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [1400] 2024-12-17T10:31:52.2456226-08:00 Loading model:mistral-7b-v02-int4-directml
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [1401] 2024-12-17T10:31:59.3162666-08:00 Finish loading model:mistral-7b-v02-int4-directml elapsed time:00:00:07.0706275
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [1401] 2024-12-17T10:31:59.3162666-08:00 Finish loading model:mistral-7b-v02-int4-directml elapsed time:00:00:07.0706275
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:32:07.3931626-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:32:07.3931626-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:32:42.981434-08:00 HandleChatCompletionAsStreamRequest elapsed time:35527 total tokens:364
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:32:42.981434-08:00 HandleChatCompletionAsStreamRequest elapsed time:35527 total tokens:364
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:32:42.9900457-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:32:42.9900457-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:06.3152454-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:06.3152454-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:09.8104685-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:09.8104685-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:17.4663293-08:00 HandleChatCompletionAsStreamRequest elapsed time:7602 total tokens:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:17.4663293-08:00 HandleChatCompletionAsStreamRequest elapsed time:7602 total tokens:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:17.4738517-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:33:17.4738517-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:13.9044882-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:13.9044882-08:00 GetChatCompletionRequestTokenAsync -> model:mistral-7b-v02-int4-directml maxTokens:(null) temperature:(null) topP:(null)
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:14.563166-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:14.563166-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:14.8806593-08:00 HandleChatCompletionAsStreamRequest elapsed time:284 total tokens:0
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:14.8806593-08:00 HandleChatCompletionAsStreamRequest elapsed time:284 total tokens:0
Error: Microsoft.AspNetCore.Server.Kestrel [13] 2024-12-17T10:35:14.8866376-08:00 Connection id "0HN8UP44R2UF4", Request id "0HN8UP44R2UF4:00000001": An unhandled exception was thrown by the application. error: [Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(1066)\onnxruntime.dll!00007FFA29772821: (caller: 00007FFA29805159) Exception(2) tid(59cc) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.
, at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x54
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.OnnxChatGenerator.GenerateNextTokenAsync(CancellationToken) + 0x21
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<>c__DisplayClass36_0.<<HandleChatCompletionAsStreamRequestAsync>g__DecodeTokens|0>d.MoveNext() + 0xa8 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.TokenPerfExtensions.<WithTokenPerfAsync>d__02.MoveNext() + 0x699
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at Microsoft.Neutron.OpenAI.TokenPerfExtensions.d__02.MoveNext() + 0x83e --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.TokenPerfExtensions.<WithTokenPerfAsync>d__02.MoveNext() + 0xa1f
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x659 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.d__36.MoveNext() + 0x7f3
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x9be --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x3b3 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x4f1 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F16C589DE9EC82483AA705851D2FE201CB4CB4AAF6561E8DE71B6A1891AD8D67F__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass10_0.<<MapPost7>g__RequestHandler|4>d.MoveNext() + 0x5c9 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequests>d__2381.MoveNext() + 0x4b1]
Error: Microsoft.AspNetCore.Server.Kestrel [13] 2024-12-17T10:35:14.8866376-08:00 Connection id "0HN8UP44R2UF4", Request id "0HN8UP44R2UF4:00000001": An unhandled exception was thrown by the application. error: [Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(1066)\onnxruntime.dll!00007FFA29772821: (caller: 00007FFA29805159) Exception(2) tid(59cc) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.
, at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x54
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.OnnxChatGenerator.GenerateNextTokenAsync(CancellationToken) + 0x21
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<>c__DisplayClass36_0.<<HandleChatCompletionAsStreamRequestAsync>g__DecodeTokens|0>d.MoveNext() + 0xa8 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.TokenPerfExtensions.<WithTokenPerfAsync>d__02.MoveNext() + 0x699
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at Microsoft.Neutron.OpenAI.TokenPerfExtensions.d__02.MoveNext() + 0x83e --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.TokenPerfExtensions.<WithTokenPerfAsync>d__02.MoveNext() + 0xa1f
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x659 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.d__36.MoveNext() + 0x7f3
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x9be --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x3b3 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x4f1 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F16C589DE9EC82483AA705851D2FE201CB4CB4AAF6561E8DE71B6A1891AD8D67F__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass10_0.<<MapPost7>g__RequestHandler|4>d.MoveNext() + 0x5c9 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequests>d__2381.MoveNext() + 0x4b1]
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:15.2934463-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:15.2934463-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Error: Microsoft.AspNetCore.Server.Kestrel [13] 2024-12-17T10:35:15.2972669-08:00 Connection id "0HN8UP44R2UF5", Request id "0HN8UP44R2UF5:00000001": An unhandled exception was thrown by the application. error: [D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommittedResourceAllocator.cpp(22)\onnxruntime.dll!00007FFA2976A921: (caller: 00007FFA2974B3DC) Exception(3) tid(7c40) 887A0005 The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action.
, at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x54
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.OnnxChatGenerator..ctor(OnnxLoadedModel, GeneratorParams) + 0x3f
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.d__13.MoveNext() + 0xe3
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x311 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x3b3 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x4f1 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F16C589DE9EC82483AA705851D2FE201CB4CB4AAF6561E8DE71B6A1891AD8D67F__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass10_0.<<MapPost7>g__RequestHandler|4>d.MoveNext() + 0x5c9 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequests>d__2381.MoveNext() + 0x4b1]
Error: Microsoft.AspNetCore.Server.Kestrel [13] 2024-12-17T10:35:15.2972669-08:00 Connection id "0HN8UP44R2UF5", Request id "0HN8UP44R2UF5:00000001": An unhandled exception was thrown by the application. error: [D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommittedResourceAllocator.cpp(22)\onnxruntime.dll!00007FFA2976A921: (caller: 00007FFA2974B3DC) Exception(3) tid(7c40) 887A0005 The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action.
, at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x54
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.OnnxChatGenerator..ctor(OnnxLoadedModel, GeneratorParams) + 0x3f
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.d__13.MoveNext() + 0xe3
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x311 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x3b3 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x4f1 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F16C589DE9EC82483AA705851D2FE201CB4CB4AAF6561E8DE71B6A1891AD8D67F__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass10_0.<<MapPost7>g__RequestHandler|4>d.MoveNext() + 0x5c9 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequests>d__2381.MoveNext() + 0x4b1]
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:16.070815-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
Information: Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx [0] 2024-12-17T10:35:16.070815-08:00 HandleChatCompletionAsStreamRequest -> model:mistral-7b-v02-int4-directml maxTokens:1024 temperature:0.7 topP:1
[2024-12-17T18:35:16.079Z] [ERROR] Failed to chatStream. provider = "ONNX", model = "mistral-7b-v02-int4-directml", errorMessage = "Error: Unable to call the mistral-7b-v02-int4-directml inference endpoint due to 500. Please check if the input or configuration is correct.", errorType = "c", errorObject = {"innerError":{"status":500,"headers":{"connection":"close","content-length":"0","date":"Tue, 17 Dec 2024 18:35:15 GMT","server":"Kestrel"}}}
[2024-12-17T18:35:16.080Z] [ERROR] Unable to call the mistral-7b-v02-int4-directml inference endpoint due to 500. Please check if the input or configuration is correct. 500 status code (no body)
Error: Microsoft.AspNetCore.Server.Kestrel [13] 2024-12-17T10:35:16.0771625-08:00 Connection id "0HN8UP44R2UF6", Request id "0HN8UP44R2UF6:00000001": An unhandled exception was thrown by the application. error: [D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommittedResourceAllocator.cpp(22)\onnxruntime.dll!00007FFA2976A921: (caller: 00007FFA2974B3DC) Exception(4) tid(6e08) 887A0005 The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action.
, at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x54
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.OnnxChatGenerator..ctor(OnnxLoadedModel, GeneratorParams) + 0x3f
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.d__13.MoveNext() + 0xe3
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x311 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x3b3 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x4f1 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F16C589DE9EC82483AA705851D2FE201CB4CB4AAF6561E8DE71B6A1891AD8D67F__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass10_0.<<MapPost7>g__RequestHandler|4>d.MoveNext() + 0x5c9 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequests>d__2381.MoveNext() + 0x4b1]
Error: Microsoft.AspNetCore.Server.Kestrel [13] 2024-12-17T10:35:16.0771625-08:00 Connection id "0HN8UP44R2UF6", Request id "0HN8UP44R2UF6:00000001": An unhandled exception was thrown by the application. error: [D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlCommittedResourceAllocator.cpp(22)\onnxruntime.dll!00007FFA2976A921: (caller: 00007FFA2974B3DC) Exception(4) tid(6e08) 887A0005 The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action.
, at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x54
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.OnnxChatGenerator..ctor(OnnxLoadedModel, GeneratorParams) + 0x3f
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderOnnx.d__13.MoveNext() + 0xe3
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b
at Microsoft.Neutron.OpenAI.Provider.OpenAIServiceProviderBase1.<HandleChatCompletionAsStreamRequestAsync>d__36.MoveNext() + 0x311 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.ThrowForFailedGetResult() + 0x13
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16) + 0x2c at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x3b3 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at Microsoft.Neutron.OpenAI.OpenAIServiceWebApiExtensions.<>c__DisplayClass2_0.<<HandleStreamRequest>b__0>d.MoveNext() + 0x4f1 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F16C589DE9EC82483AA705851D2FE201CB4CB4AAF6561E8DE71B6A1891AD8D67F__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass10_0.<<MapPost7>g__RequestHandler|4>d.MoveNext() + 0x5c9 --- End of stack trace from previous location --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb2 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4b at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<ProcessRequests>d__2381.MoveNext() + 0x4b1]

[microsoft-github-policy-service]

Thank you for contacting us! Any issue or feedback from you is quite important to us. We will do our best to fully respond to your issue as soon as possible. Sometimes additional investigations may be needed, we will usually get back to you within 2 days by adding comments to this issue. Please stay tuned.

[swatDong]

@jiaxuwu2021 - could you or anyone from your team please help on this onnx model issue?

[jiaxuwu2021]

Hi @SteveJSteiner what's the ai toolkit extension version are you trying? We cannot reproduce on the latest version (v0.8.2)
Could you please upgrade extension and try again if not the latest one? Thank you!

[SteveJSteiner]

It is Version 0.8.2. Note - I'm 90% sure this is due to the laptop GPU having 6 Gb of memory. On my desktop with a 24 Gb memory GPU there is no problem using the model across multiple turns.

…

-Steve

On Tue, Dec 24, 2024 at 8:07 PM jiaxuwu2021 ***@***.***> wrote: Hi @SteveJSteiner <https://github.com/SteveJSteiner> what's the ai toolkit extension version are you trying? We cannot reproduce on the latest version (v0.8.2) Could you please upgrade extension and try again if not the latest one? Thank you! — Reply to this email directly, view it on GitHub <#134 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPWN4BR7QZJ2PWCR67U25D2HIVRLBFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVI3TQMJVGYZTCOBQGSSG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44TQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKJXGI4TGMBSGU4DRAVEOR4XAZNFNFZXG5LFUV3GC3DVMWVDENZUGU3TMOJRGA4YFJDUPFYGLJLMMFRGK3FFOZQWY5LFVI3TQMJVGYZTCOBQGSTXI4TJM5TWK4VGMNZGKYLUMU> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .