[NVEncC 9.30] onnxruntime issues

[rainman74]

cmdline:

NVEncC64.exe -i "Test.mkv" -o "Test_realesrgan_x4.mkv" --avhw --vpp-onnx model="%CMDPATH:\=/%\bin\onnx_models\realesrgan\realesrgan_x4plus.onnx",provider=cuda,out_res=3840x2160 --codec hevc --preset p5 --vbr 8000 --audio-copy --sub-copy --chapter-copy --log-level debug --log logfile.txt -o "out.mkv"

Encoding is stuck:

2026-08-04 18:30:04.9918607 [W:onnxruntime:, transformer_memcpy.cc:111 onnxruntime::MemcpyTransformer::ApplyImpl] 4 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2026-08-04 18:30:05.0077373 [W:onnxruntime:, session_state.cc:1316 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2026-08-04 18:30:05.0136550 [W:onnxruntime:, session_state.cc:1318 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
NVEncC (x64) 9.30 (r4008) by rigaya, Aug  4 2026 12:13:43 (VC 1944/Win)
OS Version     Windows 11 x64 (28000) [UTF-8]
CPU            11th Gen Intel Core i9-11900K @ 3.50GHz [TB: 4.92GHz] (8C/16T)
GPU            #0: NVIDIA GeForce RTX 3080 (8704 cores, 1800 MHz)[PCIe4x16][610.62]
NVENC / CUDA   NVENC API 13.0, CUDA 13.3, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.264/AVC, 882x540, 25/1 fps
Vpp Filters    cspconv(nv12 -> yv12)
               onnx: realesrgan_x4plus.onnx  882x540 -> 3528x2160 (x4)  io=rgb  backend=cuda matrix=bt601 range=tv [NVIDIA GeForce RTX 3080] prec=f32 -> out_res 3840x2160 (lanczos4)
               cspconv(yv12 -> nv12)
Output Info    H.265/HEVC main @ Level auto
               3840x2160p 0:0 25.000fps (25/1fps)
               avwriter: hevc, ac3, subtitle#1 => matroska
Encoder Preset P5
Rate Control   VBR
Multipass      none
Bitrate        8000 kbps (Max: 24000 kbps)
Target Quality auto
QP range       I:0-51  P:0-51  B:0-51
QP Offset      cb:0  cr:0
VBV buf size   auto
Split Enc Mode auto
Tuning Info    hq
Lookahead      off
GOP length     250 frames
B frames       3 frames [ref mode: middle]
Ref frames     5 frames, MultiRef L0:auto L1:auto
AQ             off
CU max / min   32 / auto
Others         mv:auto
2026-08-04 18:30:13.4408620 [W:onnxruntime:, transformer_memcpy.cc:111 onnxruntime::MemcpyTransformer::ApplyImpl] 4 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2026-08-04 18:30:13.4557730 [W:onnxruntime:, session_state.cc:1316 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2026-08-04 18:30:13.4609146 [W:onnxruntime:, session_state.cc:1318 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2026-08-04 18:33:19.0929371 [E:onnxruntime:, sequential_executor.cc:572 onnxruntime::ExecuteKernel] Non-zero status code returned while running Conv node. Name:'node_conv2d_350' Status Message: E:\_work\1\s\onnxruntime\core\framework\bfc_arena.cc:359 onnxruntime::BFCArena::AllocateRawInternal Failed to allocate memory for requested buffer of size 2042295808

onnx: onnx: CUDAゼロコピー経路を初期化できないためホスト経路を使用します: Non-zero status code returned while running Conv node. Name:'node_conv2d_350' Status Message: E:\_work\1\s\onnxruntime\core\framework\bfc_arena.cc:359 onnxruntime::BFCArena::AllocateRawInternal Failed to allocate memory for requested buffer of size 2042295808
onnx: onnx: inference failed: E:\_work\1\s\onnxruntime\core\providers\cuda\cuda_call.cc:129 onnxruntime::CudaCall E:\_work\1\s\onnxruntime\core\providers\cuda\cuda_call.cc:121 onnxruntime::CudaCall CUDA failure 400: invalid resource handle ; GPU=0 ; hostname=DESKTOP ; file=E:\_work\1\s\onnxruntime\core\providers\cuda\cuda_stream_handle.cc ; line=41 ; expr=cudaEventRecord(event_, static_cast<cudaStream_t>(GetStream().GetHandle()));
onnx: .
CUDA: Error while running filter "onnx".
Break in task CUDA: unknown error..
NVDEC: in 0, out 1, pending -1, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVDEC: state 1, decoderQueue 20, dataFlagQueue 26, hdr10plusQueue 0.
AUDIO: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CHECKPTS: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: frame release: 1.
NVENC: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVENC: encodeBuffer used=0, free=22.
NVDEC: in 0, out 1, pending -1, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVDEC: state 1, decoderQueue 20, dataFlagQueue 26, hdr10plusQueue 0.
AUDIO: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CHECKPTS: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: frame release: 1.
NVENC: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVENC: encodeBuffer used=0, free=22.

avout: File header not written, unexpected error! **(~100 times!!!)**

avout: failed to write header for output file: Invalid data found when processing input

encoded 0 frames, 0.00 fps, 0.00 kbps, 0.00 MB
encode time 0:03:10, CPULoad: 9.1%

[rainman74]

C:>where nvencc64*.exe
D:\Apps\Commands\bin\nvencc64.exe

C:>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Nov__7_19:25:04_Pacific_Standard_Time_2025
Cuda compilation tools, release 13.1, V13.1.80
Build cuda_13.1.r13.1/compiler.36836380_0

C:>where cudart64_*.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\bin\x64\cudart64_13.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin\cudart64_12.dll
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common\cudart64_65.dll
D:\Apps\Commands\bin\cudart64_110.dll
D:\Apps\Commands\bin\cudart64_12.dll

C:>where cudnn_*.dll
C:\Program Files\NVIDIA\CUDNN\v8.8\bin\cudnn_adv_infer64_8.dll
C:\Program Files\NVIDIA\CUDNN\v8.8\bin\cudnn_adv_train64_8.dll
C:\Program Files\NVIDIA\CUDNN\v8.8\bin\cudnn_cnn_infer64_8.dll
C:\Program Files\NVIDIA\CUDNN\v8.8\bin\cudnn_cnn_train64_8.dll
C:\Program Files\NVIDIA\CUDNN\v8.8\bin\cudnn_ops_infer64_8.dll
C:\Program Files\NVIDIA\CUDNN\v8.8\bin\cudnn_ops_train64_8.dll
D:\Apps\Commands\bin\cudnn_adv64_9.dll
D:\Apps\Commands\bin\cudnn_cnn64_9.dll
D:\Apps\Commands\bin\cudnn_engines_precompiled64_9.dll
D:\Apps\Commands\bin\cudnn_engines_runtime_compiled64_9.dll
D:\Apps\Commands\bin\cudnn_engines_tensor_ir64_9.dll
D:\Apps\Commands\bin\cudnn_ext64_9.dll
D:\Apps\Commands\bin\cudnn_graph64_9.dll
D:\Apps\Commands\bin\cudnn_heuristic64_9.dll
D:\Apps\Commands\bin\cudnn_ops64_9.dll

C:>where onnx*.dll
C:\Windows\System32\onnxruntime.dll
D:\Apps\Commands\bin\onnxruntime.dll
D:\Apps\Commands\bin\onnxruntime_providers_cuda.dll
D:\Apps\Commands\bin\onnxruntime_providers_shared.dll
D:\Apps\Commands\bin\onnxruntime_providers_tensorrt.dll

C:>set CUDA
CUDA_MODULE_LOADING=LAZY
CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1
CUDA_PATH_V12_9=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9
CUDA_PATH_V13_1=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1

[rigaya]

This seems to be caused by memory allocation failure, NVEnc 9.31 will have reduced memory usage in this case, please have a try.

[rainman74]

This seems to be caused by memory allocation failure, NVEnc 9.31 will have reduced memory usage in this case, please have a try.

unfortunately not:

E:\Videos\Backup>NVEncC64.exe -i "Test.mkv" -o "Test_realesrgan_x4.mkv" --avhw --vpp-onnx model="%CMDPATH:\=/%\bin\onnx_models\realesrgan\realesrgan_x4plus.onnx",provider=cuda,out_res=3840x2160 --codec hevc --preset p5 --vbr 8000 --audio-copy --sub-copy
--------------------------------------------------------------------------------
Test_realesrgan_x4.mkv
--------------------------------------------------------------------------------
2026-08-08 15:43:04.6155435 [W:onnxruntime:, transformer_memcpy.cc:111 onnxruntime::MemcpyTransformer::ApplyImpl] 4 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2026-08-08 15:43:04.6303590 [W:onnxruntime:, session_state.cc:1316 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2026-08-08 15:43:04.6349053 [W:onnxruntime:, session_state.cc:1318 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
NVEncC (x64) 9.31 (r4047) by rigaya, Aug  8 2026 12:55:12 (VC 1944/Win)
OS Version     Windows 11 x64 (28000) [UTF-8]
CPU            11th Gen Intel Core i9-11900K @ 3.50GHz [TB: 5.14GHz] (8C/16T)
GPU            #0: NVIDIA GeForce RTX 3080 (8704 cores, 1800 MHz)[PCIe4x16][610.62]
NVENC / CUDA   NVENC API 13.1, CUDA 13.3, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.264/AVC, 704x396, 30000/1001 fps
Vpp Filters    cspconv(nv12 -> yv12)
               onnx: realesrgan_x4plus.onnx  704x396 -> 2816x1584 (x4)  io=rgb  backend=cuda matrix=bt601 range=tv [NVIDIA GeForce RTX 3080] prec=f32 -> out_res 3840x2160 (lanczos4)
               cspconv(yv12 -> nv12)
Output Info    H.265/HEVC main @ Level auto
               3840x2160p 0:0 29.970fps (30000/1001fps)
               avwriter: hevc, aac, aac, subtitle#1, subtitle#2 => matroska
Encoder Preset P5
Rate Control   VBR
Multipass      none
Bitrate        8000 kbps (Max: 24000 kbps)
Target Quality auto
QP range       I:0-51  P:0-51  B:0-51
QP Offset      cb:0  cr:0
VBV buf size   auto
Split Enc Mode auto
Tuning Info    hq
Lookahead      off
GOP length     300 frames
B frames       3 frames [ref mode: middle]
Ref frames     5 frames, MultiRef L0:auto L1:auto
AQ             off
CU max / min   32 / auto
Others         mv:auto
2026-08-08 15:43:11.3104533 [W:onnxruntime:, transformer_memcpy.cc:111 onnxruntime::MemcpyTransformer::ApplyImpl] 4 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2026-08-08 15:43:11.3250084 [W:onnxruntime:, session_state.cc:1316 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2026-08-08 15:43:11.3295417 [W:onnxruntime:, session_state.cc:1318 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
onnx: onnx: path cuda-zerocopy をフィルタストリーム上で初期化しました。
onnx: onnx: CUDAゼロコピー実行に失敗したためホスト経路へフォールバックします: unknown error..
onnx: onnx: inference failed: E:\_work\1\s\onnxruntime\core\providers\cuda\cuda_call.cc:129 onnxruntime::CudaCall E:\_work\1\s\onnxruntime\core\providers\cuda\cuda_call.cc:121 onnxruntime::CudaCall CUDA failure 400: invalid resource handle ; GPU=0 ; hostname=DESKTOP-PC ; file=E:\_work\1\s\onnxruntime\core\providers\cuda\cuda_stream_handle.cc ; line=41 ; expr=cudaEventRecord(event_, static_cast<cudaStream_t>(GetStream().GetHandle()));
onnx: .
CUDA: Error while running filter "onnx".
Break in task CUDA: unknown error..
NVDEC: in 0, out 1, pending -1, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVDEC: state 1, decoderQueue 22, dataFlagQueue 27, hdr10plusQueue 0.
AUDIO: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CHECKPTS: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: frame release: 1.
NVENC: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVENC: encodeBuffer used=0, free=22.
NVDEC: in 0, out 1, pending -1, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVDEC: state 1, decoderQueue 22, dataFlagQueue 27, hdr10plusQueue 0.
AUDIO: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CHECKPTS: in 1, out 1, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
CUDA: frame release: 1.
NVENC: in 0, out 0, pending 0, outQeueue size: 0, queue first(ts=-9223372036854775808,dur=0,id=-1), last(ts=-9223372036854775808,dur=0,id=-1).
NVENC: encodeBuffer used=0, free=22.

[rigaya]

Thank you for the tests and the logs, NVEnc 9.32 adds more fixes targeting the path cuda-zerocopy problem.

[rainman74]

Unfortunately, no change. I've now tested the same video again, and another one as well. The console froze completely during the second video test. Both log files are attached.

logfile.txt
logfile_freeze.txt

[rigaya]

Unfortunately, I'm not being able to reproduce this issue on my side, and have no more ideas for fix.

Might be problem of VRAM size or GPU gen, but not sure.