VMAF Failing on 4K Content

[gadkarisid]

I'm getting the following error when trying to use --vmaf with a 4K source file: nvencc: ../src/feature/cuda/integer_vif_cuda.c:107: init_fex_cuda: Assertion 0' failed.`

This error has been happening since version 9.25, and is still present in 9.27. The error is present before any frames get processed and the encoding job will not progress at all. Here's the full log from the latest failure:

[2026-07-27 09:54:04] [INFO] Full encode command: /usr/bin/nvencc --avhw -c hevc --preset p7 --tune hq --multipass 2pass-full --dynamic-rc --qp-min 19 --qp-max 23 --vbr-quality 0 --gop-len 360 --nonrefp --lookahead 25 --lookahead-level 2 --output-depth 10 --bframes 4 --bref-mode middle --multiref-l0 4 --multiref-l1 2 --aq --aq-strength 2 --aq-temporal --mv-precision q-pel --ssim --psnr --vmaf model=vmaf_v0.6.1,subsample=4 --pic-struct --chapter-copy --disable-nvml 2 --output-thread 1 --audio-copy --colorrange limited --max-cll copy --master-display copy --dhdr10-info copy --transfer auto --colorprim auto --colormatrix auto --atc-sei auto --chromaloc auto --repeat-headers --aud --metadata ENCODER_LIB="NVEnc (x64) 9.27 (r3981) by rigaya, Jul 27 2026 12:01:31 (gcc 9.4.0/Linux)" --video-metadata ENCODER_PARAMS="--preset p7 --tune hq --multipass 2pass-full --dynamic-rc --qp-min 19 --qp-max 23 --aq --aq-strength 2 --aq-temporal --mv-precision q-pel --lookahead 25 --lookahead-level 2 --bframes 4 --bref-mode middle --multiref-l0 4 --multiref-l1 2" -i "/data/videoProcessing/pre/video/S03E06.mkv" -o "/data/videoProcessing/post/video/S03E06.mkv"
--------------------------------------------------------------------------------
/data/videoProcessing/post/video/S03E06.mkv
--------------------------------------------------------------------------------
NVEnc (x64) 9.27 (r3981) by rigaya, Jul 27 2026 12:01:31 (gcc 9.4.0/Linux)
OS Version     Ubuntu 24.04.4 LTS (7.0.0-28-generic)
CPU            AMD Ryzen 7 2700 Eight-Core Processor (8C/16T)
GPU            #0: NVIDIA GeForce RTX 4060 (3072 cores, 2475 MHz)[2147483.64]
NVENC / CUDA   NVENC API 13.0, CUDA 13.0, schedule mode: auto
Input Buffers  CUDA, 38 frames
Input Info     avcuvid: H.265/HEVC, 3840x1920, 24000/1001 fps
Vpp Filters    copyDtoD
               ssim psnr vmaf vmaf_v0.6.1, threads 0, subsample 4, cuda(yv12(10bit))
Output Info    H.265/HEVC main10 @ Level auto
               3840x1920p 1:1 23.976fps (24000/1001fps)
               avwriter: hevc, ac3 => matroska
Encoder Preset quality
Rate Control   VBR
Multipass      2pass-full
Bitrate        0 kbps (Max: 24000 kbps)
Target Quality auto
QP range       I:19-23  P:19-23  B:19-23
QP Offset      cb:0  cr:0
VBV buf size   auto
Split Enc Mode auto
Tuning Info    hq
Lookahead      on, 25 frames, Level 2, Adaptive I, B Insert
GOP length     360 frames
B frames       4 frames [ref mode: middle]
Ref frames     5 frames, MultiRef L0:4 L1:2
AQ             on (spatial, temporal, strength 2)
CU max / min   32 / auto
VUI            matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,range:limited,chromaloc:left
MasteringDisp  G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(1000.000000 0.000100)
MaxCLL/MaxFALL 765/235
Dynamic HDR10  copy
Others         mv:Q-pel nonrefp aud pic-struct buf-period repeat-headers 
nvencc: ../src/feature/cuda/integer_vif_cuda.c:107: init_fex_cuda: Assertion `0' failed.
[2026-07-27 09:54:20] [ERROR] Encoding failed for S03E06.mkv (exit code: 134)

[rigaya]

I tried to reproduce the issue using the NVEnc 9.27 release binary with 4K 10-bit input and CUDA VMAF, but it completed successfully in my environment.

Could you please rerun the same command with stdout unbuffered and provide the complete output?

stdbuf -o0 /usr/bin/nvencc 2>&1 | tee nvencc-vmaf.log

There should be a code: ...; description: ... line immediately before the assertion. That CUDA error code may help identify the cause.

Please also include the output of

nvidia-smi
env | grep '^CUDA_'

[gadkarisid]

I tried to reproduce the issue using the NVEnc 9.27 release binary with 4K 10-bit input and CUDA VMAF, but it completed successfully in my environment.

Could you please rerun the same command with stdout unbuffered and provide the complete output?

stdbuf -o0 /usr/bin/nvencc 2>&1 | tee nvencc-vmaf.log

There should be a code: ...; description: ... line immediately before the assertion. That CUDA error code may help identify the cause.

Please also include the output of

nvidia-smi
env | grep '^CUDA_'

I've attached a file that captured an additional error code during the encoding attempt, and also have the nvidia-smi details you requested. Thanks!

nvencc-vmaf.log

[rigaya]

Thank you for the details, helped me a lot.

I've found out that it was a problem on the libvmaf side. I've added fix for libvmaf CUDA code, and then rebuilt libvmaf in NVEnc 9.28. I hope this fixes the problem.

[gadkarisid]

Thank you for the details, helped me a lot.

I've found out that it was a problem on the libvmaf side. I've added fix for libvmaf CUDA code, and then rebuilt libvmaf in NVEnc 9.28. I hope this fixes the problem.

VMAF on 4K content is working properly in release 9.28. Thanks for the quick fix!