[Feature] Machine Learning using Rockchip NPU on supported boards.

[yoni13]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Since immich already supports Hardware Transcoding with hardware acceleration via rkmpp, maybe we can make machine learning working on rockchip NPUs.

Links:
rknn-toolkit2
frigate's implantation
How to test

Platform

• Server
• Web
• Mobile

[yoni13]

Update: I am able to run "ViT-B-32__openai" using rknn-toolkit2 on emulator, I haven't verify the accuracy but I will say it got some progress.

[mertalev]

AnnSession uses None for the name and only provides the shape, so this can do the same (or provide None for both if the shape is also unavailable).

[yoni13]

I am out of ideas now 🫠.
I think I need to know more about insightface to solve this.

[mertalev]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

For this, just call onnx runtime to get the correct faces and unload it.

[yoni13]

(machine-learning-py3.11) root@orangepi3b:~/immich/machine-learning# ./start.sh 
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libmimalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[12/03/24 12:15:23] INFO     Starting gunicorn 23.0.0                                                                                                                                        
[12/03/24 12:15:23] INFO     Listening at: http://[::]:3003 (5620)                                                                                                                           
[12/03/24 12:15:23] INFO     Using worker: app.config.CustomUvicornWorker                                                                                                                    
[12/03/24 12:15:23] INFO     Booting worker with pid: 5621                                                                                                                                   
[12/03/24 12:15:32] I        Started server process [5621]                                                                                                                                   
[12/03/24 12:15:32] I        Waiting for application startup.                                                                                                                                
[12/03/24 12:15:32] I        Created in-memory cache with unloading after 300s of inactivity.                                                                                                
[12/03/24 12:15:32] I        Initialized request thread pool with 4 threads.                                                                                                                 
[12/03/24 12:15:32] I        Application startup complete.                                                                                                                                   
[12/03/24 12:15:34] I        Loading textual model 'ViT-B-32__openai' to memory                                                                                                              
W rknn-toolkit-lite2 version: 2.3.0
[12/03/24 12:15:34] I        Loading RKNN model from /cache/clip/ViT-B-32__openai/textual/model.rknn                                                                                         
--> Load RKNN model
I RKNN: [12:15:35.403] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [12:15:35.403] RKNN Driver Information, version: 0.9.7
I RKNN: [12:15:35.404] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU lite, target platform: rk3566, framework name: ONNX, framework layout: NCHW, model inference type: static_shape
W RKNN: [12:15:35.728] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.

Can someone help me with dynamic_shapes&static_shapes? I have no idea what are them.
Or we can ignore that warning?

[mertalev]

If it is a static shape RKNN model, please ignore the above warning message

I think you can ignore it if there are no errors.

[yoni13]

Small update: I tested the model in Immich, and the ViT-B-32 shows a latency difference of 200 ms after using RKNN(from cpu bcs I am on dietpi).

[yoni13]

outputs :  list[NDArray[np.float32]] = self.rknn.inference(inputs=inputs, data_format='nchw')

and

[01/06/25 09:12:35] I        Running inference on RKNN model                                                                                                                                         
W The input[0] need NHWC data format, but NCHW set, the data format and data buffer will be changed to NHWC.

But


Am I missing something here?🤔
Wasn't the model NCHW?

[mertalev]

Some engines prefer one or the other and will internally only use one because all their optimizations require that format. It looks like this one accepts NCHW, but only by converting it to NHWC internally.

[yoni13]

Some engines prefer one or the other and will internally only use one because all their optimizations require that format. It looks like this one accepts NCHW, but only by converting it to NHWC internally.

So, input being NCHW should not be an issue?
🤔

[yoni13]

I used a random array to test the diff between the result run by onnx and rknn.
LOG: https://pastebin.com/26JGWZUE
DIFF: https://www.diffchecker.com/sLmjBdfZ/

The model seems to be working, might there be an issue with our inputs?

[yoni13]

No, not inputs.

from rknnlite.api import RKNNLite
import numpy as np
import onnxruntime as ort


# Model paths
rknn_model_path = "/cache/clip/ViT-B-32__openai/visual/model.rknn"
onnx_model_path = "/cache/clip/ViT-B-32__openai/visual/model.onnx"

rknn = RKNNLite()
rknn.load_rknn(rknn_model_path)
rknn.init_runtime()

session = ort.InferenceSession(onnx_model_path, providers=['CPUExecutionProvider'] )

def check_for_inf(array):
  """Checks if an array contains inf or -inf values."""
  return np.isinf(array).any()

def test_models(input_data, rknn_model_path, onnx_model_path):
  """Tests the RKNN and ONNX models with the given input data."""
  # RKNN model inference
  rknn_result = rknn.inference(inputs=[input_data], data_format='nchw')[0]

  # ONNX model inference
  onnx_result = session.run(None, {"image": input_data}, None)[0]

  return rknn_result, onnx_result


# Generate random input data and test until inf is found
iteration = 0
while True:
  iteration += 1
  input_data = np.random.rand(1, 3, 224, 224).astype(np.float32)
  rknn_result, onnx_result = test_models(input_data, rknn_model_path, onnx_model_path)

  if check_for_inf(rknn_result) or check_for_inf(onnx_result):
    print(f"Model produced inf/-inf at iteration {iteration}")
    print("RKNN result:", rknn_result)
    print("ONNX result:", onnx_result)
    break

  print(f"Iteration {iteration}: No inf/-inf found yet.")

I asked Gemini to write a script to test, but after 500 iteration, no inf,-infs found.

Hope this leads us to learn more what happened.

[sgofferj]

I don't know anything about machine learning development but I do have a FriendlyElec CM3588 NAS which is mostly idling at the moment. I was anyways planning to move my Immich to that from my 15 years old Xeon home server.
I can't offer much coding help but I can offer CPU/NPU time if that helps.

[yuukh]

build镜像失败了，网上解决这个报错一般通过unset PYTHONHOME 和 PYTHONPATH 来解决，但看日志输出PYTHONHOME 和 PYTHONPATH本就没有值，你有碰到过这个问题吗？

machine translation：
I failed to build image , and the common solution to this error online is to unset PYTHONHOME and PYTHONPATH. However, according to the log output, PYTHONHOME and PYTHONPATH don't have any values. Have you encountered this issue before?

[+] Building 0.7s (14/23)                                                              docker:default
 => [internal] load build definition from Dockerfile                                             0.0s
 => => transferring dockerfile: 4.33kB                                                           0.0s
 => [internal] load metadata for docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d  0.1s
 => [internal] load metadata for docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd0  0.1s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 58B                                                                 0.0s
 => [builder-cpu 1/1] FROM docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd03efd4d  0.0s
 => [internal] load build context                                                                0.0s
 => => transferring context: 1.27kB                                                              0.0s
 => [prod-cpu 1/1] FROM docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d377e2a27d  0.0s
 => [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c123746477da6  0.0s
 => CACHED [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c12374  0.0s
 => CACHED [prod 1/8] RUN apt-get update &&     apt-get install -y --no-install-recommends tini  0.0s
 => CACHED [prod 2/8] WORKDIR /usr/src/app                                                       0.0s
 => CACHED [prod 3/8] RUN echo "hard core 0" >> /etc/security/limits.conf &&     echo "fs.suid_  0.0s
 => CACHED [builder 1/6] RUN apt-get update && apt-get install -y --no-install-recommends g++    0.0s
 => ERROR [builder 2/6] RUN pip install --upgrade pip && pip install poetry                      0.5s
------
 > [builder 2/6] RUN pip install --upgrade pip && pip install poetry:
0.405 Python path configuration:
0.405   PYTHONHOME = (not set)
0.405   PYTHONPATH = (not set)
0.405   program name = '/usr/local/bin/python3.11'
0.405   isolated = 0
0.405   environment = 1
0.405   user site = 1
0.405   safe_path = 0
0.405   import site = 1
0.405   is in build tree = 0
0.405   stdlib dir = '/usr/local/lib/python3.11'
0.406   sys._base_executable = '/usr/local/bin/python3.11'
0.406   sys.base_prefix = '/usr/local'
0.406   sys.base_exec_prefix = '/usr/local'
0.406   sys.platlibdir = 'lib'
0.406   sys.executable = '/usr/local/bin/python3.11'
0.406   sys.prefix = '/usr/local'
0.406   sys.exec_prefix = '/usr/local'
0.406   sys.path = [
0.406     '/usr/local/lib/python311.zip',
0.406     '/usr/local/lib/python3.11',
0.406     '/usr/local/lib/python3.11/lib-dynload',
0.406   ]
0.406 Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
0.406 Python runtime state: core initialized
0.406 LookupError: no codec search functions registered: can't find encoding
0.406
0.406 Current thread 0x0000ffff9d7c3020 (most recent call first):
0.406   <no Python frame>
------
Dockerfile:31
--------------------
  29 |     RUN apt-get update && apt-get install -y --no-install-recommends g++
  30 |
  31 | >>> RUN pip install --upgrade pip && pip install poetry
  32 |     RUN poetry config installer.max-workers 10 && \
  33 |         poetry config virtualenvs.create false
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --upgrade pip && pip install poetry" did not complete successfully: exit code: 1

[yoni13]

build镜像失败了，网上解决这个报错一般通过unset PYTHONHOME 和 PYTHONPATH 来解决，但看日志输出PYTHONHOME 和 PYTHONPATH本就没有值，你有碰到过这个问题吗？

machine translation：
I failed to build image , and the common solution to this error online is to unset PYTHONHOME and PYTHONPATH. However, according to the log output, PYTHONHOME and PYTHONPATH don't have any values. Have you encountered this issue before?

[+] Building 0.7s (14/23)                                                              docker:default
 => [internal] load build definition from Dockerfile                                             0.0s
 => => transferring dockerfile: 4.33kB                                                           0.0s
 => [internal] load metadata for docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d  0.1s
 => [internal] load metadata for docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd0  0.1s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 58B                                                                 0.0s
 => [builder-cpu 1/1] FROM docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd03efd4d  0.0s
 => [internal] load build context                                                                0.0s
 => => transferring context: 1.27kB                                                              0.0s
 => [prod-cpu 1/1] FROM docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d377e2a27d  0.0s
 => [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c123746477da6  0.0s
 => CACHED [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c12374  0.0s
 => CACHED [prod 1/8] RUN apt-get update &&     apt-get install -y --no-install-recommends tini  0.0s
 => CACHED [prod 2/8] WORKDIR /usr/src/app                                                       0.0s
 => CACHED [prod 3/8] RUN echo "hard core 0" >> /etc/security/limits.conf &&     echo "fs.suid_  0.0s
 => CACHED [builder 1/6] RUN apt-get update && apt-get install -y --no-install-recommends g++    0.0s
 => ERROR [builder 2/6] RUN pip install --upgrade pip && pip install poetry                      0.5s
------
 > [builder 2/6] RUN pip install --upgrade pip && pip install poetry:
0.405 Python path configuration:
0.405   PYTHONHOME = (not set)
0.405   PYTHONPATH = (not set)
0.405   program name = '/usr/local/bin/python3.11'
0.405   isolated = 0
0.405   environment = 1
0.405   user site = 1
0.405   safe_path = 0
0.405   import site = 1
0.405   is in build tree = 0
0.405   stdlib dir = '/usr/local/lib/python3.11'
0.406   sys._base_executable = '/usr/local/bin/python3.11'
0.406   sys.base_prefix = '/usr/local'
0.406   sys.base_exec_prefix = '/usr/local'
0.406   sys.platlibdir = 'lib'
0.406   sys.executable = '/usr/local/bin/python3.11'
0.406   sys.prefix = '/usr/local'
0.406   sys.exec_prefix = '/usr/local'
0.406   sys.path = [
0.406     '/usr/local/lib/python311.zip',
0.406     '/usr/local/lib/python3.11',
0.406     '/usr/local/lib/python3.11/lib-dynload',
0.406   ]
0.406 Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
0.406 Python runtime state: core initialized
0.406 LookupError: no codec search functions registered: can't find encoding
0.406
0.406 Current thread 0x0000ffff9d7c3020 (most recent call first):
0.406   <no Python frame>
------
Dockerfile:31
--------------------
  29 |     RUN apt-get update && apt-get install -y --no-install-recommends g++
  30 |
  31 | >>> RUN pip install --upgrade pip && pip install poetry
  32 |     RUN poetry config installer.max-workers 10 && \
  33 |         poetry config virtualenvs.create false
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --upgrade pip && pip install poetry" did not complete successfully: exit code: 1

I don't face that problem, have you tried to rebuild without cache?

[yuukh]

same error with --no-cache option.I'll try to figure it out later.

[yoni13]

same error with --no-cache option.I'll try to figure it out later.

maybe try docker system prune (REMOVE ALL UNUSED IMAGES,DATAS etc) and rebuild
or maybe get your pi on a VPN because looks like you are in China(?

[yuukh]

非常感谢，执行了docker system prune后可以顺利构建镜像了，不知道之前是什么奇怪的bug。网络问题我早已用代理解决了，网络一般不会造成bug，只是apt update之类的命令会很慢。可以让npu发挥作用了，谢谢你的付出。

machine translation:
Thank you very much, after executing docker system prune I can build the image without any problem. I can get the npu to work, thanks for the effort.

[yoni13]

Oh, wait!
After changing Smart Search concurrency to 1, it works! (clip) 🎉🎉🎉

It looks like it got itself race-conditioned or something.

[yoni13]

Search: (CPU)2s -> (NPU)750 ms
I just found out that the previous result's CPU model was not preloaded, there are 100~500ms improvements sometimes.

[yoni13]

https://github.com/leafqycc/rknn-multi-threaded
I wonder how this works.

[devinzhang91]

https://github.com/leafqycc/rknn-multi-threaded I wonder how this works.

Initialize different NPU cores (only rk3588 and rk3576 have multiple NPUs) into an NPU pool, similar to a thread pool. Put Fream into a queue, which is equivalent to asynchronous processing. Get Fream is the result of the previous Fream. If Fream is not processed (queue is empty), skip this loop.

[yoni13]

https://github.com/leafqycc/rknn-multi-threaded I wonder how this works.

Initialize different NPU cores (only rk3588 and rk3576 have multiple NPUs) into an NPU pool, similar to a thread pool. Put Fream into a queue, which is equivalent to asynchronous processing. Get Fream is the result of the previous Fream. If Fream is not processed (queue is empty), skip this loop.

So people might actually knows the "inf" issue, but I don't see anyone documenting it(?

[yoni13]

Ok, now all models (clip ViT-B-32 and buffalo_l) work with concurrency 1.
I will push updates and models to my repo later.
Also, I don't think concurrency 1 is acceptable, will investigate it further when I have time.

[yoni13]

We need an export script that accepts CLI arguments for the Immich model name and SoC, then exports the model and uploads it to Hugging Face. Then we can run that CLI for each model.

I think I can try to make one.

[mertalev]

Yes, that would work

also, is it appropriate to set the immich-app/ViT-B-32__openai/textual/model.rknn a dummy (to mark that this model supports RKNN) and instead load the model from path.replace('model.rknn',f'rknn/{soc_name}.rknn')? or do we have to rewrite the model_path.suffix logic 🤔

And for the file, any way to make Hugging Face not download rknn files if not using rknn? I see, line 71@app/models/base.py

They can just go in the same folder without the rknn folder.

[yoni13]

Yes, that would work

also, is it appropriate to set the immich-app/ViT-B-32__openai/textual/model.rknn a dummy (to mark that this model supports RKNN) and instead load the model from path.replace('model.rknn',f'rknn/{soc_name}.rknn')? or do we have to rewrite the model_path.suffix logic 🤔

And for the file, any way to make Hugging Face not download rknn files if not using rknn? I see, line 71@app/models/base.py

They can just go in the same folder without the rknn folder.

OK👍.

[yoni13]

Yes, that would work

also, is it appropriate to set the immich-app/ViT-B-32__openai/textual/model.rknn a dummy (to mark that this model supports RKNN) and instead load the model from path.replace('model.rknn',f'rknn/{soc_name}.rknn')? or do we have to rewrite the model_path.suffix logic 🤔
And for the file, any way to make Hugging Face not download rknn files if not using rknn? I see, line 71@app/models/base.py

They can just go in the same folder without the rknn folder.

ViT-B-32__openai

https://github.com/yoni13/immich_to_rknn2/tree/unificli
This script takes the soc name and soc name and builds model to that folder(/buffalo_l/detection/rk3566.rknn)

It hasn't generated the model.rknn dummy yet, probably later.
Done.

[yoni13]

I am adding it to rknn/export later

[yoni13]

We only load the onnx model when the get_input and get_output are required now, no more 1 gig memory usage for a textual model!

[devinzhang91]

Great job, I have an Orangepi CM4 and just made a PCB circuit board that converts PCIe (m2 adapter) to 5 SATA. I happen to need this!

[yoni13]

#15241 btw
Porting tests yesterday (my first time writing tests, really tired) and found a unintended(or not idk) bugs that some values passed to the rknn is diff from inputs, taking a look later.

But seems like everything still works(?

[mertalev]

Nice, I'll take a look at it soon

[yoni13]

After this commit yoni13@4b0f93c#diff-11bd5eb879036c49a92372828a94f25a483e31f60e829255bcee33b87ca0260b , we have

Run poetry run mypy --install-types --non-interactive --strict app/
error: --install-types failed (no mypy cache directory)

idk what happened

edit: probably duplicate-module error, just found that mypy issue thread
python/mypy#10863
wait that was fixed??
it only fails in CI

[yoni13]

That test works on my PC, so um........?

[yoni13]

I solved that issue! Wish me luck with the exams tomorrow!

[harris-shelby]

可以打包一个rk3566的镜像测试一下吗

[banjuer]

git checkout dd52c2d68ce9829424bfc15c5ecb09efdadb1ac0 should make the rknn works again

Working well, the NPU of the RK3566 is finally useful. Thank you very much. It's awesome!

兄弟，请问你有遇到可以智能搜索，但是无法【以文搜图】的情况没

[yoni13]

git checkout dd52c2d68ce9829424bfc15c5ecb09efdadb1ac0 should make the rknn works again

Working well, the NPU of the RK3566 is finally useful. Thank you very much. It's awesome!

兄弟，请问你有遇到可以智能搜索，但是无法【以文搜图】的情况没

Are you saying that I can rerun the smart search task but cannot search?
That should be the problem that RKNN does not support CumSum

你是說可以重跑smart search 任務，但沒辦法搜尋？
那應該是RKNN不支援CumSum的問題

[banjuer]

git checkout dd52c2d68ce9829424bfc15c5ecb09efdadb1ac0 should make the rknn works again

Working well, the NPU of the RK3566 is finally useful. Thank you very much. It's awesome!

兄弟，请问你有遇到可以智能搜索，但是无法【以文搜图】的情况没

Are you saying that I can rerun the smart search task but cannot search? That should be the problem that RKNN does not support CumSum

你是說可以重跑smart search 任務，但沒辦法搜尋？ 那應該是RKNN不支援CumSum的問題

对的，是这样的。我搜到了关于CumSum的issue，不过没太看懂，这个当前有解吗

[yoni13]

git checkout dd52c2d68ce9829424bfc15c5ecb09efdadb1ac0 should make the rknn works again

Working well, the NPU of the RK3566 is finally useful. Thank you very much. It's awesome!

兄弟，请问你有遇到可以智能搜索，但是无法【以文搜图】的情况没

Are you saying that I can rerun the smart search task but cannot search? That should be the problem that RKNN does not support CumSum

你是說可以重跑smart search 任務，但沒辦法搜尋？ 那應該是RKNN不支援CumSum的問題

对的，是这样的。我搜到了关于CumSum的issue，不过没太看懂，这个当前有解吗

Nope
I do have a draft PR for detecting it and preventing it from exporting the model, but I have quiz tomorrow

[banjuer]

git checkout dd52c2d68ce9829424bfc15c5ecb09efdadb1ac0 should make the rknn works again

Working well, the NPU of the RK3566 is finally useful. Thank you very much. It's awesome!

兄弟，请问你有遇到可以智能搜索，但是无法【以文搜图】的情况没

Are you saying that I can rerun the smart search task but cannot search? That should be the problem that RKNN does not support CumSum
你是說可以重跑smart search 任務，但沒辦法搜尋？ 那應該是RKNN不支援CumSum的問題

对的，是这样的。我搜到了关于CumSum的issue，不过没太看懂，这个当前有解吗

Nope I do have a draft PR for detecting it and preventing it from exporting the model, but I have quiz tomorrow

兄弟，祝你好运

[devinzhang91]

I would like to ask which version of rknn-toolkit2 are you using? I encountered an issue while using version 1.6, where the onnx model cannot be opened. The reason is that onnx runtime==1.16.0 and onnx==1.14.1 cannot support higher versions of opset in onnx (from immich-app/ViT-B-32_openi), but I upgraded onnx runtime==1.19.0 and onnx==1.17.0 and cannot meet the dependency of rknn-toolkit2==1.6.0. This makes me very distressed.

[yoni13]

I would like to ask which version of rknn-toolkit2 are you using? I encountered an issue while using version 1.6, where the onnx model cannot be opened. The reason is that onnx runtime==1.16.0 and onnx==1.14.1 cannot support higher versions of opset in onnx (from immich-app/ViT-B-32_openi), but I upgraded onnx runtime==1.19.0 and onnx==1.17.0 and cannot meet the dependency of rknn-toolkit2==1.6.0. This makes me very distressed.

We dont use rknn-toolkit2 in the app because of the dependencies issue, we convert those model first at github action.

[yoni13]

I would like to ask which version of rknn-toolkit2 are you using? I encountered an issue while using version 1.6, where the onnx model cannot be opened. The reason is that onnx runtime==1.16.0 and onnx==1.14.1 cannot support higher versions of opset in onnx (from immich-app/ViT-B-32_openi), but I upgraded onnx runtime==1.19.0 and onnx==1.17.0 and cannot meet the dependency of rknn-toolkit2==1.6.0. This makes me very distressed.

for version, we are using rknn-toolkit2==2.3.0

[yoni13]

V1.6 is a really old ! Maybe you didnt know that they changed their repo to https://github.com/airockchip/rknn-toolkit2 ?

[devinzhang91]

V1.6 is a really old ! Maybe you didnt know that they changed their repo to https://github.com/airockchip/rknn-toolkit2 ?V1.6真的很老了！也许您不知道他们将存储库更改为https://github.com/airockchip/rknn-toolkit2 ？

Yes, I have switched to version 2.3.0 and the dependencies have been resolved. Thank you for your reply.In addition, I found a slight difference in accuracy when running the rknn model in the simulator, but I believe this is a normal situation.

[devinzhang91]

@harris-shelby @yuukh
Build image and fix some, and upload Aliyun (ARC).
docker pull registry.cn-hangzhou.aliyuncs.com/devinzhang91/immich-machine-learning:rknn-toolkit-lite2-rknn
docker pull registry.cn-hangzhou.aliyuncs.com/devinzhang91/immich-server:rknn-toolkit-lite2-rknn
Converted RKNN format model ,here:
https://huggingface.co/devinzhang91/immch_rknn

@mertalev
How can I PR to mian?
my github : https://github.com/devinzhang91/immich_rknn/tree/rknn-toolkit-lite2

[yoni13]

Thx for the work and the image!

Just to remind that we are using poetry as our package manager here and the RKNN toolkit lite was already installed if you run poetry install --with rknn, so there's no need to add that in Dockerfile and adding it will result build failed in CI/CD since the 'lite' package is not for amd64 platforms.

And for the HuggingFace model repo, I guess it's better to let @mertalev to decide are we storing it on immich's repo or not.

[devinzhang91]

Thx for the work and the image!

Just to remind that we are using poetry as our package manager here and the RKNN toolkit lite was already installed if you run poetry install --with rknn, so there's no need to add that in Dockerfile and adding it will result build failed in CI/CD since the 'lite' package is not for amd64 platforms.

And for the HuggingFace model repo, I guess it's better to let @mertalev to decide are we storing it on immich's repo or not.

Thank you for your reminder.🎉 I didn't know how to use poetry for testing before, so I will use rk3566 \ rk3568 \ rk3588 for testing.
I packaged the image and rknn model for ease of download, should know that internet connectivity is a headache in China.😄

[yoni13]

Btw pointed out by @devinzhang91, the app will cache the wrong file because we use {soc}.RKNN instead of model.rknn.

Hope not a blocker(?

[Gumbees]

Are you all actively working on this feature? Be glad to help because my entire home setup runs on a cluster of orange pi's. I cannot tell, no Github label.

[yoni13]

clone my repo and follow the instructions here.
Which should work.

[NicholasFlamy]

#15241

[banjuer]

@yoni13 你好，请教个问题，我遇到一个问题，使用【智能搜索】任务可以使用npu加速，但是搜索图片时【以文搜图】却会导致机器学习自动重启，请问这是什么问题啊？内存应该是够的
OS：Armbian24，CPU：RK3566，机器学习镜像：immich-machine-learning:release-rknn，模型：XLM-Roberta-Base-ViT-B-32__laion5b_s13b_b90k
机器学习容器异常退出时的日志：

需要强调的是：在执行任务【智能搜索】整个过程中是一切顺利的，npu也有正常调用，但是无法完成【以文搜图】，非常期待你的答复！

[tlvince]

With a mainline RK3588 NPU kernel (v5) and userspace driver close to being merged, has anyone tried it and/or have an idea how it could be integrated by immich? /cc @yoni13

[yoni13]

Seems like it will be integrated as a TensorFlow Lite delegate, I haven't learn much about it yet. And also that patch seems rk3588 specific?not sure about that but I only got a 3566 Tom Vincent ***@***.***> 於 2025年5月23日 週五 上午2:18 寫道：

…

With a mainline RK3588 NPU kernel ***@***.***/> and userspace <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29698> driver close to being merged, has anyone tried it and/or have an idea how it could be integrated by immich? /cc @yoni13 <https://github.com/yoni13> — Reply to this email directly, view it on GitHub <#13243 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASF5CKGFHNMW2OSKSIS335T27YIG3AVCNFSM6AAAAABPPBM6PWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMRTG44TEMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tlvince]

Yeah, only RK3588 for now. There're more details on the author's blog: Tomeu Vizoso: rockchip. User space (mesa) is nearly there and v3 kernel patch is in progress.

In the meantime, it might be possible to forward-port rknpu to build against mainline.

[hongqigg]

I encountered some troublesome while using the RK3566.
Thanks for your work, the smart search task and facial recognition task work fine, and I can see the NPU being invoked cat /sys/kernel/debug/rknpu/load
However, I ran into problems when running the text-to-image search. The error shown in ML logs is :

Unsupport CPU op: CumSum in this librknnrt.so, please try to register custom op by calling rknn_register_custom_ops or If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3      (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm

Here are some information:
Immich
[v1.135.3]
ExifTool
13.00

Node.js
v22.14.0

Libvips
8.16.1

ImageMagick
7.1.1-47

FFmpeg
7.0.2-9

ML logs

[06/28/25 19:59:14] I        Loading textual model                              
                             'XLM-Roberta-Large-Vit-B-16Plus' to memory         
[06/28/25 19:59:14] I        Loading RKNN model from                            
                             /cache/clip/XLM-Roberta-Large-Vit-B-16Plus/textual/
                             rknpu/rk3566/model.rknn with 1 threads.            
I RKNN: [19:59:18.413] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [19:59:18.413] RKNN Driver Information, version: 0.9.8
I RKNN: [19:59:18.417] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU lite, target platform: rk3566, framework name: ONNX, framework layout: NCHW, model inference type: static_shape
E RKNN: [19:59:21.752] Unsupport CPU op: CumSum in this librknnrt.so, please try to register custom op by callingrknn_register_custom_ops or If using rknn, update to the latest toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn). If using rknn-llm, update from: https://github.com/airockchip/rknn-llm
W RKNN: [19:59:21.754] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
[06/28/25 19:59:21] I        Loaded RKNN model from                             
                             /cache/clip/XLM-Roberta-Large-Vit-B-16Plus/textual/
                             rknpu/rk3566/model.rknn with 1 threads.            
[06/28/25 19:59:39] ERROR    Worker (pid:1154) was sent code 139!               
[06/28/25 19:59:40] INFO     Booting worker with pid: 2088                      
[06/28/25 19:59:53] I        Started server process [2088]                      
[06/28/25 19:59:53] I        Waiting for application startup.                   
[06/28/25 19:59:53] I        Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[06/28/25 19:59:53] I        Initialized request thread pool with 4 threads.    
[06/28/25 19:59:53] I        Application startup complete.

The docker-compose.yml

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
    security_opt: # enables full access to /sys and /proc, still far better than privileged: true
      - systempaths=unconfined
      - apparmor=unconfined
    group_add:
      - video
    devices:
      - /dev/rga:/dev/rga
      - /dev/dri:/dev/dri
      - /dev/dma_heap:/dev/dma_heap
      - /dev/mpp_service:/dev/mpp_service
      #- /dev/mali0:/dev/mali0 # only required to enable OpenCL-accelerated HDR -> SDR tonemapping

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rknn
    volumes:
      - /mnt/storage/AppData/immich/immich-machine-learning/cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
    security_opt:
      - systempaths=unconfined
      - apparmor=unconfined
    devices:
      - /dev/dri:/dev/dri

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
    
  redis:
    command: "sh -c ' if [ -z \"${REDIS_ROOT_PASSWORD}\" ]; then\n  redis-server /etc/redis/redis.conf\n\
      else\n  redis-server /etc/redis/redis.conf --requirepass ${REDIS_ROOT_PASSWORD}\n\
      fi'\n"
    container_name: redis
    image: redis:6.2.18-alpine
    labels:
      createdBy: Apps
    ports:
      - '6379:6379'
    restart: always
    volumes:
      - ${REDIS_ROOT_PATH}/data:/data
      - ${REDIS_ROOT_PATH}/config/redis.conf:/etc/redis/redis.conf
      - ${REDIS_ROOT_PATH}/logs:/logs

The .env

UPLOAD_LOCATION=/mnt/storage/immich_photo_library
DB_DATA_LOCATION=/mnt/storage/AppData/immich/immich_postgres/data/
TZ=Asia/Shanghai
IMMICH_VERSION=release
DB_PASSWORD=19990128031x
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_ROOT_PATH=/mnt/storage/AppData/redis

Linux version:
Armbian Linux 6.1.99-rk35xx-ophub

[yoni13]

Hi, thanks for your report, we are currently unable to solve this issue because Rockchip's tool for adding a custom ops is broken.

We are waiting for Rockchip to fix it or we move on to the open source driver.

[tlvince]

The open source driver was mainlined a few months ago. Would be nice to make use of it, but Teflon/TensorFlow Lite currently is the only userspace implementation, which as I understand it immich doesn't (currently) use.