Repository navigation
Replies: 6 comments 6 replies
|
The same issue occurred while face detection with my device CPU INTEL N5095. I changed the model from One more thing to metion is when this exception was thown, the undo queue got stuck, but the ml container still reported as healthy, perhaps some exeption handling logic should be investigated. |
|
Same observation here. After an inference crash all work by the ML container stopped, but the container always says healthy. If it were possible to discover this condition, we could restart the container if unhealthy. |
|
This is not a bug as far as I can tell, but just a quirk of OpenVINO’s memory allocation. OpenVINO seems to handle batching in a way that’s efficient for speed but not memory usage. As it encounters batches of different sizes, it makes allocations specific to each particular batch size to quickly reuse when it encounters that batch size again. In this case, a batch is the number of faces in an image. This means that as it encounters images with varying numbers of faces (especially ones with many faces), it can use more memory over time. This increase plateaus when the model stops seeing new batch sizes. I can add an env to explicitly disable batching for cases when the GPU doesn’t have enough memory to handle this. |
|
Thank you for the explanation. In the interim would there be something I could add into the container health check to monitor for this state, so a restart is actioned? |
|
If you can access the machine learning logs within the health check, you can make it search for this error. I don't think the container will be automatically restarted if it's marked unhealthy, though, so you'd have to apply one of these methods to get it to actually restart. |
|
I'm not sure batching is involved in this issue. I do have the same problem and I'm on a Synology NAS (Intel J3355, Apollo Lake). If I read correctly the docs for batching, it is enabled automatically only if the GPU reports OPTIMAL_BATCH_SIZE and MAX_BATCH_SIZE > 1 (https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html#automatic-batching). In my case, the GPU reports: I had this RAM exhaustion issue for a while and I also upgraded my device from 10 to 16GB of RAM but all it did was delay the issue a bit. It shows up quickly during Face detection. Honestly I don't think it's an Immich problem - I feel like something weird is going on between the ONNX runtime and the model, a leak or maybe fragmentation issues. As someone who knows next to nothing about ML, the only workaround I can imagine adding to Immich is a parameter to unload the runtime after N inference requests. In my case, N ~= 400. |
|
That's for the automatic batching feature that internally combines multiple singular inputs into a batch. We don't use that - we explicitly submit a batch of faces to the model. |
|
I believe you can make it periodically auto-restart a worker by setting |
|
I am having the exact same issue after upgrading to Immich v3 on a Proxmox VM with GVT-g GPU (DSM 7.3.1, Intel GPU passthrough, 1024MB GPU memory). All ML jobs (OCR, Smart Search, Face Detection, etc.) fail with CL_OUT_OF_RESOURCES: Server log: ML container execution providers: OpenVINO GPU fails immediately, never falling back to CPU. Workaround: using service: cpu from hwaccel.ml.yml. Any update on fixing OpenVINO GPU support? |
|
Hardware details for context: Host CPU: Intel Core i5-8279U (Coffee Lake, 8 cores) The GVT-g virtual GPU allocates a portion of the physical Iris Plus 655 for the VM. OpenVINO detects both CPU and GPU as available devices, but GPU model compilation consistently fails with CL_OUT_OF_RESOURCES when loading PP-OCRv5_mobile or any other model. |
|
Correction to my previous update - on v2.7.5, OCR was not enabled so I can't compare directly. What I know:
I haven't tested whether Smart Search works on v3 GPU because the OCR failure blocks everything else. But the CL_OUT_OF_RESOURCES crash happens at the ONNX Runtime session level when OpenVINO GPU tries to compile the model, which suggests it would likely affect any model, not just OCR. |
|
One more crucial context: After upgrading to v3, I had a DNS resolution issue (container_name vs service name mismatch) that prevented immich-server from reaching the ML container. So ALL ML jobs since upgrading to v3 were failing with 'fetch failed' before ever reaching the GPU. I only fixed the DNS issue today. The CL_OUT_OF_RESOURCES hit was from the FIRST actual GPU ML request to reach the ML container after the DNS fix. This means I cannot be sure if this is a v3 regression or a pre-existing GPU issue that was always there but masked by the DNS problem. It's possible that on v2, I was also hitting CL_OUT_OF_RESOURCES but never noticed because DNS was working then. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The bug
When running Face Detection on a system configured with OpenVino ML - the immich_machine_learning container will eventually throw an exception complaining out of GPU resources. immich_machine_learning does not recover.
This has been recreated triggering "ALL" Face Detection, but the exception can appear at an indeterminate time after some success with face detection.
The OS that Immich Server is running on
DSM 7.2.1-69057 Update 5 | Docker version 20.10.23
Version of Immich Server
v1.117.0
Version of Immich Mobile App
v1.117.0 build.162
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
...
Relevant log output
Additional information
Host hardware:
Synology DS423+
CPU: INTEL Celeron J4125
Physical memory: 18 GB
All reactions