[mertalev]

Branched from #2642.

This PR allows models to be unloaded after a period of inactivity. The default is 300 seconds and can be changed with the MACHINE_LEARNING_MODEL_TTL flag. In addition, it also cleans up main.py and refactors some model-specific logic into a new file.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)			Jun 5, 2023 6:16pm

[jrasm91]

Did you verify an actual decrease in RAM usage when a cached model expires?

[mertalev]

I tested an earlier iteration where the ttl was a kwarg for cache.set. I removed it since the docs say you can set a default ttl, but it doesn't seem to do anything so the models weren't actually getting deleted. It works as expected from my testing now.

[alextran1502]

Hello, can you help resolve the conflict? Thank you

[mertalev]

Rebased

[alextran1502]

Do you know if it is expected not to unload all models from RAM? I saw some reduction in RAM usage, but probably not to the level I expected

• Standby was using 2GB of RAM
• Upload a few photos to trigger the machine learning jobs. RAM usage bumped up to 3GB.
• Goes back down to 2.3GB after TTL expires.

[mertalev]

I'm not sure. It's normal for Python to keep reserving some garbage collected memory for some time in case it'll get used again.

It could be that, or it could be that one of the models isn't getting garbage collected because it's still being referenced somewhere.

If unloading models and reloading them keeps increasing memory usage, it's probably the latter. But if it stays roughly the same then it's probably just Python, especially if it goes back down eventually.

[mertalev]

I did some testing and it's not so much Python holding onto memory as it is the model backends themselves. When the models are unloaded, the backend runtimes are still allocated so they take up a certain amount of memory.

As for reloading, I set up a test environment to delete and load models repeatedly. ONNX Runtime more or less maintains its memory usage over time (how much memory it frees varies a lot, but on the next load it cleans up more so it averages out). For PyTorch, reloading increases RAM usage over time, even when running in CUDA. VRAM usage is completely normal. All of this behavior is the same whether the model cache is used or the models are loaded directly.

The takeaway is that model unloading is still somewhat experimental. It is worth it for freeing VRAM, but for RAM it will save memory in the short term but possibly use more memory in the long term (>30 reloads).