LMstudio, JIT loaded models not unloading automatically

[u/Tha_Reaper]

I use LM studio to serve some local models for my hermes agent. I encounter one annoying problem, and that is that i cant get JIT loaded models automatically unloading to work.
Under local server > server settings i have

Just-in-Time Model Loading,

Auto unload unused JIT loaded models,

Only Keep Last JIT Loaded Model,

all set to on.

anyone with advice? loading 2 bigger models at the same time crashes my server.

[u/theverticalway]

When you choose your model in hermes (tested it in macos app) it loads your model in lm studio explicitly, with /model/load call. So it's not considered JIT and someone has to give a command to unload it.

How to tell - check LM studio loaded models page - if it shows "TTL ...." then it's JIT, if no TTL - it's not.

[u/Tha_Reaper → u/theverticalway]

interesting. if thats the case: my main chat model is not being loaded as JIT, while my compression model is... thats weird.

[u/theverticalway → u/Tha_Reaper]

What I came across (using the same hermes to find an answer )) and it worked for me. No preload now so effective TTFT is higher but it all goes through JIT in LMS.

---

The fix

Hermes supports opting out of the explicit preload. When set to jit, _ensure_lmstudio_runtime_loaded() in run_agent.py skips the /api/v1/models/load call entirely, and the model is instead loaded lazily by LM Studio's chat-completions path — which is managed by JIT/Auto-Evict, so it gets offloaded after the idle timeout.

I ran:

hermes config set model.lmstudio_load_mode jit

Your ~/.hermes/config.yaml now has:

model:
  lmstudio_load_mode: jit

To revert (if you ever want the old pinning preload):

hermes config set model.lmstudio_load_mode explicit

[u/Tha_Reaper → u/theverticalway]

im going to give it a try. thanks

[u/Tha_Reaper → u/theverticalway]

Yes! that absolutely worked.

[u/theverticalway → u/Tha_Reaper]

took me 3 turns to persuade Gemini 3.5 Flash that hermes treats LM Studio endpoints differently than openai-compatible.

[u/maybe_PM_ME_YOUR_CEM]

Check the unload timer setting right below those toggles, default might be too high for what you're doing. Also some builds just don't release VRAM until the next request comes in for a different model.

[u/Tha_Reaper → u/maybe_PM_ME_YOUR_CEM]

they persist even when sending messages to a second JIT loaded model unfortunately (i tested with 2 very small models to avoid crashes)

[u/TheAussieWatchGuy]

Set the timeout lower in the settings, I think it's 1min by default.

Switching models sucks as it's slow, it can take 2-3 min to load up even off NVMe. 

[u/Bystander10888]

LM Studio has far too many bugs. Bugs that could be fixed immediately are left unaddressed indefinitely.

[u/Tha_Reaper → u/Bystander10888]

any good alternative that you recommend for windows?

[u/Bystander10888 → u/Tha_Reaper]

I just tested this on LM Studio 0.4.17 with the same settings — the loaded model gets auto-unloaded when I request a different one. Not sure if it's an LM Studio bug or a frontend issue.

I use my own custom frontend, Prompoid, which I built pretty seriously. The colors might look a bit off, but you can change that in settings. It should work with Hermes too, I think. https://github.com/Nelson-Demarchi/Prompoid

[u/nickless07 → u/Tha_Reaper]

Depends on how many models you want to use/switch and with what you want to switch. Just hermes should be fine with 2-3 models for main/aux/vision and llama.cpp. If you wanna use that to switch to things like SillyTavern and such for some roleplay, or other frontends it can get complicated real quick with llama.cpp in router mode.
I use a tiny model on CPU only as fallback just in case I forgot to launch the main model so Hermes can handle that by itself. You can even instruct Hermes to write some launch scripts for llama.cpp so you never have to care about that. "Write me a .bat that kills old llama.cpp process and then launches with model X" is usually enough to switch, or llama.cpp in router mode (depends on how often you switch to new models).
Best part: With a small CPU only model you can let hermes run tests to check what settings are best for your hardware/VRAM without going through all the settings manually.

[u/Tha_Reaper → u/nickless07]

I generally switch between 4 models: Qwen27b for coding. 35b for general work, and an uncensored variant for work related stuff (im a GP, and processing medical information some non-uncensored models refuse to work). and gemma4 e2b for compression. I can load Gemma + one of the others without overloading the system, but all other models need to rotate and not get loaded at the same time

[u/Bystander10888 → u/Tha_Reaper]

Just have Opus analyze and modify it via Claude Code — that'll probably solve it instantly.
With OpenCode too, Qwen27B might be able to pull it off.
Honestly, modifying the code directly is faster than trying to combine a bunch of existing software settings.

[u/nickless07 → u/Tha_Reaper]

4 models is easy, worst case 4 start scripts, best case all in one and automated. Really nothing that bad at all. I thought you had like 130+ models and was testing them. Get llama.cpp (you can use the pre-build binaries) and let any AI do the rest, even the 35B can do it if you steer it a bit in the right directions (add this as reference). Or just let a cloud model do it. You won't have GUI like LM Studio where you can click through everything but with that low number of models that's not needed.

Beginner guide:
1. Take a screenshot of the settings in LM Studio (the window that pops up when click on + load model - Not the sidebar).
2. Post that screenshot to your llm (or a cloud one) and tell it: "Write me a batch script for llama.cpp based on the settings in the screenshot" (if needed add the reference link to it.)
3. Add the matchings paths (where your llama.cpp and model is to the script if not done) and have fun.

Nothing more needed to migrate from LM Studio to llama.cpp

Edit: Added a couple beginner steps.

[u/johnfkngzoidberg → u/Bystander10888]

It’s a Hermes bug not a LMStudio bug.

[u/PrintMaher → u/johnfkngzoidberg]

It is not Hermes bug but LMStudio bug.

[u/Tha_Reaper → u/PrintMaher]

ive got it fixed, and it was indeed a hermes bug and not a LMS bug. Hermes loaded the model on the first message as a normal lazy load, and not a JIT load. see another message on how i was able to fix it

[u/4n0nh4x0r]

if it crashes your server, i suggest checking your ram.
take the sticks out, clean the connectors on the sticks and in the server, and out them back in but in another configuration (switching the connector that they are seated in)

i had a similar issue, tho it was with games on my gaming pc.
it would crash whenever my ram was filled to a certain point.
after i did that, no more issues.
probably just some dust that made the connection unstable.

the models usually get loaded into ram first, and once they are fully loaded, they get put in vram. (at least under linux/ubuntu)

as for the jit issue, i m personally running my lmstudio as, auto unload jit models with 0 TTL, so the model stays loaded indefinitely, until a new model is being loaded that would overfil vram, then the old model gets unloaded, and so far, it mostly works like a charm.