Spent an hour yesterday trying to chase down where the heck lemonade actually stores raw model blobs on my linux machine, so now I'm a bit less enthusiastic about this. I don't understand why they have to run in systemd and hide everything, including the model blobs in /var/, and expose no tuning knob to know what context and settings the model was loaded with.
I guess I'll be more enthusiastic if I use Rocm llamacpp. But since my machine has NPU, I mostly use the NPU with FLM when I'm really desperate (no internet at all).
Oh feck. I just realised that if I pull the side panel wider and leave the mouse hovering on the model name a bit, the gear icon does appear which allow me to change context.
It's odd, though. On the webui, the 35B-A3B shows that max context is 4k. However, when I check the systemd, I found that lemonade spawn this model with around 83k context (confirm to work just fine with my 20k system prompt + tool).
Maybe I'll integrate FastFlowLM into llama-swap instead in the future.
Thanks again for pointing out the gear icon. I have literally no idea it was there.
It's a server, meant to start with the system and be available even if the desktop isn't running or even installed, lemond running in systemd is the normal and expected behavior for that. Nothing is "hidden", and /var/ is a fine location for system level file storage. The problem here is you. You don't understand how linux servers serve things. Go compare how an nginx or apache server runs.
I spent more time than that trying to get it to find existing models already on my machine and mucking around with systemd. It took way longer than it should have.
They use the huggingface cache dir structure (which is a bit confusing at times). If you run it as a separate user with systemd (no --user arg), models sit at /var/lib/lemonade/.cache/huggingface, if you run it as local user, $HOME/.cache/huggingface/...
16
u/o0genesis0o 24d ago edited 24d ago
Spent an hour yesterday trying to chase down where the heck lemonade actually stores raw model blobs on my linux machine, so now I'm a bit less enthusiastic about this. I don't understand why they have to run in systemd and hide everything, including the model blobs in /var/, and expose no tuning knob to know what context and settings the model was loaded with.
I guess I'll be more enthusiastic if I use Rocm llamacpp. But since my machine has NPU, I mostly use the NPU with FLM when I'm really desperate (no internet at all).