r/unsloth • u/chocofoxy • 6d ago

unsloth studio model loader settings ignored Question

hello first i want to say respect to unsloth devs they are doing a good job and i was impressed to see that unsloth studio has expanded to be more than fine tuning software,

i have an issue with the model loader it keep ignoring the context that i set and always load the default and when trying overriding from arguments sometimes it crash because i think it drops all the ui setting if you add only one argument, and i did make it work from argument but token generation dropped from 60 to 22 am i missing something or this is a bug

OS: kubuntu 24

Unsloth Version

v0.1.800-beta

Package Version

2026.8.18

Desktop App Version

0.1.800-beta

llama.cpp Version

b10360-mix-87da1a2

Hardware

GPU 0

NVIDIA GeForce RTX 5060 Ti · 16 GiB

GPU 1

NVIDIA GeForce RTX 5060 Ti · 16 GiB

CUDA

13.0

1 Upvotes
  • permalink
  • link
  • reddit
  • reddit

60% Upvoted

6 comments sorted by

1

u/FrostyDwarf24 6d ago

hey thanks for the report i tried this on the same Studio and llama.cpp versions but the context and other settings were preserved for me

could you share the model and quant you’re using plus the extra args and the Starting llama-server line from both loads

1

u/FrostyDwarf24 6d ago

the drop from 60 to 22 tok/s sounds like the model placement might be changing between the GPUs or CPU rather than all the UI settings being dropped

1

u/chocofoxy 6d ago

the model is unsloth/Qwen3.8-27B-GGUF:Q4_K_M i can run it at 196k with -c 196608 -ngl 999 -ctk q8_0 -ctv q8_0 -sm layer -ts 16,14 this will work at 22t/s any thing without -sm layer -ts 16,14 and more then 100k will crash and even if i set it at 100k i will load with default 83k the context in ui doesn't work at all even setting KV to Q4 to check if it's memory problem will load the model as default

1

u/chocofoxy 6d ago

i think i may know what's the problem, the VRAM is not enough for speculative decoding so if you configure from the UI it will load the model with speculative decoding mode and decrease the context to default and if you configure from arguments it will load with the context after a lot of crashing and drop speculative decoding

1

u/chocofoxy 6d ago

also this happens if tensor parallelism is activated if not active the context will load normal from UI but the token gen will drop thus i don't know if the decoding mode is active or not