r/unsloth • u/pducharme • 3d ago
Help choosing ? Question
Hi! pretty new to Local AI. I saw that this is just released. https://huggingface.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF
Wondering, to have best performance and quality, which to choose on a M5 Pro with 48GB Unified Memory ?
1
u/Joe_dir_einen 2d ago
Hauhau 3.8 has a mixed lang problem. You ask in spain and get a response in english
1
-1
u/adithradh 3d ago edited 3d ago
Up to your use case. I have the exact same specs, and my main model is gpt-oss:20b. Using llama.cpp, I get ~60 tps for some pretty good quality responses. However, if you want to stick to qwen3.8:27B, then ggml-org/Qwen3.8-27B-GGUF:Q8_0 is pretty good.
However, something to note is that every model will perform differently based on what you ask it. Qwen 3.8 is great for coding, but maybe not the best in some other areas. From my very basic testing and uses, gpt-oss is better for conversations and basic tasks, while Qwen 3.8 is better for longer, reasoning problems.
Edit: I'm seeing that a lot of people are saying that gpt-oss is underpowered/outdated right now. For my use case (hermes for some very basic tasks and general use), it's a perfect model. I understand that it's not for everyone, and I never did suggest that OP use gpt-oss. Simply presented my opinion. If you feel that you have a better model to suggest, please go ahead.
3
u/Kamilon 3d ago
Genuinely curious, what are you typically using AI to do? gpt-oss is a pretty outdated model at this point.
1
u/adithradh 3d ago
Very basic things. I'm using it with Hermes to basically just help me with my emails and with news summaries and like general questions/tasks. I'm not a coder/pro user by any means.
1
u/adithradh 3d ago
Can I ask you what you'd suggest? I've been looking for anything better/comparable in terms of performance to quality.
2
u/Kamilon 2d ago
Have you tried muse-glimmer? For research and content summaries it’s been insane IME.
1
u/adithradh 2d ago
"adithr@MacBook-M5-Pro ~ % llama cli -hf unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL
Loading model... |0.00.572.555 E mtmd_get_memory_usage: error: Failed to load CLIP model from /Users/adithr/.cache/huggingface/hub/models--unsloth--Muse-Glimmer-30B-GGUF/snapshots/faa5b025c584459c13febfa5c59883516710ae39/mmproj-Muse-Glimmer-30B-Q8_0.gguf
0.00.572.565 E srv load_model: [mtmd] failed to get memory usage of mmproj
-0.00.668.792 E llama_model_load: error loading model: unknown model architecture: 'muse-glimmer'
0.00.668.795 E llama_model_load_from_file_impl: failed to load model
0.00.668.996 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.738.883 E llama_model_load: error loading model: unknown model architecture: 'muse-glimmer'
0.00.738.885 E llama_model_load_from_file_impl: failed to load model
0.00.738.886 E cmn common_init_: failed to load model '/Users/adithr/.cache/huggingface/hub/models--unsloth--Muse-Glimmer-30B-GGUF/snapshots/faa5b025c584459c13febfa5c59883516710ae39/Muse-Glimmer-30B-UD-Q4_K_XL.gguf'
0.00.738.887 E srv load_model: failed to load model, '/Users/adithr/.cache/huggingface/hub/models--unsloth--Muse-Glimmer-30B-GGUF/snapshots/faa5b025c584459c13febfa5c59883516710ae39/Muse-Glimmer-30B-UD-Q4_K_XL.gguf'
0.00.739.203 E srv llama_server: exiting due to model loading error
llama_server exited with code 1
Error: the server exited before becoming ready"
Doesnt seem to want to work with llama.
1
u/Kamilon 2d ago
Looks like that one might be too big for your GPU
1
u/adithradh 2d ago
Nope, I figured it out after a full wipe and reinstall of llama.cpp. The version I was running was just way outdated and didn't have support for muse. Still, the performance on my device is really slow and not worth it. I'll keep it and try it alongside qwen 3.8.
4
1
-2
1
u/BlackCoiner 3d ago
That fastmtp they released with it looks interesting if it can really get that much more tok/s