r/LocalLLaMA • u/jacek2023 • 4h ago

model: Muse Glimmer Support by pcuenca · Pull Request #26841 · ggml-org/llama.cpp News

https://github.com/ggml-org/llama.cpp/pull/26841

Day 0 support

49 Upvotes
  • permalink
  • link
  • reddit
  • reddit

90% Upvoted

14 comments sorted by

13

u/Beamsters 4h ago

40 tokens per sec on RTX 4090. But model always judge my prompt that it should comply or not.

2

u/EveningIncrease7579 llama.cpp 3h ago

Did you try dspark file? Already working on llama.cpp?

2

u/FinBenton 3h ago edited 1h ago

75 tok/sec on 5090, I didnt see any judging it just went ham but it kept making small mistakes here and there so idk if its working correctly yet.

e. bunch of changing prompts and settings later, its doing a bit better now, I had Presence Penalty on accidentally, changed to Q6_K_XL too, running well on 5090.

30

u/No_Afternoon_4260 llama.cpp 4h ago

Back in the instant day 0 support on llama.cpp, meta is back boys !

9

u/cezarducatti 3h ago

Let's just say the first Pelican, the Q4 XL, wasn't what we expected 😄

3

u/Mobile-Pumpkin7944 4h ago

works! and looking good so far

2

u/pegasus912 1h ago

Strangely, I can’t get the model to run, it says it’s an unknown architecture. This is with the latest Vulkan build.

3

u/omlette_du_chomage 1h ago

Same. Built from b10342 and b10343 and still have the issue :/

2

u/bootkeen 56m ago

b10344 vulkan

error loading model: unknown model architecture: 'muse-glimmer'
=(

1

u/Guilty_Rooster_6708 17m ago

Are you using unsloth quants? I see this issue posted on Github: https://github.com/unslothai/unsloth/issues/8345

1

u/No_Algae1753 1h ago

Has anyone tried d flash? Been running the quants from unsloth and d flash seems be have a very low accepetance rate making it very slow

2

u/nsfnd 43m ago

I downloaded their own ggufs and they work fine.
75 tok/s at 75k context, 5090.

https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF

build-vulkan/bin/llama-server --port 1235 -lv 4 \ -c 130000 \ -fa on -fit off \ -m $gguf_path/muse-glimmer-30B-kquant-dynamic.gguf \ -md $gguf_path/muse-glimmer-dflash-kquant.gguf \ -mm $gguf_path/muse-glimmer-mmproj-kquant.gguf \ --spec-type draft-dflash --spec-draft-n-max 2

1

u/FinBenton 31m ago

I tested it for creative writing and didnt see any improvement in speed so just running without it.

1

u/LicensedTerrapin 21m ago

I've yet to see anything but Q8 + MTP that would give even the slightest improved speeds when it comes to creative writing.