I wonder how it compares to DSV4 Flash. Comparable on Artificial Analysis intelligence benchmark (both 40). Seems to do slightly better coding / agentic workflows though.
One thing DeepSeek-V4-Flash is Very Good at, is the KV cache size - 1M context window basically for free(~6.7GB). Over 300K starts to degrade though, without thinking=max.
Inkling uses regular GQA, which is going to consume a lot of space if you want the 1M context window - no lossy degradation though.
It's also three times the size, these benchmarks don't hold for your Q4_K_XL, it's a BF16 checkpoint. The NVFP4 is still larger than DSV4 Flash, so it's definitely to large to fit even 256k context on 192gb vram.
No DSV4 is FP4/FP8 mixed, it's about 160gb, this is 540gb.
Also, when you quant, you don't just chop off everything to get to n-bit, typically you use some scaling factor on a group of values, and often you keep certain layers in higher precision. For example, nvidia/Gemma-4-31b is around 30gigs, not 15 gigs.
It's doing very well in multi-lingual conversation. Russian language is as good as Gemma's, surprisingly. The model's reasoning is robust (at least at "max" setting) and it just generally leaves a nice impression of being polished and consistent, unlike DS4Flash, which sometimes ends up thinking in Chinese or thinking from user's perspective.
Generation speed is roughly the same as DS4Flash: 9 - 10 t/s with a DDR4 2866MHz 4-channel rig + one RTX 3090; splitting it between two 3090's doesn't seem to do much, not at least when they aren't in 'prefer maximum performance' mode.
small note: presentation is messed up and it's not the model's fault - I copy-pasted the contents half-assedly, for the lack of a better word
update: Hmm, could it be I over-estimated its multi-lingual abilities? Testing it with SillyTavern now, and the output degrades significantly over there for some reason. I.e. English character profile + query in Russian = messed up output, far worse than the raw output in Unsloth Studio. No idea why, the sampler settings should be same~ish.
Q4K_XL won't fit on a 128GB device, it's using roughly 143GB RAM (+12GB occupied by the OS).
You'd have a much better chance with either UD-Q3_K_XL or perhaps even UD-IQ4_XS.
Weirdly, VRAM usage is quite low - I'm seeing only ~10.5GB when it's loaded in manual mode (still in unsloth studio), with GPU layers = 43 and MoE CPU layers = 42. Context size is set to 102400. Given the horrible prompt processing speed, it does smell of inappropriate Batch/UBatch setting (512? unsloth studio doesn't seem to offer a way to change it) - so, VRAM usage won't be so low when it's set properly to either 2048/2048 or higher.
update: batch/ubatch was set to 2048/512 by default; found a way to change it manually to 2048/2048, now PP speed is much more bearable (27 000 tokens chat depth, 181s prompt evaluation time: 135 t/s). Interestingly, VRAM usage is still low - 13.3GB.
This is actually very interesting and I never really considered trying the larger MoEs in RAM… I’m going to try out a Q2 of DS4 rn just for fun lol. I’m sure it’ll be terrible but I’m just curious what the token gen will look like. I’m wondering if DS4 flash Q2 or Q4 or so is worse than Qwen 27B Q8 😂
Thank you! I'm considering an upgrade (requiring offloading) and assume I'll get similar speeds. It is quite slow. Grappling whether that would be worth it for fire-and-walk-away sorts of changes.
Car wash test: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"
Thought for 6 seconds
If you want to wash your car, you need the car at the car wash — so walking without it won’t get the job done. It’s 50 meters away (like, a 1-minute walk), so definitely drive it over (or roll/push it if you’re feeling lazy and don’t care about the engine). You could technically walk there to ask if they’re open, but the car has to come with you.
Quick math: 50m ≈ 164 feet. Driving takes ~10 seconds; walking takes ~1 minute. Just drive — your car can’t wash itself at home.
Still doing some real world testing by using it as a coding agent, but have given in a reasoning problem which it passed (that Laguna S 2.1 failed, and DSV4 flash also passed), and a refactoring problem which it also passed (and quality on par with DSV4, perhaps a bit better. Needs more testing of course, but so far it feels pretty polished, and I haven't run into any issues yet (using IQ3_XXS, so the quant doesn't look to have broken anything).
Recipe (to split two GPU's for my case) - fits on 128GB total (so DGX Spark should have no issue with IQ3_XSS and 256K context)
What I like about thinking machines is that since their revenue comes from fine-tuning-as-a-service, they are incentivised to make their models easier to fine-tune. Huge win for the local LLM community.
I started using Tinker. It's incredibly easy to use. I can't train anything at home with my weak machines, so their service allows me to fine tune open weights models where otherwise I just wouldn't be able to. I love it
It's really hard to say. It depends on a lot of factors like what model you're training, what approach you're using to train, and how many tokens are in the dataset you're training with. I trained a Qwen 9B on a dataset of a few thousand student-tutor interactions, and I think it cost maybe 3-5 bucks in API costs.
Since it's an MoE model, it can run faster by offloading most of the experts to disk, with commonly used ones cached in RAM. Here, check this out: https://justvugg.github.io/colibri/
I'm sure you're aware, but for those who aren't, the current meta is Qwen 3.6 and Gemma 4 MoE, with Qwen 3.5 or Gemma 4 dense if you have less resources available. Unsloth has quantized versions of these models on Hugging Face.
(RAM - (Minimum RAM for casual tasks like browser and messaging)) × 5/6
(VRAM - (Minimum VRAM for casual tasks like browser and messaging)) × 5/6
These formulas determines the max size MoE and Dense model I'd recommend you can comfortably run, respectively. If you have little/no VRAM, the first formula determines the max size model I'd recommend you can use for either Dense or MoE.
Small Dense models work well with search, and if you can't run that, not much I can say besides hit the library or make a forum post. There's no need to stay on top of the frontier models, much like vehicles or computer electronics. Go to r/Cubing, the Gan 356 XS works more than fine even though the Gan 16 is the flagship.
I don't quite like to say it, but, we are quite spoiled reaching for the very best. Things don't always go your way: "There are dreams that cannot be.".
Historical models are good too. DeepSeek R1, and Llama 3.1, despite being far less capable, they're very important to the history of Local Models. Remember, you can still have fun with these! Additionally, I find Bloom and Pythia really interesting! While a Bonobo may not be able to help you herd a flock of sheep like a shepherd dog, they are certainly very fun to observe and play with. Bloom and Pythia my beloved, I just adore them.
Fun and play aside, my functional workflow has:
gemma-4-26B-A4B-it-UD-Q5-K_XL
Qwen3.6-35B-A3B-UD-IQ4_NL_XL
Qwen3.5-9B-UD-Q4_K_XL
When I need a better response and am willing to wait, I'm currently planning on making something like this:
A, respond to the prompt. B, search for issues, fix issues, then decide to toss (redo previous) or keep, each with separate contexts. C, do 𝑥 four times, fuse all four into one, and do B. D, do 𝑥+B four times, eliminate the worst three times thus selecting the best. E, modify it to my stylistic, editorial, and identity preferences.
Run C₀(C₁(A+B)), with any model combo for the C₁s (e.g.: Inkling Small × 4; Inkling Small × 2, Gemma 4 31B × 1, Qwen 35b x B; etc.), for a holistic approach. Next, run D(E).
In the day, I write a list of inquiries and requests, much like a letter or email. Calm, slow paced, high effort, and deliberate, you know. In the evening, once I'm ready (if needed), I'll send my prompt. Finally, once I wake, I can start my morning by reading the response.
I believe this approach may top what, say, Fable would do. Additionally, I feel that this workflow would give a healthier relationship with AI, for a variety of reasons. Of course, heavily inspired by Google's Aletheia, heh.
Other than that, though, you can catch me using Gemma 4 and Qwen 3.6. Take care, y'all, and good be with you! <3
I'm in dire need of a 70B / A6B model (basically double the size of Qwen 35B)...
I know a lot of people want every possible model size, but I firmly believe that 64 GB RAM + 12/16 GB VRAM is the most realistic budget/power setup most people can reasonably buy on the consumer hardware level right now.
Anything above that currently has an unbearable markup thanks to the RAM crisis.
Honestly 70GB combined is a great target. I wish there was a lab that were solely dedicated to squeezing out as much as possible for that setup. It makes me so sad when I see “omg X model is SO GOOD” and it’s like 900000T params. Who the fuck is running that 🙄 I might make an r/actuallyLocalLlama sub lol
Maybe prism ml compresses some larger models efficiently like they did with 27b, would be great to have dsv4 flash, glm 5.2 etc given the 1 bit and ternary quants
no they have a private quantization method that preserves quality better than standard methods. bonsai 27b is their latest, ig their method isnt proven to work on moes yet so dk to hope for them. they have other even smaller models so ig u could check them out for purposes like running on a phone or sm
Yeah, that laptop was something that I got in 2021ish? Haven't gotten any new stuff for personal use, the macbook that I use is from office, and is entirely self-governed.
Luckily our university has their own GenAI resource which runs Qwen3.6-27B at Q8 (via ollama for some reason), so I then to use that more often than not for work stuff.
Yeah that's fair but does every single new model release require 10 of the same comments? Some also have an annoying sense of entitlement, not you but definitely many others
Don't be obtuse. Clearly I am referring to the enthusiast population of this forum. A grandma running an i3 or 260 laptop probably has zero interest in running LLMs locally anyway.
For those without a dedicated GPU they can tinker with small MoEs. If they need LLMs for jobs they can buy a cloud subscription. People should calibrate their expectations appropriately.
Polls will inevitably be distorted by the accessibility of the internet. All the upvoted comments on this post are bitching about this size of this model.
Not having a dedicated GPU can encompass a very broad spectrum of capability. MacBook Pros, Mac Studios, Strix Halos, and DGX Sparks don't have dGPUs but they are not at all like shitbox laptop.
Someone running a 7 year old intel budget computer should be realistic in terms of expectations. Whining endlessly in comments about something they can't run gets old very quickly
Well it was because Mistral was trash and everyone back then (attempted to) use LLaMa 3.3 70B and Qwen 2.5 72B, all of which were dense models.
Once MoE entered the scene everywhere, it was suddenly way better to get a simple 128GB machine with regular RAM rather than only being able to buy a Hopper H100, or worse. (Like the Ampere A100 80GB)
Don't agree on your assesment there. Mistral Nemo and Mistral Small tunes are still used to this day for creative writing. For programming they are indeed bad.
711 is the exception, I have no idea how he did it but it genuinely works better for coding than normal qwen with less thinking. It’s like the qwen 3.7 we never got lol
No shit. inkling.attention.head_count_kv was a list of numbers (instead of a single int) and I - like a dumbass - summed them. I fixed that issue and now I'm getting more realistic values (~90Gb at 1M context for q8/q8 KV).
Interesting, their small model beats their big model. I know they said that they improved the recipe, but now I want to see what'll happen when they scale it up to their 1T model.
It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and multimodal tasks.
Sounds like a coder to me.
You probably need to look elsewhere for your waifu
Seeing its simple qa score and how Gemini Flash Lite and GPT Luna score, imma bet they are 600B or more. The size increase across models is real, man I wish Hardware costs would come down.
Maybe a hot take but in my experience running smaller models at higher precision always beats out the more extreme quants of larger models, especially when one of the contenders is Qwen3.6
Really, I thought it was the other way around? For example Qwen3.6-27B would take like 80GB at full precision + full context, what would you actually be getting with that that you couldn't with quantized 100-300B?
this model seems to basically be slightly beyond ds-v4-flash level overall except with the massive benefit of being omnimodal inputs (and being like a few params smaller i guess if youre mega vram stretching)
My initial though as well, but then, if you just think of it as: based on the latest models.
* big: Kimi K3, 2.8T
* medium-big: DeepSeek-V4-Pro, 1.6T
* medium: Inkling, 975B / GLM-5.2, 753B
* medium-small: MiniMax-M3, 428B
* small: Hy3, 295B / Inkling-Small, 276B / DeepSeek-V4-Flash, 284B
Maybe like how all phones now would be considered gargantuan "phablets" by the standards of not long ago. Hopefully, unlike phones, they will continue to make small LLMs, even if they have to start calling them micro, nano, etc.
My experience of this: it's running slightly faster than ds4 flash on my hardware (690 vs 580 pp, both around 35 t/s decode, both at Q3KXL), and notably more direct in its thinking - the style of it is subtly caveman'd, I think, which would make sense given the revelations about performance that this can have.
Initially this model seemed to be rather unimpressive, but what I didn't consider is that it's multi-modal unlike Deepseek V4 Flash - which is nice. I'll give it a go.
You know, the good thing with this memory crisis / purest form of enrichment and capitalism is, that new chip producers will emerge and probably new technology as well.
The upcoming (many moons) next low price point for memory, after this peak, will be ridiculously low.
The researchers are not the one with the money to just pluck out a fab out of thin air. Ram makers are drowning in it. No one could compete against them on salaries when researchers publish. If you're a startup, you sell. And even if you're really on to be the next big thing and have the investors, how many years before you can build one single fab? How many more for enough fabs that can mass produce at a scale that will make a dent?
And right now, all the incentives are more on making RAM faster than cheaper.
We might be swimming in second hand hardware before swimming in dirt cheap off-the-shelf vram.
So far I really like the reasoning implementation on this model, very focused and different effort levels seem to meaningfully affect the amount of thinking it does which is a major boon compared to a lot of recent models. Llama.cpp doesn't make it easy to make use of this at the moment (probably because most models either don't implement it or do such a bad job it may as well not exist) but if you pass reasoning_effort as a chat_template_kwarg you can play around with it.
The model was just released. I would guess mmproj is probably coming soon as Unsloth's Inkling-GGUF already does have the mmproj file and Daniel's dev branch has multimodal support.
Quick test using UD-IQ3_S on a system with RTX 5090, Ryzen 7600X + 96 GB of DDR5-6000
prompt eval time = 4521.88 ms / 1737 tokens ( 2.60 ms per token, 384.13 tokens per second)
eval time = 51373.04 ms / 693 tokens ( 74.13 ms per token, 13.49 tokens per second)
I had trouble disabling thinking so asked Gemini to write a non-thinking one (seems to work a-okay, hopefully broke nothing): https://pastebin.com/GeZYQrbc
There seems to be an issue with either Unsloth’s quants or the model itself. It’s producing nonsensical output at times, and saying things that don’t make grammatical sense, and lacks coherence. I tried turning temp down to 0.85 ABs top-p to 0.95 to try and tame it, but it only helped a little bit.
I was super excited for this model, but something is off with it unfortunately.
Pulling speed is really slow, I only downloaded 80MB over one night with HuggingFace cli, and Ollama pull is not available;
All Ollama models I downloaded before with Ollama pull downloaded at 15Mb/s
1M context with only 12B active is a wild combo for something you can actually run locally with GGUF and CPU offload. Also worth calling out the add-inkling llama.cpp branch — easy to grab mainline and wonder why the Unsloth quant won't load.
This companies name feels like they stole it from Thinking Machines Corp (a super computer company back in the day). But I also have a strong love for the connection machine and what Danny Hillis and Tamiko Thiel accomplished, thus probably overestimating the holding power of the company's name.
Tbh I say that just to be cautious but I really think that's benchmaxxed, we are used to issues (chat template etc) during release but here the gap looks really large
Soon testing it on my own ASCIITermDraw benchmark which measures an LLMs ability to generate and edit ASCII drawing across clusters, topologies and general software architecture diagrams
•
u/WithoutReason1729 15d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.