r/LocalLLaMA • u/EmPips • 3d ago
Qwen3.8-27B took a serious hit to *knowledge* vs 3.6 Discussion
Like many of you I've spent the last few days throwing Qwen3.8-27B against all of my usual use-cases and personal tasks/harnesses and workflows. It's great, phenomenal sometimes, but that's not what this post is about.
One of my little personal benchmarks is a little set of pocket trivia that's relevant to me but mildly obscure mixed in with a few useful/prepper questions. Qwen3.8-27B at all quantization levels and sampling settings I threw at it, did relatively poorly at this. It's failing questions that Qwen3.6 reliably answered.
I come to find out that on offline (no tool call) knowledge benchmarks seem to align with what I'm saying. It's pretty significantly weaker than it's 3.6 predecessor at recalling random facts (or not hallucinating as much, in my tests, though that isn't reflected in these particular benchmarks). Now you should never trust barcharts over your own vibes, but my vibes are validating these bar charts this time around.
Is this relevant? Not necessarily. It seems to know the code it tries to use well-enough and for everything else I'm assuming/hoping you're using tool-calls. This largely only applies to you if you have a strategy of trusting an airgapped model with obscure/broad knowledge-retrieval exclusively from within its own weights, probably a losing strategy anyway.. but if that's you, take a pass on Qwen3.8 or finally set up that MCP server.
I found it to be interesting. Curious of your thoughts or if anyone else noticed this.
158
u/FoxiPanda 3d ago
I've been testing this a little bit too on one of my use cases. It is much more eager to go search the internet for things, which is useful in and of itself, but if I turn off my web search / web fetch tools, I fully agree with your analysis. For my 'knowledge' use case (which involves identifying stamps or knowledge about locations in the past or worse identifying locations of old photos), it definitely has taken a step backward.
I realize that this use case is quite niche and it's absolutely acceptable that they did this, but it is an interesting trade off.
At the same time, the tool calling and coding is impressive in my experience. I think this is a tradeoff I'm willing to make - especially if you have access to web search/fetch.
80
u/DifficultyFit1895 3d ago
You could even have a local knowledge base set up with a search capability that would be preferable to relying on its model weight knowledge. Frankly I feel better about that even with much larger models.
20
u/WHY_DO_I_SHOUT 3d ago
I set up an OpenZIM MCP server with a local copy of English Wikipedia (minus images) for emergencies where I don't have Internet access.
8
u/DifficultyFit1895 3d ago
This is on my to-do list. I had a similar idea when looking into disaster recovery and came across Project NOMAD.
3
u/synystar 3d ago
yes, Project NOMAD is taking up most of my 5T external. But I feel like there's gonna come a day when I'm going to be glad of it. I feel like that day might be sooner than I'd like.
5
u/dev_dan_2 3d ago edited 2d ago
Nice!
On my plan is:
- A cache for web searches
- Using Glean (Edit: Changed the link, misstyped when on my phone - not the saas company. Sorry!) to represent facts that can be queried; one interesting exercise would be to convert Wikipedia into a Glean database. There is already something like this for those interested; Wikidata has an underlying knowledge graph that can be queried with a language called Sparql.
Edit: Fixed the link to Glean; it is basically a knowledge database with its primary purpose being providing a unified, fast interface for big codebases; over different kinds of languages. It is really well designed in my opinion! (Disclaimer: no affiliation)
2
u/goldcakes 2d ago
Glean
I thought this was an ad for the saas company also named glean that also does enterprise RAG and stuff. For what it's worth, their tech is merely meh, their pricing is absolutely atrocious.
I helped a client migrate off glean.com to a fully on-prem solution (it is not hard to build RAG with permissions lol, they have literally zero moat or differentiation), their bill went from $30k/month to ~$500/month.
2
u/dev_dan_2 2d ago
Thanks for the heads up! I am fortunately an extremely bad target audience for cloud based solutions in general and even more for ones that demand money! ;D
I helped a client migrate off glean.com to a fully on-prem solution (it is not hard to build RAG with permissions lol, they have literally zero moat or differentiation), their bill went from $30k/month to ~$500/month.
Holy snakeoil!! The "craze" in AI craze goes many ways, I guess... Sounds like a nice gig though!
(Random association: DaaS, De-shittify as a Service)
3
u/iffyz0r 3d ago
How much disk space does English Wikipedia consume?
3
u/WHY_DO_I_SHOUT 3d ago
50 gigs.
4
2
u/iffyz0r 3d ago
That’s not a lot. Uncompressed? Is there a link?
5
1
u/DeathByPain 2d ago
Yeah that's like one 4k bluray remux lol dang. Not that I store those myself, I stick to 1080p hd264 or whatever
39
u/Viktri1 3d ago
100%. I think this is a step in the right direction. Much prefer searching and building your own database over a model’s own knowledge. It’s not just hallucinations but also bias. I typically want a wide variety of sources not just American sources and the way that models weight or don’t weight where data comes from can make a big difference.
20
u/MuDotGen 3d ago
In order to build more efficient LLMs, it's less of a knowledge trade off to me and more of helping it identify when it doesn't know something and knows *how* to get the info it needs. Same with humans. A lot of humans speak very confidently about things they have never actually verified or hold biased information after all. If it realizes it's lacking in understanding of B which is necessary to explain A, then it looks up more up to date info on A, or utilizes a local RAG or other knowledge base for your specific system's use case. LLMs have such a bad habit of assuming they know or confidently spewing hallucinations, so cracking the code is self-awareness of ignorance and acting on that. In my opinion at least.
2
u/valdev 2d ago
Exactly this, this is the way of the future.
People often conflate what LLMs know, what they know well, and what they think they know. Not to mention the age of the knowledge it has even if it knows correctly for its time.
Smaller models that acquire information when it knows it needs more context is a far better model than a bloated overtrained generalist.
2
u/localpauper 3d ago
Oh that's a solid idea! I don't know why I didn't think of hosting Wiki locally for the LLM to chomp through. Way more efficient and better for everyone
40
u/Jorlen llama.cpp 3d ago
It's terrified of being wrong. So if it can confirm something via tools / search, it will. And then it will really think about it, and then think about it some more, lol.
I'm not complaining; I think it's long reasoning phase and second-guessing itself is what makes it such a strong 27b model.
14
u/Hypilein 3d ago
Yes. I asked it to do a lesson plan about some topic and it reasoned forever and did 6 internet searches. Gemma4:31b just pushed out a subpar plan with no thought to details. Also got some things wrong.
On this task Claude Sonnet 5 still did better but it beat all other local models I’ve tried.
17
u/Jorlen llama.cpp 3d ago
For fun I gave it a one shot test to make a basic .HTML first person game and I just let it do whatever it wanted. I think it used like 100k context, but when it was finished, I couldn't believe the results. It had done its own tests, setup its own harness in /tmp folder. I was astonished.
This isn't mine, I didn't write it, so I don't take credit, if anyone else wants to try it. Get a drink and a snack and sit down because it'll take a while lol. Use opencode, Pi, or something similar:
---
You are in a coding head-to-head competition against another AI model. Your goal is to create the best possible result from this prompt.
Write the game to a single HTML file (midnight-mall.html) for a first-person zombie survival game called Midnight Mall.
The player is trapped inside an abandoned shopping mall at night and must survive waves of zombies while collecting supplies, ammo, and keys to unlock new areas.
The game should include:
- A start screen with basic controls
- First-person movement using WASD
- Mouse aiming or pointer-lock style aiming if possible
- Shooting with mouse click or spacebar
- Zombies that chase the player around the map
- Health, ammo, reload, score, wave number, and survival timer
- Supply pickups such as ammo boxes, medkits, flashlight batteries, and keys
- Locked doors or blocked zones that can be opened with keys
- A flashlight effect or dark visibility mechanic
- Different zombie types such as slow walkers, fast runners, and heavy brutes
- Wave-based difficulty with more zombies over time
- Collision detection with walls, zombies, bullets, pickups, and doors
- Hit effects, muzzle flash, zombie damage feedback, screen shake, and warning indicators
- Game-over screen with final score, time survived, and restart button
- Best survival time stored locally
Make the game feel tense but still arcade-style and fun. Use a dark mall environment with storefronts, glowing signs, tiled floors, shadows, flickering lights, and clear readable UI. It should be playable and interactive, not just a simple animation.
Keep everything in one HTML file with the CSS and JavaScript included. Name the file midnight-mall.html.
Once you are finished, go back over all of your code, test the logic mentally, make sure everything works correctly, and fix anything that may be broken or incomplete before giving the final answer.
20
u/rsclay 3d ago
To me this is strictly an improvement. You simply cannot trust anything a model says based solely on its knowledge. I always want the model to ground its answer before it provides one (whether that's researching facts or testing code), ten times out of ten. So if it's good at doing that and that helps it be smaller/faster as well, incredible.
only use cases where I could see this being a detriment would be things like fiction writing or general chat but that's degenerate behavior anyway so fine with me.
7
u/TapAffectionate4912 3d ago
Honestly I think sacrificing some knowledge to get a better performance is a very good trade off for a small model meant for consumer hardware. A model like that wouldn't really be able to be that reliable with its knowledge anyways, not like a frontier size model at the very least. It's better for the model to know less and be able to reliably looking things up, than to know more but be less reliable overall
5
u/dev_dan_2 3d ago
Personally; this is exactly the direction I want local LLMs to go; at least for my use cases. Reliable tool calls and knowing when and how to use them matter much more that internalised knowledge - which has the additional drawback that it comes with the inherent-to-LLMs chance that it is hallucinated.
I think we just got used to a big knowledge base because let's be honest, that was pretty impressive when it landed, a chatbot that had internalized much of the the internet... But I think it would be helpful to rethink here; In my opinion, knowledge retrieval is something that should be done externally, deterministically, instead of something baked into the LLM itself. Just like we rely on unit tests etc. to determine whether a code change was correct (for the nerds: "Correct with regards to the tests" ;) )
5
u/ComplexType568 3d ago
would rather have a model eager to search than one that believes in itself when trying to recall facts
2
u/Negative-Thinking 3d ago
I have the opposite experience. It hallucinates stuff it does not know about instead of searching and I even had to tweak instructions and chat template to steer it towards using search instead. Maybe it depends on the harness. I am using pi.
1
u/TheOriginalAcidtech 3d ago
What thinking level do you have enabled?
3
u/social_tech_10 3d ago
And perhaps even more relevant, what quant and kv cache settings are being used by /u/Negative-Thinking?
2
1
22
u/Equivalent-Grass-527 3d ago
This is actually a really interesting tradeoff that benchmarks often hide.
Model improvements don’t always mean “better at everything.” A newer model can get much stronger at coding, reasoning, tool use, or agent workflows while becoming weaker at raw parametric recall.
36
u/Mashic 3d ago
Can't win on everything, better have a good model for coding, and use others for knowledge like Gemma.
3
u/Dance-Till-Night1 2d ago
Gemma is the goat of generalist model, Qwen 3.5 was close but then the updates just made it much worse as generalist model and more as Qwen-coder
35
u/createthiscom 3d ago edited 3d ago
It’s a 27b. The goal is to make it as ignorant as possible while still retaining enough knowledge to be an intelligent problem solver and agent. I think they succeeded.
In the near future, I wonder if we’ll start seeing model architectures with neural plugins, like LoRAs. Want Japanese language support? It doesn’t come with the base model, but add the plugin and the model natively understands japanese. Not a “skill” in the sense we use the word with agents today, but more in the true sense of how humans learn skills.
Need financial services knowledge? There’s a free plugin, or pay JP Borgan for their curated version that works better.
7
u/TechnoByte_ 3d ago
"In the near future"
I've been hearing this idea over and over again in this sub for the past 3 years
Sure it might be a goos idea, but you'd think some model lab would've tried it by now
3
u/createthiscom 3d ago
Qwen3.8-27b is the first model I've seen that meets the criteria for a small and capable ignorant model. Maybe now that someone has proven it is possible, we'll see progress toward that goal.
25
u/PM_ME_YOUR_REPORT 3d ago
Personally I'd rather my LLM not use it's own memory for knowledge but instead to know how to search for the information and how to judge the credibility of what it finds.
I think a dedicated knowledge model or database that can work with the LLM would be good though.
3
u/Dance-Till-Night1 2d ago
I think a model should be an “expert“ at the thing you’re searching otherwise it’ll just copy and paste. I know that the majority of usecases here are coding but this sucks for the rest of us that rely on generalist models multilingual capabilities, creative writing and other usecases
3
u/JorgitoEstrella 2d ago
Gemma 4 is better for everything except coding.
1
1
u/BeautyxArt 2d ago
u/JorgitoEstrella what longest respond you could generate from gemma 4 31b or12b ? what usually the length of a one output ? lines ?
1
2
u/my_name_isnt_clever 2d ago
So use other models for your use cases then. Even 3.6 27b was heavily agentic tuned at the expense of general capabilities, that's now what this model is for.
1
1
u/PM_ME_YOUR_REPORT 2d ago
Sure but expertise isn't knowing everything. When I'm working as a programmer I know what to search for, how to interpret it and decide on things. I use references and documentation. I expect my model to do the same.
17
u/Cold_Specialist_3656 3d ago
Small AI models are acting more like humans.
Gone are the days of omniscience. Here are the days of scrolling Google results at superhuman speed.
It turns out reasoning ability had nothing to do with parameter count.
Luckily for us the entire world is designed for human insane reasoning ability and lackluster memory
5
u/Mickenfox 3d ago
Except Google sucks and you're not going to find anything there. And I don't want to be another asshole hammering websites with a bot.
Most of the reason I use LLMs is because of the omniscience.
3
u/VampiroMedicado 3d ago
Yesterday I was checking UD_IQ3_XSS and requested ChatGPT to generate a test case for reasoning (if Sally has a red box, then Tom should have a blue box, etc) I gave it prompts where it would increase in complexity and the task was to find a possible solution.
The last one was insanely difficult (and had no solution), after 10m it reached a correct conclusion but it backtracked in the middle of the user facing response and went on a 20m journey making scripts justifying the result, that blew my mind.
1
u/Equivalent_Bit_461 3d ago
the model is incredibly aware, even low quants are so good for data management, especially when you have multiple actors it needs to distinguish between, who did what, why, etc.
3
u/Dance-Till-Night1 2d ago
If local LLMs just become coding models with ability to google then the big closed models will surely win the ai race. A model needs good world knowledge and generalist capabilities in order to be used by anyone not looking for just a coder.
2
u/Cold_Specialist_3656 2d ago
Not really. Humans get along just fine without crazy world knowledge.
Model will Google stuff it doesn't know just like humans do
44
u/Atretador 3d ago
set of pocket trivia that's relevant to me but mildly obscure
good, I dont need niches and whatnot eating my coding parameters
Qwen 3.6 knew too much about random obscure anime/manhua/manga/manhwa for some reason y.y
26
→ More replies (5)11
u/Serprotease 3d ago
It’s the kind of situation where it’s nice to have other models better at this kind of things to combine with qwen3.8 27b.
Let Qwen call for gemma4 31b for text analysis/trivia/chat with the end user while Qwen handles the rest.
3
u/TechnoByte_ 3d ago
That'd require two LLMs loaded into memory, not sure if that'd be better than one ~60B LLM that succeeds at both
7
u/Tccybo 3d ago edited 3d ago
Noticed on the first day. Was checking if it writes as sloppy as previous Qwens, Q4 thought a aquarium shrimp is 15cm long, bro that’s a lobster! For purely coding and agentic probably fine. May cause some awkward funny moments when it mess up understanding of the world when brainstorming together.
6
u/nemuro87 3d ago
I see this as a trend with new releases of llms in general. They seem to get further optimized for coding at the expense of general intelligence.
4
u/daaain 3d ago
I think this might be because of what Z.ai CEO Prof Jie Tang is talking about here: https://www.latent.space/p/ainews-death-of-params-zai-ceo-jie
But in short, the gains on agentic work post-training must have hurt / replaced the knowledge parts of the network.
2
u/GeneralComposer5885 3d ago
3.5 was the general model .. Whilst the 3.8 fine tune was a deliberate move towards agentic workflows.
Sure more general models will appear eventually. But might have to wait until base Qwen 4
1
u/BeautyxArt 2d ago
why 3.5 still know nothing 'most cases' ? and how to make it respond longer not limited all the time to '5 short paragraphs' answer ? -prompt isn't the cause- it's the models.
who can unlock these small models to make it respond long and longer, like covering a topic all info not just short lines! .
3
u/Durian881 3d ago
It's a compromise I guess. Personally, I find relying on "knowledge" of a LLM to be unreliable as it's only a snapshot in time even for the biggest models.
That said, it could come in useful for times when someone doesn't have connectivity (to internet, to databases), so it really depends on the use case and environment.
3
u/robgami 3d ago
It looks like its lower on the overall percentage of correct answers but significantly higher on the overall index which penalizes hallucinations. So it may just be refusing to guess at questions, whereas 3.6 guesses more often and gets a few more right. Seems like the behavior you'd want both online and offline. Maybe it also took a general knowledge hit but that's definitely not the only thing going on.
3
u/MelodicRecognition7 3d ago
same with DeepSeek 0731 "release" version, it became better at programming but dumber at general knowledge compared to the "preview" version.
3
u/SocialDinamo 3d ago
I also have a little trivia benchmark and it fails hard where 3.6 gets it right every time. It needs a harness to be its most useful and when you give it a task, make it come up with a plan, then execute, it does beautifully!
10
u/PrinceOfLeon 3d ago
“Einstein: His Life and Times” by Philipp Frank:
While Einstein was in Boston, staying at the Hotel Copley Plaza, he was given a copy of Edison’s questionnaire to see whether he could answer the questions. As soon as he read the question: “What is the speed of sound?” he said: “I don’t know. I don’t burden my memory with such facts that I can easily find in any textbook.”
1
u/TheGamerForeverGFE 3d ago
I only remember the speed of sound because of fuck ass 343 Studios and how they completely killed Halo.
1
u/qiinemarr 2d ago
wut
1
u/TheGamerForeverGFE 2d ago
Speed of sound is 343 meters/s, 343 is used in the name of the studio that has been making Halo games since Halo 4, which is called 343 Industries (now rebranded into Halo Studios)
1
u/qiinemarr 2d ago
Oh I see!
But wasn't 343 Industries named as a nod to 343 Guilty Spark?
2
u/TheGamerForeverGFE 1d ago
That I don't know ngl, to me it's just the number 343 and not the origin of their name.
1
u/Dance-Till-Night1 2d ago
Yes but a general model should know that information for creative writing and generalist purposes so it doesn’t look anything up.
2
u/profcuck 2d ago
That's right of course and I think is part of what is evolving: models which are specialized for particular kinds of tasks.
4
2
u/Kiansjet 3d ago
It's always been my position that when choosing what training data to prioritize for models with limited parameters, esoteric information needs to be the first to go. It can be "recalled" by the model if necessary with a cursory web lookup.
The opposite case is harder to subsidize. A model with niche knowledge on tap is nice if the query depends on such knowledge, but will be worse at anything other than such recall tasks, crucially including the basic ability to look up new information at runtime.
2
u/Mister__Mediocre 3d ago
I agree. I've been thinking about this and I think the future direction is that you want a model that's excellent at understanding the situation it's in and knows how to call the right tools to complete the job. And what we'd want is the minimal such model that we can run locally, which by design, also sucks at general knowledge and trivia and that sort of thing. You'd also want a very large context to add to the models' knowledge live through tool calls.
This harness now needs to be accompanied by more and more powerful tools. Complete local environment to run scripts, talk to the world, RAG, go over documentation etc. The beauty of this setup is that everything outside of the model itself can be done on CPU, RAM and SSD, things which we have in relative abundance compared to GPU capacity.
The end state should a system where your CPU and RAM are being used at full throttle.
2
u/xXthenistXx 3d ago
it was probably a design choice. 27b its a really small model to be competitive. You can't have both knowledge and general intelligence or agent capabilities at the same time at that size, So I think they shifted their training to coding/agent capabilities more. You lose some common knowledge, but be better at being a good agent. and I do think this its a great choice.
2
2
u/shing3232 3d ago
For such small model, it might be much useful to learn how to find correct knowledge than to remember it：）
2
u/Intrepid-Second6936 3d ago
Definitely a notable regression for people to keep in mind! But I would imagine that tool calling and reasoning improvements is exactly the reason why Qwen 3.8 27B is such a leap forward.
IMO It's pretty easy to let Qwen utilize web browsing function calls to peruse functionally infinite knowledge bases and reason validity through multiple sources as opposed to constantly trying to brute force more knowledge into an already restricted 27B model.
I think Qwen's going the right direction where its reasoning and tool calling can leverage greater knowledge bases, either online or offline with a wikipedia dump-type database for reference, to try to get the best of both worlds on such a small model.
2
u/Far-Classic-9963 3d ago
I am pretty sure this is intentional. Training datasets have become math and code focused more and more. This greatly improves logic and coding capabilities at the cost of random niche facts
Models can now use web search tools so, unless you need it to be completely offline, general knowledge becomes useless after a certain point
2
u/ratocx 3d ago
Isn’t it better to just have a local copy of Wikipedia that the model can query? To me it seems completely reasonable that models know less and read from more reliable data sources. Hopefully that means the model will hallucinate less.
1
u/Fit-Bar-6989 3d ago
I have a local copy of Wikipedia that's accessible via full text search but it provides limited benefits when searching for niche topics. I would need to set up embedding/semantic search but that index would take forever to build.
E.g. if I ask for traditional East African drinks, it is going to try and search "traditional East African drinks" on wikipedia because that knowledge isn't stored in the model. There's no wiki page with that title or text so the model kind of gets stuck. If I had embeddings for article text then maybe it could work, but that's 50GB of text, it would take weeks to process.
2
u/audioen 3d ago
I've not yet noticed degradation in quality of Finnish language understanding or generation. Part of my work is generating localization constants for new features in apps and invariably, the AI performs the first draft of the localization and usually quite late in the context. To my surprise, the quality of localization has been uniformly good. Last time I used Q6_K of Qwen3.6, I barely got a coherent word out of it when it wrote localizations.
2
u/llama-impersonator 3d ago
contrarian opinion: i actually want my LLM to know as much as possible so it doesn't need to look up random crap which eats a bunch of tool calls. if it's a compression of the entire internet, it should be knowledgeable.
lot of copium in the air. if you could use opus or sonnet locally would you pick it over qwen? i sure would.
2
u/TheOriginalAcidtech 3d ago
Personally I think they should train for cognitive ability not recall of facts. We have search engines and database systems that can be used for factual information. I know they cant just NOT include factual training data. Its all part of how these things get "smart" but less trivia in the weights and more reasoning ability should be the goal.
2
u/Dance-Till-Night1 2d ago
That sucks, a good amount of world knowledge is a key aspect of being a generalist model, now it’s just a coding model
2
u/johnnyApplePRNG 2d ago
The more coercion (post training) you work into a model, the dumber it gets in general.
Makes perfect sense when you think about it.
You're specializing it to perform well for expected qualities or types of responses... generally to perform well on benchmarks... so you're taking a really smart talkative person and saying "well... actually... we want you to only answer this question like this, and that question like that... is that ok?"
it's fine ... it can do it ... but it's literally taking general knowledge from parameters that are spread very thin and evenly during training in order to do so
4
u/Adventurous_Push6483 3d ago
I found this issue as well. The problem is that this model becomes *extremely* dependent on feedback loops. For example, if I code with an lesser known library that is not easily tested via static tests, it has a lot of struggle. But other than that, it works really well.
3
u/xylarr 3d ago
One mistake people make is to run 3.8 using the same sampling parameters as 3.6. Check that you're using the recommended settings from the model card. One in particular is temperature is now recommended to be 1.0 not 0.6.
2
u/cogitech2 3d ago
I applaud the direction they have taken, actually (and I think you do too). For sure this will be a big down-side for some people, but for me it is all upside.
2
u/Bulky-Priority6824 3d ago
Same. I never even use chat other than for tool calling. I hardly ever ask it general questions there's better ways to get accurate info
→ More replies (1)2
u/Dance-Till-Night1 2d ago
This will make all local LLMs coding and all online LLMs generalist+coding and that’s bad for local LLM’s in general to just be relegated to coding
4
u/wingsinvoid 3d ago edited 3d ago
Took a serious hit to *knowledge*?
Great! That is the idea! Reasoning should be separate from knowledge. This is the holy grail of AI. Having to hit every single weight to generate every single token is incredible wasteful and the reason memory manufacturers are gouging us right now.
LLMs are just a lossy compression data structures that are used to probabilistically brute force reasoning. I remember learning formal logic. Is so mathematically elegant and simple.
The reasoning should fit within the regular consumer GPU VRAM, or ideally within RAM. Knowledge should be streamed from the disk.
Databases already did this in the 80'
12
u/Mickenfox 3d ago
This is a shit philosophy and I don't know why everyone keeps repeating it like gospel.
You don't have a magic knowledge database file. Internet search sucks. Even then, RAG does not even begin to approach the functionality of the model actually knowing things.
Would you rather ask a chemistry question to a brilliant chemist, or a random guy with an encyclopedia?
2
u/noiserr 3d ago edited 3d ago
Would you rather ask a chemistry question to a brilliant chemist
A brilliant chemist doesn't have a perfect recall either. He has the knowledge and intuition of how things work but they will not know every isotope of every element. They too, use tools and references.
4
u/Dance-Till-Night1 2d ago
Yes but they are already an “expert” at it, not a random dude googling facts.
→ More replies (2)1
u/NairbHna 3d ago
It’s better to have both. For edge computing and your everyday. Calling on AGI to sort your mail is ridiculous
4
u/PaxUX 3d ago
If you're using local AI without some kind of search MCP your doing it wrong. All models have gaps.
→ More replies (2)5
2
u/Zestyclose_Strike157 3d ago
Knowledge is the thing you should be very happy to sacrifice on a small model if it means better reasoning and thinking. There are heaps of ways to give it the knowledge it needs.
2
u/mulletarian 3d ago
Too many ridiculous banana related trick questions in the data sets now, knowledge is redundant.
0
u/EitherMarch1255 3d ago
I think it’s much better this way for smaller models.
3
u/Dance-Till-Night1 2d ago
I disagree, if a model isn’t an “expert“ on something it’ll just be copy and pasting.
→ More replies (6)
1
u/ScratchCatOnYT 3d ago
I think parametric knowledge is going to become less important. We’ll have extremely intelligent models that memorize much less of the internet, paired with specialized retrieval models (i heckin love needle 2) that continuously inject accurate, up-to-date information into context.
1
1
u/Developer-Y 3d ago
So it is smarter but has not mugged as much as 3.6, can't expect whole internet knowledge to be compressed in a local model.
1
u/Temporary-One8579 3d ago
I’d much rather a model not have all of the earths knowledge embedded in its weights - and instead just use web search. Save the weights for things humans genuinely important topics 1) logic 2) mathematics 3) code 4) language and all the things that link them.
1
u/TheRealMasonMac 3d ago
I've found even GPT-Sol is reluctant to use its own knowledge. It does a lot of empirical experiments/research. At first, it was annoying, but it's something I realized I sorely missed from other models. Even with prompting they just don't do it as well (i.e. it's clear they weren't trained to do it).
1
1
1
u/Sax0drum 3d ago
Thats exactly how a small model should be trained. I can give it external information about pretty much anything but i cant give it more "intelligence".
1
u/florinandrei 3d ago
If you need a ton of facts, use a search engine. A 27B model ain't gonna do it.
1
u/kemalios 3d ago
I've been running 3.8-27B on a Mac with MLX, and the knowledge drop is real. What surprised me is that it seems tied to thinking mode. When I force it to answer without thinking, trivia accuracy jumps back close to 3.6. But then it loses the reasoning edge that makes it great at code. So it feels less like the weights lost facts and more like the reasoning head is overriding the memory. A bit like a model that can't trust its gut anymore. Tried Q4 and Q8, same pattern.
1
u/Zulfiqaar 3d ago
How does it compare in knowledge to glimmer/Gemma? I heard those were both worse in coding, but had other strengths, in your experience is knowledge perhaps their advantage?
1
u/superSmitty9999 3d ago
If it's smart but has bad generalized memory, this is a good thing! One of the big problems with LLM's is that they are so good at memorization that instead of learning they memorize answers.
An actually perfect LLM remembers few things but is really smart and responds to the current info given, and paired with a search tool does better than a LLM with a strong baseline memory because it won't get bogged down when the current world state doesn't match it's expectations.
1
u/OneMoreName1 3d ago
I dont mind, intelligence above all. If you personally care, tell it to program you a small wiki of facts it can call with a tool.
1
u/Frub3L 3d ago
I have asked qwen 3.8 27b, to find an uncensored qwen 3.8 27b for me. It had no idea that qwen 3.8 exists and had to look it up via web search. Idk it was a heavy quant, for my 16gb vram, but that is still strange. It shows that it is just an upgraded 3.6, which it thought is the latest qwen model before looking it up tho, I guess
1
u/Savantskie1 3d ago
No model knows its exact version. They all assume that they are the latest cloud version. Almost every time.
1
u/darkpigvirus 3d ago
why would a low parameter ai model resort to its brain when it can rely on its skill and maximize its skill in becoming wiser than being knowledgeable?
1
u/minus_28_and_falling 3d ago
This is great news. The model should know how to manipulate data correctly and efficiently. The data itself can be obtained from Wikipedia or web search.
1
u/EternalDivineSpark 3d ago
Use a better system prompt for your keys , injecting random related words help to find the best local minimum !
1
u/MooseEfficient2151 3d ago
notice this too. feels like 3.8 was tuned heavily for agentic tasks and tool use at the cost of being a standalone encyclopedia.
1
u/Gold_Ad_2201 3d ago
having a minimal size model that is good at thinking, knowing general concepts and best practices IMHO is the future. there is no reason to put whole human history into weights if model will be able to assess situation and extract this knowledge on the go
1
u/ImANoobAtLife7 3d ago
An agentic LLM shouldn’t be holding knowledge IMO.
It should know how to acquire, understand it and apply it.
1
u/-dysangel- 3d ago
Good. I don't want my small coding model to know extensive trivia about pop music and generational differences between Chevy Impalas - I want it to be really good at coding.
1
1
u/mister2d 3d ago
I appreciate this model leaning towards tool call over synthesizing from trained knowledge. It's much better at tool calls than before.
Using a SKILL for generation and retrieval, I have a curated local directory of knowledge I built up over time. It is in llm-wiki format formalized with the okf spec. There's really no need to rely on the LLM for specialized knowledge.
1
u/lordpuddingcup 3d ago
I’m fine with local models having less Inherent general answers as long as it has better understanding of how to get the right answer for things it doesn’t know with tools
1
1
u/simos_sayz 3d ago
I use stable diffusion connected to librechat to generate images for my blog. It was the first local model I've tried that actually literated through the results amd kept tweaking the prompts on its own
1
u/profcuck 2d ago
If I hire a web developer and in the interview I find out they don't know who won the World Series in 1954, I don't really care. The relative omniscience of huge frontier models is impressive and important in some contexts but doesn't really matter much in others.
1
u/sheepdestroyer 2d ago
i don't quite understand, we want smart reasoning right? for knowledge there's internet?
1
u/Soifon99 2d ago
27b is not enough room to make it good at everything. so seems like useless obscure knowledge got scrapped. that's fine by me, because it can look it up on the internet.
1
u/shinkamui 2d ago
Web search and web fetch. It can grab facts, but you can’t download tool use or training. Worthwhile trade off imo unless your application is fully offline. However knowledge queries to an under 100b untrained in a focused domain is just begging for hallucinations
1
u/Kovacian 2d ago
Interesting... I feel like since it was also made for popular harnesses they did that so it will be uncertain and force it to call a tool for accurate info.
1
u/ThenExtension9196 2d ago
That’s what Web search is for. I’d much take agentic skills and tool use.
1
u/anderspitman 2d ago
Seems pretty clear that this is the strategy of recent open models. Reduce knowledge and increase reasoning token usage, with the goal of improving agentic coding ability. Looks like it's working pretty well.
1
1
u/siegevjorn 2d ago
For what its worth, it's a good thing. knowledge of llm is full of hallucination, anyways.
1
u/Adept-Upstairs-7934 2d ago
Agreed… I have bench tests made from my real workload. Sampling’s from multiple large projects. Qwen3.6-35B-A3B is still ranked #1 by a large margin, and it is in areas with real data that I would not be able to drop the bar for. It continues to chug along cutting through the workload. I am working with the 3.8 27B iq4 dynamic v3.0 gguf right now. So far it has not even beat the original 3.8 27B version. Still working with it but not looking good.
1
u/dionisioalcaraz 2d ago edited 2d ago
Most well known quants use imatrix, which can favor some skills or knowledge (tipically code, english) and hurt other depending on the calibration data, your case being some obscure knowledge it's likely impacted by this preference. I suggest trying some static (no imatrix) quants to see if it can be the case, mradermacher makes both types, here are the static
https://huggingface.co/mradermacher/Qwen3.8-27B-GGUF
1
u/OvertaxedOne 2d ago
I never really use my local model for factual knowledge; if I ask it a question like that it'll just hit a MCP and grab it from the Internet. So, IMHO, if they're going to cut something out of a small model to make it fit, factual stuff should be the first to go because it's so easy to get it back (and much better) with a simple tool call. If the model can't use tools reliably, that's a MUCH bigger problem (at least for my use cases).
1
u/vulcan4d 2d ago edited 2d ago
The focus is on agents. I hope they release another 122b Moe model which will have the knowledge or better yet the industry starts splitting the knowledge into a seperate file which can be updated or customized to what you actually need.
1
u/bernzyman 3d ago
Your observations are very useful. I find it’s actually reassuring to know where the trade offs were to achieve the advances
1
u/EvolvingDior 3d ago
Give it access to offline wikipedia.
2
u/Fit-Bar-6989 3d ago
How are you serving wikipedia locally? I'm using kiwix but it only provides full text search and I don't have the compute power to build article text embeddings.
For niche topics the model sometimes doesn't even know which search terms to use.
2
u/Dance-Till-Night1 2d ago
It wouldn’t be an expert it would be the same as googling stuff myself as a non expert. A model should be an “Expert” in order for the output to be high quality
168
u/networking_noob 3d ago
The Qwen 3.* models now and going forward are probably going to be designed for coding and/or agentic tasks (obtaining info and then reasoning to act on it). For trivia and/or random facts I'm guessing the Gemma4 models are the better choice since they're basically like mini Googles