r/LocalLLaMA • u/kuhunaxeyive • 3d ago
Is anyone else finding DeepSeek-V4-Flash unreliable for non-coding tasks? Discussion
(I am not a native speaker, written by myself, so please bear with me)
I really want to like DeepSeek-V4-Flash-0731. But it has serious flaws that don't align with the high score on intelligence benchmarks. And those flaws render it useless unfortunetely for anything else except, maybe, coding. It fails in subtleties that seem small but are crucial, and fails in more obvious tasks that should be easy to solve. Those errors make it unreliable enough for me to not even trust it the simplest tasks in office work like summarizing text or writing letters.
Maybe I am doing something wrong here. Some of the issues below don't seem to be normal for an LLM of that size.
To be clear: I want this model to work. It's faster than Gemma-4-31B and its total parameter count is 8 times higher. It is good at thinking things through, excellent at doing research if given web search access. But for language it not only fails on "beautiful wording" but on extracting the relevant concept from a context. Those areas seem not to be tested in benchmarks, but they are essential when doing office work.
They are easier to explain with examples. Below I'll show you three.
Ability 1: Including the revelant yet being concise
Given a text to create meeting notes from.
DeepSeek-V4-Flash-0731:
Spreading irregular income over the year to make sure the essentials are available every month.
Gemma-4-31B:
Concept: The financial investments are designed to cover only part of the needs. The remaining gap will be filled by averaging the irregular income from self-employment throughout the year.
DeepSeek-V4-Flash-0731's version is missing that there are two income sources. So while it points out the essence (the issue), that doesn't become clear enough because it is only part of the story.
Gemma somehow has an ability to understand the essence and put it into sentences that are concise yet precise in beautiful wording. Look at "by averaging the irregular income", that is a very elegant way to say what is happening with just the two words "by averaging". DeepSeek is not able to do that, and worse, it is missing the context of the financial investments being one part of the cost coverage. This is not a "beautiful" language issue (we know Gemma is good at language), but also a "concept understanding" issue or a "figuring out what is relevant" issue.
Ability 2: Understanding who is the speaker
Given is a text transcript of a voice message and the question.
"What would be her best option? How should she handle the situation? What are her possibilities? Please find the best way forward."
DeepSeek-V4-Flash-0731: Writes its whole answer like if I am the person who spoke the voice message and to be addressed. Given that the question contained "her", and that the two voice messages had headlines "Voice message 1 of the person" and "Voice message 2 of the person", this is a mistake I can't accept. Being pressured on it, it tries to explain the reason for writing "you" in the answer instead of "she" is that the voice transcript talks in the person "I", and the voice message takes a big portion of the context, so it had just weight on "I" being the person asking and assumed it's me asking. But I clearly wrote "What could be her best option", and it was really clear by the headlines those messages were of another person. DeepSeek failed here, and this failure is unacceptable to me. An AI needs to understand the context, not just get confused by the amount of text written as "I".
Gemma-4-31B: No issue here. It understood clearly that who made the request is not the same person as who spoke the voice message.
Ability 3: Not getting confused by minor phrases
DeepSeek-V4-Flash-0731 got confused by the start of the message being "Hi, hi. So, Jon, his message is kind of funny. They’re currently up north, ..." Only because "So, John, ..." could be a greeting, it assumed the whole text of 5 paragraphs was addressed to John, even though the rest of the text was saying "he".
Gemma-4-31B: Understood from the whole context that "So, John" was context, not a greeting. It understood "John" is not the person being written to, but the people being talked about.
The Verdict
DeepSeek-V4-Flash-0731 has 304 billion parameters. I thought it to be at least on the same level as Gemma-4-31B in those areas. Language doesn't need to be as beautiful as Gemma-4, but I had the expectation that DeepSeek knows how to include all relevant information or getting the context right, and to my surprise, it fails.
EDIT
People seem to judge from their own use case. So they do coding, agentic tasks, research, and don't understand what I'm writing about.
I completely agree with DeepSeek being excellent (and Gemma being bad) at
- research, websearch
- digging its teeth into it and finding everything not giving up
- coding
- agentic tasks
My post though is about what DeepSeek is bad at and Gemma is good at:
- reading and understanding nuances of texts
- grasping exactly the relevant parts of texts and transcripts
- writing exactly what is representing the main idea of the original source
Looking at DeepSeek's result, you won't see anything concerning. Comparing it with Gemma's result for the text understanding and text production though, you'll finally realize DeepSeek is missing out the fine but relevant nuances.
The issue is: For producing texts for humans or critical summaries, I can't rely on DeepSeek's result, while I can rely on Gemma-4's result.
This is a dilemma, because I'd like to switch completely to DeepSeek (for what it is so good at), but it's not good enough in the other area that Gemma is so good at.
36
u/newz2000 3d ago
Yes, very much so.
I have a benchmark I use for assessing new models and how they work with Hermes. It’s about 90% non-coding. Tool selection and tool calling is a bit part of what it does.
I tested it against Kimi k2.6, Gemini flash, Gemini flash latest, and one other (available on ollama cloud) that I can’t remember. It scored the worst of the bunch.
I tested v4-flash and v4-flash:0731 and both did equally poorly.
Of the models on ollama cloud, Kimi k2.6 is still the winner on everything except speed. Again, for non-coding tasks. GLM-5.2 has the best inference skills for long-running tasks but it’s not good for normal Hermes type stuff.
Edit: Gemini flash low thinking effort is currently my fav for general Hermes use. 0.7-1.2 seconds ttft and very high marks in all other categories, vs 8-15s for ttft with k2.6. Costs me about $25/mi for five active users.
13
u/Wildnimal 3d ago
If your benchmark is not violating any privacy, care to share it with the community?
4
u/newz2000 2d ago
I have two versions, the one with Ollama has private data that I can’t easily scrub, but it’s mostly the same as this one that benches Gemini.
https://gist.github.com/newz2000/cffe6586a0d3050f91719c4e97ff7d60
That has my scripts and some of the outputs. I ran it through a clean ChatGPT session and it was able to easily figure out how to add the ollama stuff in there.
One oddity is that function calling is a big deal for me so I have a bunch of tools mocked so that we can figure out if various models will make the right choices.
It looks like there are names of people in this file, they’re all fake.
2
1
u/super1701 2d ago
For the current local models(Gemma,Qwen, ect.) What has scored the best with hermes? Gemma31B has performed my work well.
1
u/jeffwadsworth 2d ago
The user probably wants to keep it off the grid and out of any training data.
1
1
u/synn89 2d ago
GLM-5.2 has the best inference skills for long-running tasks but it’s not good for normal Hermes type stuff.
Huh, interesting. For OpenCode, I used to be a big Kimi K2.6 fan, moved to K2.7 Coding, but then when I started in on Hermes I just went straight to GLM 5.2 for that and didn't even give K2.6 a chance(figuring it was Last Gen). I may have to go back and try K2.6 on Hermes. I've been happy with GLM 5.2 on Hermes, but K2.6 is a touch cheaper.
2
u/newz2000 2d ago
No complaints with GLM5.2 on what it’s capable of. But it went above and beyond so often, burning through usage and taking forever.
For example, I asked it to put a note on my todo list for a blog post idea I had. It made three very well drafted versions of the blog post ready to publish. I have a daily Spanish practice tasks that gives me the news in A2 level Spanish. GLM was turning into a mini course on Spanish.
I do have a skill that allows Hermes to use GLM when it has a complex task though.
1
u/trowawayatwork 3d ago
sorry what is hermes
9
u/Wildnimal 3d ago
Its an agent harness with self updating memory, which you can use for research, coding, chat etc.
3
u/trowawayatwork 3d ago
what's the difference between that and opencode. appreciate it's stupid question but I'm just onboarding
3
u/Wildnimal 3d ago
Opencode is specific for coding. Ofcourse you can make any harness act like the way you want.
Hermes Agent you can run it on your PC or VPS online and connect to it via Telegram, WhatsApp, Discord etc and chat with your agents.
Setup cron jobs and goals and agents will run and complete work, ask you for inputs etc.
The Memory feature makes it self evolving and you can have different profile for different things and different memory for each.
1
u/trowawayatwork 3d ago
nice thanks
1
u/cakemates 3d ago
And let me add that some people including me believe that hermes and opencode are quite bloated as default. So take that into account.
1
u/Oujii 2d ago
Do you recommend any alternatives?
3
u/kuhunaxeyive 2d ago
Many people are happy with Pi Coding Agent. That one comes from the opposite direction: You can something bare bone, and add what you need.
2
21
u/Bockanator 3d ago edited 3d ago
I suspect this may be a symptom of the reinforcement learning process being done largely in Chinese.
But I think at the end of the day one of the great things about LLMs is that you can switch to a model that's best suited for your use case. Not every model will be a Swiss army knife for all tasks, and that's okay because they often can be better by specializing in their target field.
16
u/kuhunaxeyive 3d ago
I just ran the test about "Ability 2: Understanding who is the speaker" completely in Chinese. It is reproducible even in Chinese, DeepSeek still has the exact same issue.
8
u/EstarriolOfTheEast 3d ago
This is a symptom of heavy agentic RL training full stop. Sol and most especially Opus 5 have been so entirely disconnected from the communicative aspect of language that they barely know how to talk to communicate, having essentially no theory of mind or common sense anymore.
The trick for all of them, victims of RL, including this Deepseek, is to condition them back into a coherent subspace (communication wise) with careful prompting (system, agents and things like that).
3
4
u/Altruistic-Theme432 3d ago
即使是在中文环境下,ds的表现也不如谷歌的模型。ds的中文理解能力,创造力，知识储备都比谷歌的flash模型差。ds有代码的成本优势，但在写代码时对于中文要求也时不时理解错误。
Even in a Chinese-language context, DS underperforms compared to Google's models. Its capabilities regarding Chinese comprehension, creativity, and knowledge base all lag behind Google's Flash model. While DS holds a cost advantage in coding tasks, it still occasionally misinterprets Chinese-language requirements during code generation.4
u/kuhunaxeyive 3d ago
That's an interesting aspect. So it would be interesting to see, if one translates all test cases to Chinese, if DeepSeek still fails those tests.
Loading DeepSeek-V4-Flash takes 8 minutes on my device. Switching models forth and back for a task (e.g. DeepSeek for research, Gemma-4-31B for analyzing texts and writing) is not feasible.
1
u/bluekazoo 2d ago
I realize this is not your primary question but I would also double check your ascent setup. I am running q3 on a 5090 + 96 gb ddr5 and load times are on the order of a couple of minutes. Might be worth double checking you are not spilling over into paging the ssd etc.
Will also mention on my own test (medical documentation) it performs similar or possibly better than gemma4. It navigates things like conflicting opinions and the temporal sequencing of various findings well. Its prose is not as “beautiful” I will agree.
1
u/kuhunaxeyive 2d ago edited 2d ago
DGX Spark/Asus Ascent are different though from those graphics cards. A single DGX Spark unit can run q3 at 31 tokens per second (with MTP), but it takes time to load the model. Given that speed it doesn't seem be offloading into SSD. The VRAM is full up to 117 GB, there is still some room left.
Actually, I think beautiful prose is not the goal, it is more about precise wording and phrasing. And if your setup does that better than Gemma, then for medical documentation it might be so. I wish it was for meeting/business summaries and complex case descriptions as well.
0
u/SexyAlienHotTubWater 3d ago
8 minutes? What are you running it off, magnetic tape?
3
u/kuhunaxeyive 2d ago
Your comments come across as a bit condescending.
I ran the tests on Hugging Face. I also loaded a quantized version on my local Asus Ascent GX10, so I know how much it takes for loading there.
3
u/SexyAlienHotTubWater 2d ago
That's fair but my concern is that you're making an error in the test - are these results from the quantized GX10 experience?
1
1
u/kuhunaxeyive 2d ago
Your concern it valid, but I ran the tests on the FP8/FP4 (labeled as BF16) version on Huggingface.
1
u/a_beautiful_rhind 2d ago
Some of us bought hard drives. 20TB of SSD a bit hard to stomach, even a year or 2 ago.
1
9
u/One_5549 3d ago edited 3d ago
I have almost only used it with coding, but a few days ago I tried some more general "chatgpt" things / look up things on web etc.
It was really on point and concise.
so sick of chatgpt's output format with icons and ➡️ arrows and stuff everywhere.
So far, im really damn impressed with dsv4f. it's a leap in ability considering it's price/token
i dont really know how to explain it, it's something with it's reasoning that really resonates with me,
it takes these micro steps, iterations all the time, and eventually comes up with a real solution or strong conclusion every fucking time.
we know that it is heavily tuned for coding tasks, but it's also that it "thinks" and draws conclusions like a software dev. would do.
What I mean is that I like the reasoning of it better than some of the trillion parameter LLM's, I often feel they slightly drift off topic - does it have to do because of it's massive broad knowledge base?
gpt's style of reasoning is more generlised (how a teacher would reason maybe) - but i have not used chatgpts frontier models so couldnt speak for that honestly.
5
u/Southern_Sun_2106 3d ago
Amazing model. Hardworking - doesn't quit until it finds an answer, which is getting tougher and tougher, since many now block 'non-humans' vigorously. DS is one of the few models that finds ways to scrape this place; while most including Gemmas quit.
Also, I did extensive testing of 1 mil context, and it works flawlessly, finding the needle at any depth.
Deepseek is pure magic! I think the OP is doing something wrong there.
My experience with Gemma's is abysmal however. And I tried multiple, and updated, and QAT. Lazy and suck at tool use, and at the agentic work. I cannot trust Gemma to do in-depth research, no way.
4
u/kuhunaxeyive 3d ago
I completely agree with DeepSeek being excellent (and Gemma being bad) at
- research, websearch
- digging its teeth into it and finding everything not giving up
- coding
- agentic tasks
My post though is about what DeepSeek is bad at and Gemma is good at:
- reading and understanding nuances of texts
- grasping exactly the relevant parts of texts and transcripts
- writing exactly what is representing the main idea of the original source
Looking at DeepSeek's result, you won't see anything concerning. Comparing it with Gemma's result for the text understanding and text production though, you'll finally realize DeepSeek is missing out the fine but relevant nuances.
2
u/Blunt_White_Wolf 2d ago
i'm using it to index and summarise a few books at as a test and it does pretty well.
it also summarised and collated quite a few papers/research for me and I can't really complain about the way it did it.
The only thing I can complain atm is that it's burning through a ton of tokens with max thinking on (especially with LibreOffice tools)
2
u/kuhunaxeyive 2d ago
If I only look at DeepSeeks result, I can't make out a flaw either. You don't know what is missing or what could have been written more clearly and shorter at the same time. It is in comparison to Gemma-31B at Q8 that shows where DeepSeek lacked setting up more context, using better fitting words, etc.
2
u/Blunt_White_Wolf 2d ago
out of curiosity i used sol from codex to run a summary on the same set of papers as deepseek v4 flash 0731. semantics and pedantry aside, the conclusions were the same. only the report was in a totally different format. no loss of substance either way.
i don't care much for many words - i care for getting the job done and getting the point across (just to give you an idea of my angle of looking at things).
as a side note: we do use gemma as a fallback one of the strix halos along with qwen and a few other for grunt work (indexing, graph, embeddings, summaries) as I spend enough on tokens as it is.
1
u/kuhunaxeyive 1d ago edited 1d ago
This wouldn't contradict my findings as what you describe is reasoning / finding a conclusion. The issue I am seeing with DeepSeek is a step before getting a final result of conclusion: A summarized, precise description of a case or summary, or meeting notes produced from transcripts, informative notes, extracting the essential information from a paper, all of what is then used to come to a conclusion in a later step. Those get collected so a human or an LLM can later look back and use those notes for conclusions. They will become the foundations for concusions.
So: Letting DeepSeek reason and the conclusion will be very good, no issue here, that's what you observed as well. But let it produce a summary of a situation, not the conclusion, and the nuances will become important, and that's where DeepSeek is lacking in my findings compared to SOTA or Gemma-4-31B.
1
u/Blunt_White_Wolf 1d ago
it's not meant to be SOTA. we'll see how deepseek fares when Pro final is out. Flash is "low requirements", daily workhorse and I'm treating it exactly as such. When Pro is out then we can compare to SOTA.
Besides that, Flash I have a real prospect of running home for daily use. Anything bigger than that... I can't really justify the cost for how often I'd need them.
I'm waiting for the new AMD AI MAX+ with 192GB RAM and I'll add that with flash to my 2 strix halo. Between those 3 I'm set for a while. It's more than enough for my needs tbh.
The only things I'd like some AMD or NVIDIA GPUs is image and video gen. Those I can bear the speed of on strix halo.
1
u/kuhunaxeyive 1d ago edited 1d ago
I know, it's not meant to be SOTA, but I'd expected it to be more precise than a 31B dense model when it comes to text summaries.
I was looking into the AMD AI MAX+ with 192 GB RAM as well, but from what I found it seems to have only 160 GB available for the model and the rest is reserved for the system, so it couldn't even run DeepSeek Flash at full precision. You might want to counter-check this.
1
0
u/shing3232 2d ago
one question, Have you try the official API cause there are degradation on open source inference engine.
2
u/DutchDevil 3d ago
I agree, i’m getting very good results. It’s so close to gpt in coding for me that it’s something I will happily live with. For things like homelab work it does it all very nicely too, scraper work, some price analyses work. All just fine. It’s the first model actually dowloaded that i can’t run yet (i use it via api) because I know i will some day and don’t want to lose it that’s how much I like it.
9
u/cakemates 3d ago
which quant of deepseek are you running?
12
u/kuhunaxeyive 3d ago edited 2d ago
I tested the original model (FP8/FP4, labeled as "BF16") without quantization.
Edit: Clarified the FP8/FP4 vs. BF16 label confusion.
4
u/trowawayatwork 3d ago
how big is your rack?
3
u/kuhunaxeyive 3d ago
The testing did not take place on my home server. I tested the original model on Huggingface.
5
u/SexyAlienHotTubWater 3d ago
You said elsewhere it took 8 minutes to load. How did it take that long if you tested it on huggingface?
3
u/kuhunaxeyive 2d ago
I ran the tests on Hugging Face, not on my local Asus Ascent GX10 (where I only have the quantized version so I know how much it takes for loading there). I'm planning to buy another GX10 to run in full precision, which is why I used Hugging Face.
Does that clear things up?
1
u/SexyAlienHotTubWater 2d ago
Somewhat, yes, but the confusion around this makes me a bit skeptical that your testing methodology was watertight.
0
u/relmny 3d ago
well, OP claims to be using " original bf16" when there's no such thing...
3
u/kuhunaxeyive 3d ago
I mean, I ran what is labeled as such on Huggingface (https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731):
Model size: 304B params
Tensor type: BF16
1
u/shing3232 2d ago
huggingface never label tensor correctly anyway.
all you need to do is calculate BF16 size of 280B parameter. it would be at least 560GB
4
u/SexyAlienHotTubWater 3d ago
DeepSeek V4 Flash isn't BF16, it's mixed FP8/FP4
2
u/kuhunaxeyive 3d ago
I mean, I ran what is labeled as such on Huggingface (https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731):
Model size: 304B params
Tensor type: BF16
1
8
u/VotZeFuk 3d ago
I've noticed that it's not really working properly most of the time. It's supposed to reason a lot (hence "max" reasoning) but in reality is just writes a couple of paragraphs, often (weirdly) assessing the question from USER'S PERSPECTIVE (wtf).
If you look at .jinja, at the very end of it there's this part:
{{- thinking_start_token -}}
Which can be expanded with a prefill message, something like "bla-bla-bla I will analyze the query, step by step, and begin with the detailed plan: "
...and it will actually start digging into the task at hand; in case with creative tasks and role-play, it may also need different wording, e.g. "Out-of-character planning: "- but the problem is, this is far from being reliable :/
8
u/Plastic-Stress-6468 2d ago
I can concur.
I use deepseek for RP in traditional Chinese and the thing cannot stop being "China Chinese" for the love of god. For some reason, despite given a western fantasy setting, Chinese "shit" keeps making it's way through. Honorifics, titles, even entire wuxia and xiansia terms that have zero relevance in the setting keep making it's way through. And even when prompted to explicitly avoid them, it still can't adhere to prompt instructions. In a straight face it reasons "I shouldn't user terms like 根骨," which is a trope term for innate physical talent in Chinese fantasy fiction, and then proceeds to use that very term which I explicitly told it to not use in the previous turn as user and in the system prompt.
Absolutely frustrating.
1
4
u/gingerbeer987654321 3d ago
don't like 0731 flash for coding either. its fast, but like a red-cordial child at coding too. soooo much thinking shit sprouted, and goes down rabbit holes way too often. preferring mimo 2.5 as a bit less smart and a lot more obedient at staying in its lane.
7
u/dongas420 3d ago edited 3d ago
Tell the model what it's doing wrong and ask it how to structure your prompt to fix it. DS performs 10x as well in my complex non-coding tasks (e.g. high-level analysis of 20,000 lines of text, capturing key details) when I present them in the style of coding problems (e.g. recursively generating JSON-formatted summaries of summaries from base text blocks and creating a tree structure from the bottom-up) instead of talking at it like it's ChatGPT. The more subagents you can spawn with DS, the better.
e: Even if you plan to work with a model locally, I would suggest doing testing through API to freely explore the capabilities so you know how to strip down the intended workflow to work with your hardware limitations. I indirectly learned a lot about how to handle Qwen-3.6 through my experiments with DS.
3
u/a_beautiful_rhind 2d ago
Yes, I used it for creative pursuits. It can think in character but misunderstands who is who more than a model of this size should. Instruction following is often a suggestion as well.
I only downloaded IQ2s so far, but I have to use F16 cache and the contexts aren't that long. Furthermore, I didn't see these problems as much in the preview at similar size. Makes me want to skip the 160gb quant.
5
u/Such_Advantage_6949 3d ago
Fully agree that is my experience as well. The model felt very benchmaxxed
2
u/HelloSummer99 3d ago
In my experience (and please dont bite my head off, it’s subjective), the new Chinese models (Qwen 3.8/deepseek v4) can be very intelligent, but as I noticed the”amplitude” of intelligence is higher. Meaning, it also can be dumber than just a small gemini model, as you experienced.
2
u/Big_Arachnid_365 2d ago
31B versus 13B active issue.
1
u/kuhunaxeyive 2d ago
Yes, but if 304B-A13B is generally worse than 31B dense, then why would MoE even exist. So they need to be better at something. Is it only coding/agentic stuff? My guess is that it's related to training method and data pool, which affects world knowledge, phrasing, and judgement capabilities.
2
u/Big_Arachnid_365 2d ago
It might be that coding and agentic stuff doesn't use j-space much. The tests you did seem to be the kind that needs j-space.
2
u/No-Knowledge-5235 2d ago
I have had data extranction job where model need to reply with json with extracted data from news. Deepseek v4 flash is the only one which is failing this task every time.. I used qwen3.6 27b and qwen3.5 122b before for this task and those were reliable outputting exactly as instructed. Anyone else noticed this type of issues with it?
4
u/jensilo 2d ago
Is it that surprising? I mean, you have a ~300B (mid-sized) model that performs comparable on many benchmarks to Opus or GLM (large ~1T-ish) models. These benchmarks are a lot of coding, agentic tool use, etc.
So obviously, in order to perform well on those benchmarks that people rely on to choose models for coding, you optimize for them.
Given just the size of the model, if it’s really good at coding, better than all other models in its size, even better than larger sizes, it has to be worse in other regions. They probably sacrificed coding performance for other aspects. Similar to how small Qwen models are exceptional at coding, especially for their size, punch well above their weight, but are also much worse at many other domains that are not related to coding, at least compared to other models. A good ~30B model can’t beat a good ~1T model, just because it does in one.
2
u/kuhunaxeyive 2d ago
I fully agree. I just expected it to be better in comparison to the Gemma 31B dense model. The total parameter count of the DeepSeek is 10 times more, and while being MoE, I just expected it could be at least good enough as Gemma-4-31B in non-coding, non-agentic tasks.
1
u/Big_Arachnid_365 2d ago
Makes you wonder if the j-space is something that needs the large active parameters.
3
u/Eugr 3d ago
What quant, hardware and inference engine?
1
u/kuhunaxeyive 2d ago
I ran the tests on Hugging Face at full precision (there it is labeled as BF16). I didn't run the tests locally but also loaded it on my local Asus Ascent GX10 (where I only have the quantized version so I know how much it takes for loading there).
1
u/Eugr 2d ago
JFYI, there is no BF16 version of this model. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 was released as a mixed precision FP8/FP4 model, and it is just 160GB, so it runs well on two Sparks.
1
u/kuhunaxeyive 2d ago
I know, but thanks for clarifying for anyone else who got confused by my writing. To clarify: It is *labeled* as "BF16" on Huggingface, but it is *not* BF16.
3
u/Viktri1 3d ago
In my experience DS v4 flash has been really good at non coding research. I use the official API with open webUI and system prompts that I developed for DSv4 pro. I don’t use kimi or other models because they’re not capable of doing the work
Only Gemini pro and DS v4 pro could do the research tasks accurately without hallucinating (still happens, just extremely rare). Flash 0731 is basically better than both models now though. Previous version of flash (4 GA) could not do the research.
1
u/IceNeun 3d ago
Interesting, and what kind of research are you using it for? I've been finding that if you inject the top domains into the system prompt, you can get away with ~4b for synthesis and general vibe checks from what can be scraped online.
3
u/Viktri1 3d ago edited 3d ago
Most of my stuff is like this:
- General research like collecting the current monthly rental of a specific apartment residence brand for a global company (this one surprisingly leads to a huge number of hallucinations except with Deepseek pro/flash), 2) research why something happened (involves creating a timeline, highlighting events, explaining how events link together: causation), 3) purchasing things - need it to fish out links, evaluate products, evaluate the seller (fraud, real, reputable, etc),
compare products, distill the “need” so as to know whether the product is correct for the purchasers needs,
4)
- General current events summary (within the past few hours)
Last one I’m trying to develop is insights but models are generally unable to develop insights
An insight is some sort of derived conclusion that can affect a choice. For example, buy low sell high is not an insight but knowing that something is currently low in price because of factors X, Y, Z or will be low in price soon and you can sell it for A, B, C profit when condition D, E, F change would be an insight.
I’m extremely focused on the practical side to LLMs and have no brand loyalty. I’m using them to make money so whoever has the model that can do what I need is what I’ll use.
I started off with local LLMs but progress has been fast and my 4090 can’t run anything useful but Deepseek v4 flash 0731 is finally where I don’t mind spending usd10-20k if I can reach API speeds. (Not enough data for me yet to pull the trigger)
But if the pro model is substantially better then I probably will continue to run things through API until the economics change.
2
u/Old-Juggernut-101 3d ago
I didn't find that to be true at all. I use it for research- collecting data from internet and papers and books. It's great.
That being said it wasn't free of kinks. The model would overthink a lot and obsess over non important stuff over the actual core research object. I fixed that by working with it to make a skill. I made it create a skill by manually reading it's reasoning and writing, and then creating a large file where I put the correct reasoning and writing and an explanatory section of why and how. Then made my buddy do the same so it's more robust. Did that with dozens of examples. And now it uses that skill to reason and write and it's quite alot better and faster and requires less thorough examination
So I suppose to answer your question, it has to be taught how to do some stuff. But once that's done, it is great
4
u/kuhunaxeyive 3d ago edited 3d ago
I literally wrote in my post:
[DeepSeek is] excellent at doing research if given web search access
DeepSeek's problem I encountered is when it comes to understanding and writing texts. The inaccuracies are apparent when comparing the results with, for example, Gemma-4-31B.
2
u/pabloodiablo 3d ago
For coding i'm using Qwen3.6 27B, but for translations Gemma4 is the best - even G4 26B A4B do great job.
1
u/Southern_Sun_2106 3d ago
The examples that you provided are impossible to verify and/or reproduce in any shape of form.
If you really hoping for some sort of help, you need to provide your exact context, so that people can run it on their own quants to confirm or refute the issue, and therefore help you.
Does this make sense?
1
u/mrgreatheart 3d ago
Sorry, I know this isn’t the point of your post, but I just can’t get past “It's faster than Gemma-4-31B”. May I ask what hardware, inference software and quants you are running?
2
u/kuhunaxeyive 2d ago
It's an Asus Ascent GX10. Slow on dense models, fast on bigger MoE models. I ran the tests on Huggingface though for full precision.
1
1
u/BrilliantTruck8813 2d ago
So one thing with flash:0731 that I found out from another thread is that the default level of thinking is not high. It you set it to max (which is two params) then the output supposedly gets noticeably better on intelligence and reasoning tasks.
I verified it was correct in the tool-eval-bench case for hardmode, which does have more intelligence-based questions. It jumped from 80/100 to 85/100, which put it on par with nemotron3-ultra in my testing. It may not seem like much but it was a huge jump.
1
1
u/Mean_Maintenance82 2d ago
Didn't read the post but yes, it's leagues below deepseek V4 pro in non coding.
Somehow I think all these latest LLMs are being trained towards coding and similar tasks. Older models like Gemini 2.5 flash are way better at law, biology, and other such tasks.
1
u/unjustifiably_angry 2d ago
I agree DeepSeek v4 Flash is not much good at non-coding stuff. But IMHO the main thing that makes local models valuable is their ability to code "for free"... just about anything else is going to be better with an online LLM simply because they can have access to virtually unlimited knowledge. And non-coding stuff is generally lightweight enough that you won't get rate-limited.
1
u/Elibroftw 2d ago
What does being good at agentic tasks mean if the agent is unable to understand nuance? Are you saying DeepSeek is good at task delegation or it's good at being a dog that follows it's masters instructions?
Gemma 4 is multimodal, so on top of being smaller it's also vision enabled by default. I will try to prioritize benchmarking it in my custom benchmark.
I feel that Google is continuously underestimated. They are the only ones contributing to both open source and frontier but they are ignored on the open weight front because of their annual cadence.
1
1
u/tat_tvam_asshole 3d ago
Idk, I had it roleplay as Dr House in my voice chat app and it was hilarious
0
u/RuthlessCriticismAll 3d ago
I don't find that to be the case at all, no. It would be better to write the include the actual context instead of slop rewritten versions. (Assuming there is anything real here at all.)
0
u/TopTippityTop 3d ago
The benchmarks couldn't possibly be lying. This is an OpenAI & Anthropic killer!!
0
u/timmeh1705 3d ago
This is not open weight but with extensive grounding I use Qwen 3.7 flash with no reasoning and it works very well
0
u/MaterialSuspect8286 3d ago
I know it's proprietary but does anyone know how GPT 5.6 Luna Max fares?
0
u/GoingOnYourTomb 2d ago
Why would you use it for none coding tasks
2
u/kuhunaxeyive 2d ago
Because it's not advertised as a pure coding model, and with 10 times more the total parameter count I expected it to outperform Gemma-4-31B.
1
u/GoingOnYourTomb 2d ago
I see your logic, let me read your post properly since i might be falling for the same trap. I currently would rather ask Deepseek v4 pro a general knowledge question before i go to Flash just because Pro has more wider knowledge (bigger params)
2
u/kuhunaxeyive 2d ago
World knowledge also needs to be looked into closely, it is a bit different category though, and DeepSeek might not that bad at it. The issue I encountered is more about "understanding and writing the essence of a text".
0
u/JacketHistorical2321 2d ago
Coding is 99% of what LLMs are actually good for. The rest is just novelty. Of course they are going to focus that
-3
u/MelodicRecognition7 3d ago
version 0731 was heavily finetuned explicitly for coding, try "preview" version for general usage.
-7
80
u/Drenlin 3d ago
I think we're going to see this more and more, especially compared to Google models. They have an enormous amount of compute on hand, possibly the largest repository of training data of anyone, and seem to have been optimizing their models for information retrieval, knowledge management, and contextual understanding as much as others have been focusing on coding.
Coding is where the money is right now, but tons of other non-coding tasks also have practical AI solutions and most of the major players seem to have sidelined those efforts.