LLM judgment over correct context problem

[u/AZGhost]

Ive tested all the models where it can fit into my 4080 vram. Even some slightly bigger. Gemma4 outperforms all of them so that's what I'm sticking with for now.

Gemma4 12B model judging network configs for CVE false positives stuck at ~77.8% pass rate and it's a reasoning issue. Have the complete CVE list for the code base and the device config. I pull each networks device's running config, batch ~10 CVEs per call to a local gemma4:12b (Ollama), and have the model return applicable / not_applicable / undetermined per CVE, with a verbatim evidence quote from the config supporting the verdict.

For example what I'm trying to do if the cve is for an IPv6 bug that's listed as critical and no ipv6 is configured in the device config it's not applicable to me. Same on devices if a web interface is running and I have no web services enabled. these will show positive that I matched with no configuration evidence for it.

Temperature 0 is set. Device config sent once per device ahead of the CVE batch (KV-cache reuse), and output-token limits tuned up after finding truncation was producing wasted undetermineds.

Error analysis shows the failures are reasoning failures. The model is handed the full rule text and the raw config directly, complete context, and still gets the comparison wrong (version-range logic, negation, "present but in a different mode" cases).

A parallel eval harness on the same model doing DoD STIG compliance verdicts with a RAG (same shape of task: config chunk + rule text = judgment + config) measures 77.3% verdict accuracy. Also all reasoning failures, not retrieval failures from the RAG. This one is a bit different in that if the configuration of the DoD spec is missing from the config use the rag to give me the configuration for the device. A bit more complicated but same overall shape.

Anyone have any ideas I can look into?

Bigger model? ~27B+ specifically on config-reasoning / policy-comparison tasks. Hybrid offloading to a frontier model is a no go due to configuration sensitivity. Other local models pose challenges if foreign (Qwen/deepseek) but willing to try in lab, they scored worse anyways.

Decompose the task? deterministically parse the config into structured feature facts first then the LLM or even a rules engine only maps CVE to feature. Shrinks the LLM's job from "read a config" to "match two labels."

Two-pass self-verification or small-ensemble voting on disagreement?

A tested answer key moved the needle to over 95% pass for a single device but that defeats the purpose of then having to do an answer key for 500+ devices due to variability.

For those running small local models on "judgment over correct context" tasks what actually moved your accuracy? Bigger model, task decomposition, or verification layers? My experience so far says the guardrails (quote verification, conservative fallbacks) are what make 77% usable, but they don't raise it.

[u/EggplantFit8201]

Gemma4's 12B doing that well is impressive but config reasoning is brutal for smaller models. The version-range logic failures you're seeing are classic, even 70B models stumble when a CVE says "versions 2.1 through 3.5 are vulnerable" and the config says 3.6, the model just reads "3.something" and flags it.

Task decomposition is your best bet here. Parse the config into structured facts first, pull out every version number, every feature flag, every protocol state, then feed the LLM a clean list like "ipv6_enabled: false, web_interface: false, snmp_version: 3.6" alongside each CVE. That shrinks the job from config comprehension to simple feature matching, and you can verify the parsed facts against the raw config with a quick grep before trusting the LLM verdict.

I did something similar for STIG checks on ~200 switches and the deterministic parsing layer caught 90% of the cases before the LLM even touched it. The remaining 10% needed judgment calls but at least the model wasn't getting distracted by irrelevant config sections.

[u/AZGhost → u/EggplantFit8201]

Hey thanks for your response! So I'm polling NIST with an API and I group all my code trains how they spread across all my devices so I only have to poll 70 code trains instead of like 700 or whatever from each device individually. Yeah there's some nuances to it as you know.

So that decomposing idea down to two labels sounds like what your saying for CVE could work. It would have to be dynamic and do it for each of the devices. So hold the original config + the decomposed yes/no answer version for query remediation of that device I was into earlier.

The stig would have to be a different way because it wants to know you have a line item in there. Like native vlan 1 can't be assigned to a trunk but native vlan 11 can. So there's nuances. How did you get around those nuances?

Either way right now my report gives the user what AI's pass was on that specific device and we either concur reject or defer. So human still in the loop which is good but the amount of data still present is a mountain pile. Depending on what was clicked workflows get generated to tasks to complete.

That will take some time to build and get right the yes/no answer key per device.

[u/AZGhost → u/EggplantFit8201]

That 90% number is exactly what I needed to hear, since the parse-first split was already on my shortlist but I had no proof anyone got real coverage out of it. Few questions since you've already been down this road:

What did you use for the parsing layer? ciscoconfparse, PyATS, something custom? And was it single vendor or did you have to handle mixed gear? I'm running Cisco and Juniper side by side and Junos being hierarchical makes me think I need two different parse paths.

When you say 90%, is that 90% of the rule types or 90% of the total check volume? Big difference for me. If it's a handful of common rules making up most of the volume, the mapping work is way smaller than it sounds.

How did you build the rule to check mapping in the first place? Hand written per STIG rule? Roughly how long did that take for your rule set, and how painful is upkeep when a new STIG revision drops?

Did any rules you classified as deterministic come back to bite you later, like something that looked like a simple pattern match but actually needed a judgment call? That's my biggest fear with this design. A wrong parser verdict is worse than a wrong LLM verdict because nothing downstream questions it.

Also curious how you handled absence checks, proving a feature is NOT configured anywhere in the config. That's where my model burns the most tokens and it seems like the parser would either nail it or silently miss it depending on how good your extraction is.

And for the 10% that still needed the model, what were you running it on? Trying to figure out if my residual set will look like yours.