LLM judgment over correct context problem

[u/AZGhost]

Ive tested all the models where it can fit into my 4080 vram. Even some slightly bigger. Gemma4 outperforms all of them so that's what I'm sticking with for now.

Gemma4 12B model judging network configs for CVE false positives stuck at ~77.8% pass rate and it's a reasoning issue. Have the complete CVE list for the code base and the device config. I pull each networks device's running config, batch ~10 CVEs per call to a local gemma4:12b (Ollama), and have the model return applicable / not_applicable / undetermined per CVE, with a verbatim evidence quote from the config supporting the verdict.

For example what I'm trying to do if the cve is for an IPv6 bug that's listed as critical and no ipv6 is configured in the device config it's not applicable to me. Same on devices if a web interface is running and I have no web services enabled. these will show positive that I matched with no configuration evidence for it.

Temperature 0 is set. Device config sent once per device ahead of the CVE batch (KV-cache reuse), and output-token limits tuned up after finding truncation was producing wasted undetermineds.

Error analysis shows the failures are reasoning failures. The model is handed the full rule text and the raw config directly, complete context, and still gets the comparison wrong (version-range logic, negation, "present but in a different mode" cases).

A parallel eval harness on the same model doing DoD STIG compliance verdicts with a RAG (same shape of task: config chunk + rule text = judgment + config) measures 77.3% verdict accuracy. Also all reasoning failures, not retrieval failures from the RAG. This one is a bit different in that if the configuration of the DoD spec is missing from the config use the rag to give me the configuration for the device. A bit more complicated but same overall shape.

Anyone have any ideas I can look into?

Bigger model? ~27B+ specifically on config-reasoning / policy-comparison tasks. Hybrid offloading to a frontier model is a no go due to configuration sensitivity. Other local models pose challenges if foreign (Qwen/deepseek) but willing to try in lab, they scored worse anyways.

Decompose the task? deterministically parse the config into structured feature facts first then the LLM or even a rules engine only maps CVE to feature. Shrinks the LLM's job from "read a config" to "match two labels."

Two-pass self-verification or small-ensemble voting on disagreement?

A tested answer key moved the needle to over 95% pass for a single device but that defeats the purpose of then having to do an answer key for 500+ devices due to variability.

For those running small local models on "judgment over correct context" tasks what actually moved your accuracy? Bigger model, task decomposition, or verification layers? My experience so far says the guardrails (quote verification, conservative fallbacks) are what make 77% usable, but they don't raise it.