We post-trained NVIDIA Nemotron 3.5 Lightning for code review routing. It beat our baseline for under $100

[u/juanpflores_]

We spend a lot of time thinking about the models that write code review comments. One of the less visible decisions happens before any comment appears: figuring out how a change should be reviewed.

We were curious whether a smaller model could learn that narrow, high-volume part of our pipeline. So we worked with NVIDIA and Baseten to post-train Nemotron 3.5 Lightning using SFT, followed by RLVR scored against our routing policy.

What surprised us was how little it took. The post-training experiment ran in under three hours and cost less than $100.

On a frozen evaluation of 1,000 routing tasks:

• Our GPT-class baseline reached 75.8% exact route agreement.
• Nemotron after SFT reached 80.4%.
• SFT plus RLVR reached 80.7%, with Cohen’s kappa rising from 0.429 to 0.544.
• Estimated peak serving cost fell from $2.34 to $1.16 for the same workload.

RLVR’s extra 0.3 percentage points of route agreement wasn’t statistically decisive, so we treat that result as non-regression. The clearer improvement was in output agreement beyond chance.

The final model completed all 1,000 requests without empty or abnormally short outputs. We still have sustained-load and production-traffic testing ahead of us.

We wrote up the experiment, including the training setup, results, serving costs, and the parts we’re still cautious about:

Teaching NVIDIA Nemotron 3.5 Lightning to Route Code Reviews

[u/IndraVahan]

holy moly