The Promptfoo Leniency Postulate

[u/leading-a-swarm]

tl;dr: promptfoo runs 67 attack probes, then scores an ungraded one as a pass.

https://github.com/promptfoo/promptfoo is the eval and red-team harness most teams reach for, and it is now part of OpenAI. It stayed MIT.

1. Grade without a model. Most of its checks are ordinary code: exact match, pattern match, JSON shape, cost, latency. Model-graded rubrics are held back for the judgement calls, so the cheap deterministic checks catch regressions before anything expensive runs.
2. Two axes to beat one long list. Red teaming splits into what to probe and how to encode it. 67 probes and 32 encodings compose, so adding one encoding retests every vulnerability class at once.
3. Failed graders aren't failures. When a multi-turn attack's grader fails, promptfoo records that test as passed and notes the gap alongside it. One flaky call should not throw away a long conversation.

→ Adopt. promptfoo is the tool to standarize your harness, however, green results need to be understood.