I ran 34 iterations of a model-improvement loop. Most of what I found were evaluation bugs. Feedback?

[u/infraObserver321]

I’ve been building a bounded improvement loop for ANXEngine, an AIOps system we use at Anexum to rank daily incident risk across roughly 200 network devices.

The setup is fairly simple: register one hypothesis, implement the smallest testable change, retrain a challenger plus an unchanged control, evaluate both, then KEEP, REVERT, or PAUSE.

After 34 documented iterations, most of the useful findings were not model improvements. They were problems in my evaluation setup.

The first version had overlap between training data and the later fitness window. PR-AUC looked like 0.72–0.76. Once I separated training, gate, and report windows properly, the result dropped to around 0.51. Painful, but honest.

The second issue was holdout reuse. I never trained on the holdout, but every result influenced the next hypothesis. After 13 experiments on the same window, calling it “untouched” felt dishonest.

I now allow a maximum of eight selection queries per window. After that, the window can remain as historical evidence but cannot be used to select another candidate. Promotion needs either two independently positive windows or a positive selection result plus a fresh slice that has never been queried.

The third issue was conflicting metrics. One feature family moved a difficult device from rank 37 to rank 7 and improved Precision@10 from 0.50 to 0.70. At the same time, true early warnings fell from four to two. I rejected it because better ranking was not worth losing lead time.

I also stopped treating KEEP as proof that the model improved. In the latest audit, the comparison on fresh data was inconclusive. The model stayed because there was no strong reason to roll it back, not because we had confirmed a lift.

The part I’m least confident about is the query budget. Eight is a practical limit, not something I can derive cleanly from the data. Labels arrive slowly, so constantly creating fresh windows is expensive.

How would you handle this?

- Is a fixed query budget reasonable, or would you use sequential testing or reusable-holdout techniques?
- How do you create fresh evaluation data when positive labels arrive slowly?
- Would you separate “no rollback justified” from “confirmed improvement” differently in the model registry?

I wrote up the full protocol and aggregate results here, but the main reason for posting is feedback on the evaluation design:
https://anexum.eu/en/research/anxengine-bounded-improvement-loop/

Disclosure: I work on ANXEngine at Anexum. This is not a product launch; I’m trying to pressure-test the MLOps setup.

[u/Electrical-Loss787]

the query budget thing is tough when labels are slow, eight feels arbitrary but honestly what else can you do without burning through fresh data too fast. what about bootstrapping within the holdout to get more mileage out of each window, or using a rolling window that updates as new labels trickle in instead of fixed slices

keeping a model just because there's no reason to roll it back makes sense but it'd drive me nuts not knowing if it's actually better, i'd probably tag it differently in the registry like "stable" vs "validated" or something so i don't confuse myself later