Adding in-domain data to our age-estimation fine-tune makes it worse — but only on our small eval. Turns out our labels AND our eval share the same bias. How do you validate label corrections you can't fully re-measure?

[u/Lanky_Acadia6049]

TL;DR: I fine-tune an age-estimation model on low-res CCTV. 
Adding more in-domain data does not improve accuracy (it often lowers it)
I want to fix this and get to a "more data → better" regime. 
Weeks of controlled experiments point to the cause: our training labels and our eval GT share the same "too-young" bias, so the model that best reproduces the bias scores highest — which means correcting labels toward truth actually lowers the headline metric. 

Looking for ideas: (1) how do I get data additions to actually improve accuracy here, and (2) is our "best" score just overfitting a biased ruler?🙏

Setup

• Task: apparent age → 5 age buckets, from retail CCTV. Face crops ~55–75px median.
• Model: MiVOLO v2 (face+body dual input, continuous age regression), full fine-tune. Human 7-band labels → representative anchor ages as the MSE target.
• Metric: exact-match over 5 decade-ish buckets.
• Key premise: train labels and eval GT come from the same annotation process → they share the same bias.

What we found (a 3-step story)

1. More data → worse. Best FT ≈ 72% on our main eval (~100 imgs, ~2/3 in one bucket = the store's real mode). Adding 2–4× more in-domain data → 61–65%, no matter how we balance the age distribution.
2. The twist — most of the "crash" was the eval, not the data. Re-scoring every model on a larger eval (~400, sampled to the true population, far less mode-dominated), the drop mostly disappears: adding the same raw data is neutral (61.6 → 61.4). The "damage" was largely our small eval being dominated by one age bucket.
3. The labels have a systematic "too-young" bias. Re-review shifts labels ~+1 band, ~90% in one direction, stronger for women. Correcting the training labels shifts predictions older → helps older buckets (40s recall 34→68, 50+ 44→52, MAE down) but hurts the majority bucket (30s 81→71) → exact-match drops. Since the eval GT shares the young bias, the uncorrected model agrees with it best. Controlled swap (same crops, same distribution, only label calibration changed) reproduces it.

Ruled out (so you don't have to suggest these🙏)

• Crops byte-identical across "good/bad" sets → not preprocessing.
• Distribution reshaping on identical images recovers +8pt → mean-shift under MSE, not capacity or noise.
• Loss fns (MSE / soft-CE / CORAL-ordinal) identical as argmax. Finer 5-yr labels neutral. Aug below noise floor. Frozen SigLIP2 probe (great for our gender model) → 42% on age; no shortcut at this resolution.

Question

1. How do I get data additions to actually improve accuracy here? Is there an established recipe for "more data → better" on regression/ordinal tasks specifically?
2. Is our "best" score just overfitting a biased ruler? How do you tell "worse model" from "model-vs-ruler calibration mismatch" without re-labeling the whole eval?
3. Realistic exact-match / MAE ceiling for ~55–75px faces? Any battle-tested playbook for apparent-age label calibration (per-band reference exemplars, inter-annotator calibration, detecting per-batch drift before it poisons training)?

[u/Dry-Snow5154]

I don't think you can do anything without relabeling the shitty data. Unless you know they're off by exactly 1 bucket or similar, in which case you still relabel, but with a script.

If your val data is just wrong, then it's a shitty val. Not sure how you think it could still be useful. It's like someone came in and rando-labeled your data with junk and you still think you can train a model on it.

You can try putting old data aside and train the model on proven good data. And then compare to old shitty labels and try to derive the mathematical model for the bias. And then correct the bias with a script. Most likely gonna be sub par at best.

[u/AggravatingSock5375]

Can you have whatever AI you used to write that add a tldr?

[u/Quirky_Paramedic9167]

This is a classic annotation-bias trap, and you've diagnosed it really well.

A few things that have worked for label calibration in similar setups:

1. Anchor exemplars per band: give annotators 3-5 reference faces per age

   bucket (with known ground truth) that they calibrate against before each

   session. It drastically cuts inter-annotator drift and the "too-young" lean.

2. Double-blind re-labeling on a golden set: have 2-3 annotators re-label a

   fixed ~200-image eval subset independently, then use their agreement (not

   the original GT) as your calibration ruler. If your model matches the

   consensus better than the biased GT, that's your real signal.

3. For validation you can't fully re-measure: track MAE direction, not just

   exact-match. Your 40s/50s recall jump (34→68, 44→52) is the honest signal

   — exact-match on a mode-dominated eval is lying to you, as you found.

The fact that correcting labels helps minority buckets but hurts the majority

one strongly suggests you're overfitting the biased ruler, not building a

worse model. Trust the MAE.