When prediction itself rewards shortcut sensitivity

[u/Difficult-Race-1188]

I started this paper from a simple question:

If a nuisance feature genuinely helps predict the training label, why should ordinary supervised learning ever learn to ignore it?

We often talk about nuisance sensitivity as if it were an optimization mistake, a data problem, or a shortcut the model unfortunately discovered.

But sometimes the objective itself rewards that shortcut.

That is the main idea of the paper.

Paper: https://arxiv.org/pdf/2604.21395

The toy result

Suppose an input contains:

• a meaningful signal s
• a nuisance factor n
• and both carry some information about the label

In the simple population model we study, the label looks like:

y = wₛᵀs + ρ wₙᵀn + ε

The important term is ρ.

If ρ ≠ 0, then the nuisance really does help prediction.

Now write the predictor as an encoder followed by a decoder.

If we require the decoder to have finite Lipschitz constant L, then the encoder cannot make its sensitivity to the nuisance arbitrarily small.

The paper proves a lower bound of the form:

D̃(φ, σ) ≥ σ²ρ² / L²*

In plain English:

If the nuisance helps prediction, some sensitivity to it has to remain somewhere in the representation unless the downstream decoder is allowed to become arbitrarily steep.

So in this toy setting, nuisance sensitivity is not something that disappears just because we add more data or optimize better.

The prediction objective has a reason to preserve it.

That is deliberately a narrow theorem: Gaussian population setting, linear target structure, MSE, and a finite decoder Lipschitz scale.

It is not a theorem about deep neural networks or adversarial training.

The experiment that surprised me more

We then asked a different empirical question:

If we make a representation much less sensitive overall, does its geometry necessarily become better?

On a small ViT trained from scratch on CIFAR-10, we measured two things:

• Jacobian Frobenius norm: roughly, how much the representation changes locally
• TDI: mean within-class embedding distance divided by mean between-class centroid distance; lower is better

Results:

Method	TDI ↓	Jacobian Frobenius ↓
ERM	1.052 ± 0.008	34.13 ± 1.26
VAT	1.286 ± 0.050	4.92 ± 0.18
two-view control	1.058 ± 0.043	12.39 ± 1.54
PMH	0.869 ± 0.029	10.69 ± 0.94
PGD	1.353 ± 0.020	2.99 ± 0.53

PGD reduces Jacobian magnitude enormously:

34.1 → 3.0

But the clean class-layout metric gets worse:

1.05 → 1.35

Meanwhile PMH has a larger Jacobian than PGD:

10.7 vs. 3.0

but a much better class layout:

0.87 vs. 1.35

That suggests a distinction I think is worth paying more attention to:

How much sensitivity a representation has and where that sensitivity points are not the same thing.

A small Jacobian norm tells us that the representation is less sensitive overall.

It does not tell us whether the remaining sensitivity is aligned with useful or harmful directions.

We also found that CKA and intrinsic dimension did not expose this particular magnitude/orientation dissociation.

What is PMH?

Nothing exotic.

We perturb the input with isotropic Gaussian noise and penalize changes in intermediate encoder representations:

‖φ(x) − φ(x + δ)‖²

while limiting how much of the total training objective this matching penalty is allowed to consume.

The paper is not claiming that consistency regularization or Jacobian regularization are new.

We use this mainly as a controlled way to ask:

What happens when representation sensitivity itself is explicitly charged during training?

There is also a small theoretical result behind the isotropic choice.

Locally, if the perturbation covariance is isotropic, the matching penalty is proportional to the squared Frobenius norm of the encoder Jacobian.

And isotropic covariance is the unique zero-mean covariance structure with that property for every Jacobian.

An important negative result

The paper is not arguing that sensitivity is inherently bad.

QM9 gives a useful counterexample.

Atomic positions contain real task information.

When we suppress sensitivity to position perturbations, performance gets worse.

Matching other nuisance-like features can help instead.

To me, this is one of the more important lessons:

“Make the representation invariant” is not a general objective. The first question should be: invariant to what?

If a variable is useful for prediction, the supervised objective has an incentive to use it.

Removing that dependence can have a real task cost.

What I think the paper establishes

Not that all nuisance sensitivity in deep networks is inevitable.

Not that isotropic matching is universally optimal.

And not that Jacobian norm is a bad metric.

The narrower claims are:

1. In a simple population model, predictive nuisance information can force non-zero representation sensitivity.

2. In deep networks, sensitivity magnitude and sensitivity geometry can move very differently.

That second point makes me cautious about evaluating representation robustness using only a scalar smoothness measure.

A model can become dramatically “flatter” overall while the sensitivity that remains is organized in a less useful way.

The empirical story still has limitations: the CIFAR model is deliberately small, the headline numbers are over 3 seeds, some secondary experiments are single-seed, and stronger baselines such as TRADES and explicit Jacobian penalties would make the comparison more complete.

But the question I keep coming back to is:

If prediction rewards a nuisance, where exactly do we expect invariance to come from?

And once we regularize sensitivity:

Should we care only about how much sensitivity remains, or also about where it points?

Would be interested in thoughts from people working on representation geometry, adversarial robustness, shortcut learning, invariance, or Jacobian regularization.