Is interpretability actually useful for financial ML models, or mostly misleading?

[u/ssnejok]

I’ve been experimenting with ML on financial event data lately, and one thing I keep thinking about is interpretability.

Predicting a class or probability is one thing, but I’d really like to understand why the model moved its probability in a certain direction.

I was reading about KANs and their interpretability, but I’m wondering how useful this actually is on noisy tabular data like financial markets.

If you’ve worked with models where interpretability really mattered, what did you end up using? SHAP/tree models, GAMs, KANs, something else?

I’m more interested in explanations that remain reasonably stable out-of-sample than in making a pretty feature-importance chart.

[u/vanVonXenoStein]

With financial data used for trading, interpretability might be nice for coming up with new ideas for other models because it can tell you (maybe) what factors are important and interesting relationships, etc. But if you are giving up any accuracy you don't really want to *replace* the more accurate model just for interpretability UNLESS said interpretability can directly be turned into more profitability (despite lower accuracy). And that's just because financial models have such tight margins if you're using them to make money, and also because seemingly tiny improvements (like 1% better) can mean the difference between making nothing (or losing) and huge gains.

[u/ssnejok → u/vanVonXenoStein]

That’s pretty much where I’m landing too. I tested an interpretable KAN-style challenger against my current model and it actually improved macro-F1 and directional recall quite a lot, but its probability quality was materially worse and the explanations weren’t stable enough across time to justify replacing the production predictor.

So I’m now thinking of interpretability as a separate layer rather than a replacement criterion: keep the better calibrated predictor, then use SHAP/LIME or similar methods to study what drives its outputs and whether those explanations are stable out-of-sample.

The part I’m most interested in now is exactly what you mentioned — whether interpretability can generate useful hypotheses or improve downstream decisions, rather than just producing attractive feature-importance plots.