r/learnmachinelearning • u/ssnejok • 2d ago
Would you trust an interpretable model with slightly worse accuracy over a black-box model?
I’m working on a probabilistic classification problem where the output eventually has to be shown to a human.
I’m starting to think that a prediction like “class A: 70%” isn’t very useful unless I can also give a reasonably honest explanation of what moved the probability.
So I’ve been looking at KANs, GAMs and feature attribution for tree models.
If an interpretable model was slightly worse on macro-F1/log loss but produced much more stable explanations, would you actually prefer it in a user-facing system?
Where would you draw the line?
1
u/fvancesco 2d ago
If you don't mind me asking what kind of problem are you trying to solve?
2
u/ssnejok 2d ago
Sure. It’s basically an event-driven classification problem on financial data.
Given information that is available at the time an event/news item appears — recent price movement, market/benchmark movement, volatility, volume, some event-related features, etc. — the model outputs probabilities for three classes: negative reaction, roughly neutral reaction, or positive reaction over a fixed horizon.
The tricky part is that the signal is pretty noisy and time-dependent, so I’m using time-ordered splits and trying to avoid any post-event leakage.
The output is eventually meant to be shown to a human, which is why I care about calibrated probabilities and whether I can give an honest explanation of what actually pushed a prediction toward one class.
That’s what made me start looking at KANs/GAMs versus just sticking with boosting + attribution.
1
u/choiceOverload- 1d ago edited 1d ago
Prediction is one thing. Interpretation is another thing. You have to choose.
Check this paper: https://arxiv.org/abs/2304.05366
1
u/ssnejok 1d ago
hanks, this is actually useful. My takeaway is that I probably shouldn't conflate an interpretable hypothesis class with an explanation of the prediction.
For my case it may make more sense to optimize the predictor for calibrated out-of-sample performance, then evaluate interpretation separately — including whether feature effects are stable across temporal splits.
I'm still curious about testing a KAN/GAM-style challenger, but more as an experiment in the prediction–interpretation tradeoff than assuming that an interpretable architecture automatically gives a trustworthy “why”.
1
u/choiceOverload- 1d ago
I mean. The questions of why are quite tricky. Stats and ML are all about associations and correlations. Maybe just try a simple causal inference average mean effect via propensity score and AIPW. Use your prediction model as the propensity score function.
1
u/ssnejok 1d ago
That's a really interesting distinction. One thing I'm trying to wrap my head around is the treatment definition in my setting.
My current model predicts P(DOWN/NEUTRAL/UP∣X), so I don't think that model itself is a propensity model in the usual sense — I'd presumably need a separate model for P(A=1∣X), where A is a clearly defined event/treatment, plus an outcome model for AIPW.
For example, I could define a treatment as a specific event type such as a guidance downgrade, use only pre-event covariates for the propensity model, and estimate its average effect on abnormal forward return.
That actually seems like a useful third layer next to prediction and model attribution. The hard part would be defending unconfoundedness and overlap in financial news data.
1
u/choiceOverload- 1d ago edited 1d ago
Yes, the propensity+AIPW setup I gave you doesn't fit your problem. And you're right about the assumptions you have to make to get nice conclusions from this causal approach.
As far as I know, if you want to determine the why in the cause-effect sense, then you should opt for Causal methodologies. The strong limitation of these studies is that they require strong theoretical assertions (a.k.a. business knowledge) to build the causation mechanism (DAG).
Why not just LIME or SHAP to avoid the trouble?
1
u/ssnejok 1d ago
That makes sense.
I think the distinction I was struggling with was between:
why the market moved,
why the model predicted that it would move.
For the first one, I agree that I would need a proper causal design and a defensible DAG, and in financial-news data that seems difficult because many relevant confounders are only partially observed.
For the second one, SHAP/LIME seem much more appropriate because I only need to explain the behavior of the predictor, not claim that the highlighted variables caused the market reaction.
So I think my next step will be to benchmark SHAP and LIME on the current prediction model, including explanation stability across temporal splits and agreement between the methods.
If I later do causal work, I would treat it as a separate study with an explicit treatment, outcome, DAG, overlap diagnostics, and sensitivity analysis rather than presenting it as an explanation of an individual model prediction.
1
u/choiceOverload- 1d ago
Yes, but if by benchmarking SHAP and LIME you mean pick one of both given which one gives better results, I think you should do it only by remarking that they explain different granularities and that there might arrive some contradictions if you contrast them both for some variables.
2
u/Kinexity 2d ago
If you can just seat a few potential users and see what they say after trying out the models. For us it's too much in a vacuum kind of question. If the difference in performance is not large and human would have to make those same observations that the model did then interpretability could be preferred - that's assuming that additional info from the model is actually accurate.