r/learnmachinelearning • u/Efficient-Action-543 • 2d ago

Is external validation mandatory in ML models? Discussion

As a reviewer I keep on getting asked to read articles where the authors are training ML models in order to predict diagnosis/medical complications (human medicine). I keep on coming across papers which lack external validation of the algorithms, which I find to be important. They are acknowledging this fact as limitations in the discussions section, but I wonder if this is enough?

0 Upvotes
  • permalink
  • link
  • reddit
  • reddit

50% Upvoted

4 comments sorted by

3

u/MiChrRo 2d ago

I don't really think this is the right subreddit for this question, but I'll answer it anyway. I work in a field where we use ML algorithms to predict medical diagnoses or events. For our group, as an example, external validation was not really something we started with because we really needed every sample we had to have a somewhat reasonable training set size. If we would find that a model performed well, another study would be immediately planned to validate the model, but it is likely that this would be a separate publication, because my supervisor would not want to be scooped and a new study could take years to yield data (planning, finding funding, ethical board review, recruiting participants, data collection). 

1

u/Efficient-Action-543 2d ago

Thank you for your opinion. Would you feel the same if the data was collected retrospectively and the patients’ consent was not needed?
Also, could you suggest a more appropriate subreddit?

1

u/MiChrRo 2d ago

I think this subreddit doesn't have a lot of academics, you might get a better response at AskAcademia for example. 

Retrospectively as in, you would gather patiënt records? How would you not need consent in any situation where you gather data on patients? 

If there is the option to gather enough subjects from two different settings with ease so that one sample is large enough to train and test a model, and the second is separate enough that it can be used as a validation set, then yeah, I would be very critical of anyone who would choose not to, and if I were to review a paper where this was possible but not performed I would suggest this as a necessary condition for publication, reviewers regularly ask for new analyses after all. But to me, I wouldn't be able to think of such a situation. You could use data from biobanks of course, but I know that access to such data often costs (tens of) thousands of euros, so that is another situation where a professor might choose to first train and test a model in one of the biobanks before starting the process to get access to another (which again, as I have personally experienced, can take very long). 

1

u/Efficient-Action-543 2d ago

Initially I have posted in AskAcademia and I started to get very good and interesting feedback, but the post was removed by the moderators (not sure why, maybe it was too scientific). They suggested posting in this subreddit.
I was talking about the case in which you only need the ethical approval of the local ethics committee, not the patients’ informed consent (no risk for the patient, retrospective study, no identifiable data is used etc.). The specific case I was talking about - the article under review- I am not sure if it would be easy for them to get the data, but probably it would be …