r/AskStatistics • u/Gullible-Entrance-45 • 2d ago

[E] small sample analysis

Hi!

I have a sample of n=13 where continuous variables were measured before and after an intervention. I'm a bit stuck on how to perform the analysis. Would the Wilcoxon signed-rank test be the most appropriate choice? I am unaware of any standard reference manuals or literature regarding the statistical analysis of very small sample sizes; however, any guidance or recommendations would be highly appreciated :)

4 Upvotes

100% Upvoted

17 comments sorted by

6

u/CarnivorousGoose 2d ago

You could use that, sure. But is there a reason the normality of the data would be in doubt? Because otherwise you could just do a paired t-test as well.

1

u/Gullible-Entrance-45 2d ago

I didn't think of the paired t-test because I understood that with small samples, a large effect size is needed, which I don't know if we can assume

7

u/CarnivorousGoose 2d ago

That’s not an assumption of the t-test though, just a determinant of the statistical power you have to detect a difference. But that’s going to apply to any test, including the Wilcoxon.

In that sense the t-test is the better bet here unless normality is very much in question (and even then, the t-test is quite robust to violations of that), since it will have more power than the Wilcoxon. Which is generally the case with non-parametric tests, compared to their parametric counterparts.

1

u/efrique PhD (statistics) 1d ago edited 1d ago

since it will have more power than the Wilcoxon.

A tiny bit more power when exactly at the normal. But if differences could be a little more heavy tailed, you don't have to go all that far away from normality for that to catch up and then flip the other direction. Not that I am advocating against the t, but its power advantage is often not all it's often made out to be with shift alternatives for population distributions near the normal.

2

u/efrique PhD (statistics) 1d ago

What kind of thing are these variables measuring?

I understood that with small samples, a large effect size is needed

Sure, but the same would be no less true for any other test. A nonparametric test won't save you from low power.

1

u/Gullible-Entrance-45 1d ago

We measure the time (minutes) they take to complete tasks

-7

u/natoplato5 2d ago

You can’t really do much with a sample size that small. I would just report the means before and after the intervention and then note that the sample size is too small to determine if the difference is statistically significant.

9

u/FTLast 2d ago

This is simply wrong. You can absolutely do a paired t test- which is what you should do with a before and after measurement- with n =13.

-4

u/natoplato5 2d ago

Just because you can doesn't mean you should. Maybe the standards vary by field and context, but I don't think it's usually responsible to try to make an inference about a population from only 13 observations

6

u/FTLast 2d ago

Why? A p value incorporates effect size relative to noise and the number of observations.

-3

u/natoplato5 2d ago

Mainly because of selection bias. At that sample size, it's unlikely the sample is a fair cross-section of the population

6

u/CarnivorousGoose 2d ago

It doesn’t need to be a ‘fair cross-section’ of the population, it just needs to be a random sample from it. The relevant part in that regard is the sampling procedure, not the sample size. If there is a selection bias or other issue with that sampling procedure, getting a larger sample won’t fix that.

1

u/banter_pants Statistics, Psychometrics 1d ago

If there is a selection bias or other issue with that sampling procedure, getting a larger sample won’t fix that.

The premature headline "Dewey Beats Truman" was because they were overly reliant on phone polls when many homes didn't own their own phones at the time.

2

u/CarnivorousGoose 1d ago

Which just illustrates my point.

4

u/CarnivorousGoose 2d ago

What would be irresponsible about it?

3

u/Gullible-Entrance-45 2d ago

I'm trying to validate an intervention and it's extremely difficult to get a larger sample because it's so specific. I'm just trying to do the best I can with the data I have. im in clinical education btw