You've got data. So close. Because of that, 06. You run a t-test because that's what you learned in stats class. Remove that one outlier? tweak something? On top of that, you've got a hypothesis. Still, the p-value comes back 0. You wonder — what if I just... Switch to a non-parametric test because "it's safer"?
Stop. That's not how this works.
The choice between parametric and non-parametric tests isn't a backup plan. It's not a "plan B" when your data misbehaves. It's a decision you make before you collect data — or at minimum, before you analyze it — based on what your data actually is. And get this wrong and you're not just picking the wrong tool. You're answering the wrong question.
What Is a Non-Parametric Test
Parametric tests assume your data follows a specific distribution — usually normal. They lean on parameters like mean and standard deviation. In real terms, t-tests, ANOVA, Pearson correlation — these are parametric. They're powerful when their assumptions hold.
Non-parametric tests don't care about the shape of your distribution. They don't assume normality. They don't assume equal variances. They work with ranks, medians, and order instead of raw values and means And that's really what it comes down to..
That's the short version. But "non-parametric" doesn't mean "assumption-free.Consider this: " It means fewer assumptions about the underlying population distribution. You still need independence. You still need your observations to come from the same general population (unless you're comparing groups deliberately). And you still need to understand what you're actually testing Small thing, real impact..
The rank-based difference
Most non-parametric tests convert your data to ranks. Now, ties get average ranks. The smallest value gets rank 1, the next gets rank 2, and so on. Then the test works with those ranks instead of your original numbers.
This sounds like throwing away information — and it is. But that's the trade-off. You lose some power when parametric assumptions are met. You gain validity when they're not.
Why It Matters / Why People Care
Here's what nobody tells you in intro stats: parametric tests are surprisingly dependable. A t-test with 30+ samples per group? It laughs at moderate non-normality. On top of that, aNOVA? Here's the thing — same story. The central limit theorem does heavy lifting That's the part that actually makes a difference. That's the whole idea..
But robustness has limits.
When robustness fails
Small samples. Skewed distributions with hard boundaries (like reaction times, which can't go below zero). Heavy tails. Ordinal data where the distance between "agree" and "strongly agree" isn't the same as "neutral" to "agree." Outliers that aren't errors — they're just how the system works.
In those cases, a parametric test can give you a p-value that's wrong. Not "borderline.You think you have evidence. Still, confidence intervals don't cover what they claim. Which means " Wrong. Think about it: type I error rates inflate. You don't Practical, not theoretical..
The median vs. mean problem
This is subtle but critical. Think about it: parametric tests compare means. Non-parametric tests typically compare medians (or more precisely, mean ranks — which correspond to medians under certain conditions).
If your distribution is symmetric, mean = median. Practically speaking, the median doesn't. The mean gets pulled toward the tail. But if it's skewed? Now, no problem. A treatment that shifts the median but not the mean — or vice versa — tells you something different about your phenomenon.
Ask yourself: what does "effect" mean in your context? Average change? Typical change? They're not the same question.
How to Decide: A Decision Framework
Don't memorize a flowchart. Understand the logic. Here's how I think through it Less friction, more output..
Step 1: What type of data do you have?
Continuous, ratio/interval scale — height, weight, temperature, test scores (maybe). Parametric tests might work. Keep going.
Ordinal — Likert scales, rankings, severity grades. Parametric tests are technically invalid here. The numbers are labels with order, not quantities. Use non-parametric The details matter here..
Nominal/categorical — completely different toolbox. Chi-square, Fisher's exact, logistic regression. Not this conversation Which is the point..
Step 2: How big is your sample?
Large (n > 30 per group, roughly) — parametric tests are usually fine unless the distribution is wildly non-normal or you have extreme outliers. But "usually fine" isn't a guarantee. Check Worth keeping that in mind..
Small (n < 30 per group) — this is where assumptions matter. You can't rely on the central limit theorem. You need to actually check normality. And you often can't check it well with small samples. Catch-22.
Step 3: What does the distribution look like?
Plot it. Always plot it. Histogram, boxplot, Q-Q plot.
- Roughly symmetric, bell-ish? Parametric is probably fine.
- Skewed? Heavy tails? Hard bounds? Bimodal? Non-parametric starts looking better.
- Outliers? Are they errors? Remove them. Are they real? Parametric tests will chase them. Non-parametric tests won't.
Step 4: What are you comparing?
Two independent groups — Mann-Whitney U (Wilcoxon rank-sum) vs. independent samples t-test Not complicated — just consistent. Simple as that..
Two paired/related groups — Wilcoxon signed-rank vs. paired t-test Worth keeping that in mind..
Three+ independent groups — Kruskal-Wallis vs. one-way ANOVA.
Three+ related groups — Friedman test vs. repeated measures ANOVA.
Correlation — Spearman's rho vs. Pearson's r.
Each pair answers a slightly different question. Know which question you're asking It's one of those things that adds up..
Common Mistakes / What Most People Get Wrong
Mistake 1: "My data isn't normal, so I'll use non-parametric"
You ran a Shapiro-Wilk test. p = 0.04. "Aha! Non-normal!" You switch to Mann-Whitney.
Two problems. Worth adding: first, normality tests are oversensitive in large samples — they detect trivial deviations that don't matter. Second, they're undersensitive in small samples — they miss deviations that do matter.
Don't let a p-value decide. Look at the plot. They're ex-Gaussian. Reaction times are never normal. Think about the data-generating process. That's not a violation — that's the truth No workaround needed..
Mistake 2: Treating non-parametric tests as "the same test, but for non-normal data"
Mann-Whitney doesn't test "difference in medians" unless the two distributions have the same shape (same spread, same skew). If shapes differ, it tests whether a randomly selected value from group A tends to be larger than a randomly selected value from group B. That's a stochastic dominance question. Not a median question Turns out it matters..
If you report "median difference" from a Mann-Whitney test with different-shaped distributions, you're misinterpreting your own result.
Mistake 3: Ignoring ties
Likert scales. Here's the thing — if more than, say, 20-30% of your data are ties, think harder. Discrete measurements. Most software handles ties with a correction. So ties everywhere. But heavy ties reduce power and can bias some tests. Maybe ordinal regression is better.
Mistake 4: "Non-parametric tests don't need assumptions"
They need fewer distributional assumptions. They still need:
- Independence (or known dependence structure)
- Same measurement scale across groups
- Random sampling (or random assignment)
- For rank tests: the response variable should be at least ordinal
Violate independence and no test saves you That's the part that actually makes a difference. Took long enough..
Mistake 5: Reporting only p-values
Mistake 5: Reporting only p‑values
If your manuscript contains a single line that reads, “The Mann‑Whitney U test revealed a significant difference (p = 0.And 02). ” you’re missing the most informative part of the story. A p‑value tells you whether the null hypothesis can be rejected, but it says nothing about effect size, direction, or practical relevance That's the whole idea..
| What to add | Why it matters |
|---|---|
| Rank‑based effect size (e., r = Z/√N or Cliff’s Δ) | Quantifies the magnitude of the difference. So naturally, |
| Illustrative plots (boxplots, violin plots, ECDFs) | Allows readers to see the shape and spread of each group. So |
| Confidence intervals for the median difference or for the rank‑sum statistic | Shows the precision of your estimate. g. |
| Narrative interpretation | Connects the statistical result to your research question. |
Mistake 6: Treating the test statistic as if it were a parameter
orte. The U statistic is a function of the ranks, not a direct measure of a population parameter. But consequently, you cannot plug it into a regression model or use it in a meta‑analytic formula without transformation. If you need to combine studies, convert U or the rank‑sum to a standardized effect size first Worth keeping that in mind..
Mistake 7: Over‑reliance on “default” software defaults
Most statistical packages will automatically apply a continuity correction or a tie correction. Always check the algorithm your software uses, especially when you’re dealing with highly skewed data or many ties. These adjustments are not universal: some are appropriate for large samples, others for small samples. If the default is not suitable, use the alternative implementation that matches your data characteristics Small thing, real impact. No workaround needed..
Mistake 8: Ignoring the power of rank tests
Rank tests are often perceived as “weak” because they ignore diversified information (exact values). Still, this perception is misleading. The power depends on the shape of the underlying distributions. For two distributions that differ mainly in location (e.g., two normal curves with the same variance), the Mann‑Whitney U test is nearly as powerful as the t‑test. When the distributions differ in shape (skew, kurtosis, or spread), the U test can actually outperform the t‑test.
Basically the bit that actually matters in practice Easy to understand, harder to ignore..
Tip: Run a quick simulation with your own data distribution to gauge power before deciding on a test.
Practical Checklist for Choosing and Reporting a Non‑Parametric Test
- Define the research question (difference in location, difference in shape, correlation, etc.).
- Verify measurement level – at least ordinal.
- Check independence – random sampling or assignment; if not, use a matched‑pairs design or a mixed‑effects model.
- Inspect the data – visualise (histograms, boxplots, ECDFs) and quantify skewness/kurtosis.
- Decide on the test – use the table in the “What are you comparing?” section as a guide.
- Choose the right software implementation – mind continuity/tie corrections.
- Compute effect size and confidence intervals – report them alongside p‑values.
- Plot the data – include a visual representation of the distribution and the estimated effect.
- Interpret the result – link it back to the theoretical framework and practical implications.
- State assumptions – even for non‑parametric tests, list the assumptions you rely on (e.g., independence, same measurement scale).
When to Consider Alternatives
| Situation | Alternative |
|---|---|
| Large sample, small ties, continuous data | Parametric test ( CFD ) |
| Moderate sample, many ties, ordinal data | Ordinal logistic regression |
| Repeated measures with non‑normal residuals | Mixed‑effects model with reliable standard errors |
| Time‑to‑event data | Kaplan–Meier / Cox proportional hazards |
Choosing the “right” test is less about rigid rules and more about aligning your statistical method with the underlying data structure and the substantive question you’re asking Practical, not theoretical..
Final Thoughts
Non‑parametric tests are not a one‑size‑fits‑all remedy; they are a family of tools designed for specific data scenarios. The most common pitfalls arise from over‑simplifying the decision process, misinterpreting the meaning of the test statistic, and neglecting effect sizes and visual context. By treating each dataset as a unique story, asking the precise question you want answered, envelopes, and following a transparent reporting routine, you can harness the full power of rank‑based inference.
Remember: the goal of statistical analysis is not to win a “parametric vs. non‑parametric” battle but to obtain a clear, honest, and reproducible understanding of the data you’ve collected. Use the right tool, report the full picture, and let the evidence guide your conclusions.
