You've got data. You've got a hypothesis. You run a t-test because that's what you learned in stats class. The p-value comes back 0.06. So close. You wonder — what if I just... On top of that, tweak something? Remove that one outlier? Switch to a non-parametric test because "it's safer"?
Stop. That's not how this works.
The choice between parametric and non-parametric tests isn't a backup plan. It's not a "plan B" when your data misbehaves. That said, it's a decision you make before you collect data — or at minimum, before you analyze it — based on what your data actually is. But get this wrong and you're not just picking the wrong tool. You're answering the wrong question Easy to understand, harder to ignore..
What Is a Non-Parametric Test
Parametric tests assume your data follows a specific distribution — usually normal. They lean on parameters like mean and standard deviation. T-tests, ANOVA, Pearson correlation — these are parametric. They're powerful when their assumptions hold Not complicated — just consistent. Turns out it matters..
Non-parametric tests don't care about the shape of your distribution. They don't assume normality. Practically speaking, they don't assume equal variances. They work with ranks, medians, and order instead of raw values and means The details matter here..
That's the short version. But "non-parametric" doesn't mean "assumption-free.In real terms, you still need independence. You still need your observations to come from the same general population (unless you're comparing groups deliberately). " It means fewer assumptions about the underlying population distribution. And you still need to understand what you're actually testing.
The rank-based difference
Most non-parametric tests convert your data to ranks. Ties get average ranks. In practice, the smallest value gets rank 1, the next gets rank 2, and so on. Then the test works with those ranks instead of your original numbers Most people skip this — try not to. Less friction, more output..
This sounds like throwing away information — and it is. But that's the trade-off. You lose some power when parametric assumptions are met. You gain validity when they're not But it adds up..
Why It Matters / Why People Care
Here's what nobody tells you in intro stats: parametric tests are surprisingly strong. On the flip side, a t-test with 30+ samples per group? And it laughs at moderate non-normality. ANOVA? On top of that, same story. The central limit theorem does heavy lifting.
But robustness has limits.
When robustness fails
Small samples. Heavy tails. Skewed distributions with hard boundaries (like reaction times, which can't go below zero). Ordinal data where the distance between "agree" and "strongly agree" isn't the same as "neutral" to "agree." Outliers that aren't errors — they're just how the system works.
In those cases, a parametric test can give you a p-value that's wrong. Which means confidence intervals don't cover what they claim. That said, " Wrong. Which means not "borderline. You think you have evidence. Type I error rates inflate. You don't.
The median vs. mean problem
This is subtle but critical. Parametric tests compare means. Non-parametric tests typically compare medians (or more precisely, mean ranks — which correspond to medians under certain conditions).
If your distribution is symmetric, mean = median. No problem. But if it's skewed? In real terms, the mean gets pulled toward the tail. The median doesn't. A treatment that shifts the median but not the mean — or vice versa — tells you something different about your phenomenon.
Ask yourself: what does "effect" mean in your context? Average change? Typical change? They're not the same question.
How to Decide: A Decision Framework
Don't memorize a flowchart. Understand the logic. Here's how I think through it.
Step 1: What type of data do you have?
Continuous, ratio/interval scale — height, weight, temperature, test scores (maybe). Parametric tests might work. Keep going.
Ordinal — Likert scales, rankings, severity grades. Parametric tests are technically invalid here. The numbers are labels with order, not quantities. Use non-parametric Took long enough..
Nominal/categorical — completely different toolbox. Chi-square, Fisher's exact, logistic regression. Not this conversation Less friction, more output..
Step 2: How big is your sample?
Large (n > 30 per group, roughly) — parametric tests are usually fine unless the distribution is wildly non-normal or you have extreme outliers. But "usually fine" isn't a guarantee. Check Not complicated — just consistent. No workaround needed..
Small (n < 30 per group) — this is where assumptions matter. You can't rely on the central limit theorem. You need to actually check normality. And you often can't check it well with small samples. Catch-22 Still holds up..
Step 3: What does the distribution look like?
Plot it. Always plot it. Histogram, boxplot, Q-Q plot.
- Roughly symmetric, bell-ish? Parametric is probably fine.
- Skewed? Heavy tails? Hard bounds? Bimodal? Non-parametric starts looking better.
- Outliers? Are they errors? Remove them. Are they real? Parametric tests will chase them. Non-parametric tests won't.
Step 4: What are you comparing?
Two independent groups — Mann-Whitney U (Wilcoxon rank-sum) vs. independent samples t-test.
Two paired/related groups — Wilcoxon signed-rank vs. paired t-test.
Three+ independent groups — Kruskal-Wallis vs. one-way ANOVA Easy to understand, harder to ignore..
Three+ related groups — Friedman test vs. repeated measures ANOVA.
Correlation — Spearman's rho vs. Pearson's r Most people skip this — try not to..
Each pair answers a slightly different question. Know which question you're asking.
Common Mistakes / What Most People Get Wrong
Mistake 1: "My data isn't normal, so I'll use non-parametric"
You ran a Shapiro-Wilk test. p = 0.In practice, non-normal! In real terms, "Aha! In real terms, 04. " You switch to Mann-Whitney.
Two problems. Worth adding: first, normality tests are oversensitive in large samples — they detect trivial deviations that don't matter. Second, they're undersensitive in small samples — they miss deviations that do matter And it works..
Don't let a p-value decide. Look at the plot. Think about the data-generating process. Here's the thing — reaction times are never normal. They're ex-Gaussian. That's not a violation — that's the truth Small thing, real impact. Still holds up..
Mistake 2: Treating non-parametric tests as "the same test, but for non-normal data"
Mann-Whitney doesn't test "difference in medians" unless the two distributions have the same shape (same spread, same skew). Think about it: if shapes differ, it tests whether a randomly selected value from group A tends to be larger than a randomly selected value from group B. This leads to that's a stochastic dominance question. Not a median question Most people skip this — try not to. But it adds up..
If you report "median difference" from a Mann-Whitney test with different-shaped distributions, you're misinterpreting your own result.
Mistake 3: Ignoring ties
Likert scales. Discrete measurements. Consider this: ties everywhere. Now, most software handles ties with a correction. But heavy ties reduce power and can bias some tests. Day to day, if more than, say, 20-30% of your data are ties, think harder. Maybe ordinal regression is better Nothing fancy..
Mistake 4: "Non-parametric tests don't need assumptions"
They need fewer distributional assumptions. They still need:
- Independence (or known dependence structure)
- Same measurement scale across groups
- Random sampling (or random assignment)
- For rank tests: the response variable should be at least ordinal
Violate independence and no test saves you Less friction, more output..
Mistake 5: Reporting only p-values
Mistake 5: Reporting only p‑values
If your manuscript contains a single line that reads, “The Mann‑Whitney U test revealed a significant difference (p = 0.But 02). ” you’re missing the most informative part of the story. A p‑value tells you whether the null hypothesis can be rejected, but it says nothing about effect size, direction, or practical relevance Less friction, more output..
| What to add | Why it matters |
|---|---|
| Rank‑based effect size (e.g., r = Z/√N or Cliff’s Δ) | Quantifies the magnitude of the difference. |
| Confidence intervals for the median difference or for the rank‑sum statistic | Shows the precision of your estimate. |
| Illustrative plots (boxplots, violin plots, ECDFs) | Allows readers to see the shape and spread of each group. |
| Narrative interpretation | Connects the statistical result to your research question. |
Mistake 6: Treating the test statistic as if it were a parameter
orte. On the flip side, the U statistic is a function of the ranks, not a direct measure of a population parameter. So naturally, consequently, you cannot plug it into a regression model or use it in a meta‑analytic formula without transformation. If you need to combine studies, convert U or the rank‑sum to a standardized effect size first.
Mistake 7: Over‑reliance on “default” software defaults
Most statistical packages will automatically apply a continuity correction or a tie correction. These adjustments are not universal: some are appropriate for large samples, others for small samples. Always check the algorithm your software uses, especially when you’re dealing with highly skewed data or many ties. If the default is not suitable, use the alternative implementation that matches your data characteristics.
Mistake 8: Ignoring the power of rank tests
Rank tests are often perceived as “weak” because they ignore diversified information (exact values). On the flip side, this perception is misleading. Day to day, the power depends on the shape of the underlying distributions. And for two distributions that differ mainly in location (e. Day to day, g. In practice, , two normal curves with the same variance), the Mann‑Whitney U test is nearly as powerful as the t‑test. When the distributions differ in shape (skew, kurtosis, or spread), the U test can actually outperform the t‑test.
Tip: Run a quick simulation with your own data distribution to gauge power before deciding on a test.
Practical Checklist for Choosing and Reporting a Non‑Parametric Test
- Define the research question (difference in location, difference in shape, correlation, etc.).
- Verify measurement level – at least ordinal.
- Check independence – random sampling or assignment; if not, use a matched‑pairs design or a mixed‑effects model.
- Inspect the data – visualise (histograms, boxplots, ECDFs) and quantify skewness/kurtosis.
- Decide on the test – use the table in the “What are you comparing?” section as a guide.
- Choose the right software implementation – mind continuity/tie corrections.
- Compute effect size and confidence intervals – report them alongside p‑values.
- Plot the data – include a visual representation of the distribution and the estimated effect.
- Interpret the result – link it back to the theoretical framework and practical implications.
- State assumptions – even for non‑parametric tests, list the assumptions you rely on (e.g., independence, same measurement scale).
When to Consider Alternatives
| Situation | Alternative |
|---|---|
| Large sample, small ties, continuous data | Parametric test ( CFD ) |
| Moderate sample, many ties, ordinal data | Ordinal logistic regression |
| Repeated measures with non‑normal residuals | Mixed‑effects model with dependable standard errors |
| Time‑to‑event data | Kaplan–Meier / Cox proportional hazards |
Choosing the “right” test is less about rigid rules and more about aligning your statistical method with the underlying data structure and the substantive question you’re asking.
Final Thoughts
Non‑parametric tests are not a one‑size‑fits‑all remedy; they are a family of tools designed for specific data scenarios. Think about it: the most common pitfalls arise from over‑simplifying the decision process, misinterpreting the meaning of the test statistic, and neglecting effect sizes and visual context. By treating each dataset as a unique story, asking the precise question you want answered, envelopes, and following a transparent reporting routine, you can harness the full power of rank‑based inference.
Remember: the goal of statistical analysis is not to win a “parametric vs. non‑parametric” battle but to obtain a clear, honest, and reproducible understanding of the data you’ve collected. Use the right tool, report the full picture, and let the evidence guide your conclusions Practical, not theoretical..
