Statistical Significance Is Not a License to Ship
Statistical significance means a result would be rare if nothing changed. P-values, 0.05, practical vs statistical significance, and peeking's false positives.
Updated 11 min read
Statistical significance means a result at least as extreme would be rare if nothing actually changed: if the variant and control were the same. Product teams use it as a noise checkpoint, not as proof to ship. NN/g shows an 85.0% vs 85.2% checkout can still hit p = 0.03.
A significant experiment result answers one question: would this gap be unusual under a "no difference" model? It does not measure effect size, economic value, or whether engineering time is worth spending. Amplitude took the Statsig brand in May 2026; this page is the concept, not that product.
Key Takeaways
- A result is statistically significant when data at least this extreme would be rare if the null hypothesis were true.
- A p-value is P(data at least this extreme | H0). It is not P(H0 is true) and not a 97% chance the variant is better.
- 0.05 is a Fisher-era rule of thumb, not a law. Crossing it is not a ship license.
- Practical significance asks whether the magnitude is worth the cost. A 0.2 pp checkout lift can be significant and still tiny.
- Watching a live p-value and stopping at 0.05 inflates false positives. Johari found Type I error can rise fivefold even at n = 10,000.
What Is Statistical Significance?
Statistical significance is a decision rule for noisy data. You assume a null model (usually "no difference") and compute how often a result at least as extreme would appear. You reject the null when that probability falls at or below a pre-chosen threshold α.
The ASA informal definition is the one to keep: the probability, under a specified statistical model, that a statistical summary of the data would be equal to or more extreme than its observed value. Product translation, from Atticus Li: a p-value is a statement about the data, not about the hypothesis.
Every experiment is a sample. A 2% conversion lift on 800 users can be a real change or a Tuesday. Significance is the shared language for "this gap is hard to chalk up to sampling noise," so a PM, a designer, and a data scientist are arguing about the same checkpoint.
Significance is not importance. A tiny checkout change on a huge sample can star. A large change on a two-week test can miss.
Pair this checkpoint with UX research methods when the question is what to measure, and with qualitative research when you need to know why the number moved.
How Statistical Significance Works
A significance test has four pieces: a null, an α, a test statistic, and a p-value. The first three sit here. Skip any one and the dashboard number is theater.
The Null Hypothesis
H0 is the boring story: variant = control, or "this copy change did nothing." You never "accept" the null. You reject it or you fail to reject it.
Fail-to-reject means the data were not unusual enough, not that you proved the variants are identical.
The Significance Level
α is the false-positive rate you accept if H0 is true. Teams set it to 0.05 before the test, then treat 0.049 and 0.051 as different universes. They are not.
The Conversation traces 0.05 to Ronald Fisher in the 1920s: a rule of thumb that hardened into a rule. If there is no real effect, about 1 in 20 tests still alarm. That 1-in-20 rate is a choice, not a discovery.
The Test Statistic
A two-proportion z-test, a t-test, or a chi-square test is only the machine that produces a p-value from your metric and sample. Name the primary metric and the randomization unit before you look. Then stop: this page is not a methods class.
P-Values in Plain Language
On r/statistics, the recurring confusion is swapping the condition. People want P(the null is true | the data). A frequentist p-value does not give it to you.
A p-value is the probability of seeing data at least as extreme as yours, if the null is true: P(data at least this extreme | H0). The Conversation flags the named mistake: a p-value is not the chance the result happened by accident.
"The p value is the probability of our data given that the null hypothesis is true: P(data | null). What everyone intuitively wants it to mean is the probability of the null given our data: P(null | data). But we know these two things are not equal." (u/Maple_shade in r/statistics, Jan 2026)
Ronny Kohavi hits the product version of the same swap. Teams read 1 − p as "chance B is better."
"Many people assign one minus p-value as the probability that your treatment is better than control. That is wrong." (Ronny Kohavi in Lenny's Podcast, 1:02:33)
Honest stakeholder script: 100 runs on a no-difference page would show a gap this dramatic about 5 times (α = 0.05). That is a smoke detector: statistically unlikely there is no fire. It does not tell you there is a fire, how big, or whether to evacuate (Atticus).
What a P-Value Doesn't Tell You
Head search results still teach the inverted definition, or treat a small p as proof of cause. The missing corpus is the ASA's 2016 statement (Wasserstein and Lazar). Six principles, translated for a shipping meeting:
- A p-value says how incompatible the data are with a specified model, if the assumptions hold.
- It does not measure P(the hypothesis is true) or P(chance alone). It is a statement about data relative to a hypothetical explanation, not about the explanation.
- A business decision should not rest only on whether p crosses a threshold. A finding does not become true on one side of 0.05 and false on the other: treating p < 0.05 as a license "leads to considerable distortion."
- Cherry-picking, optional stopping, and unreported slices make the reported p uninterpretable. Full reporting is part of the inference.
- Statistical significance does not measure effect size or importance, and it is not scientific, human, or economic significance. Tiny effects yield small p when n is huge; large effects look weak when n is small.
- By itself, a p-value is a poor measure of evidence. Report the effect estimate and an interval.
Greenland et al. add the assumption trap: the p-value tests the entire model, not just H0. A tiny p does not tell you which piece failed (a false null, a protocol violation, or selective reporting).
Atticus compresses the product don't-list to five. A p-value is not P(the variant works), the effect size, whether it will replicate, whether you should ship, or whether the test was well designed.
Kyle Giddon is blunt on the fourth item. Even a significant and practically large result is still a cost-benefit call under uncertainty. Frequentist tests are silent on P(you are right).
Daniël Lakens's shorter version: p-values are not a coherent measure of evidence.
Common mistake, even among statisticians, to think p-values are measures of evidence. I explain why that is not a coherent way of thinking about p-values here https://t.co/5To0bsR5IZ https://t.co/UTNbeWgTYC
Causation is not in the p-value either. Random assignment can support a causal design. A starred dashboard cannot.
Practical vs Statistical Significance
Noise is not value. Statistical significance asks whether the pattern is distinguishable from sampling noise under the model. Practical significance asks whether the magnitude is worth acting on.
Type | Best For | Key Characteristics |
|---|---|---|
Statistical significance | Noise checkpoint | p ≤ α; says the gap is unusual if H0 is true |
Practical significance | Ship / no-ship | Magnitude vs cost, noticeability, opportunity cost |
Nonsignificant result | Avoiding false zeros | Fail to reject H0; not proof of no effect |
NN/g (6 Mar 2026) uses a checkout of 85.0% vs 85.2% with p = 0.03. "That's all statistical significance means. It says nothing about how large, valuable, or noticeable the effect is."
Their own coda: a 0.5% drop in checkout errors on 2 million checkouts a year is 10,000 fewer failed transactions. Tiny can still be real money. Run the money math anyway.
Atticus runs the same logic on a conversion baseline: 3.00% to 3.03%, n = 500,000, p = 0.001. On 100,000 monthly visitors, that is +30 conversions a month. At $50 AOV it is about $18,000 a year.
If implementation costs $25,000, you are underwater for 17 months. Statistically significant. Practically not worth it.
On r/AskStatistics, u/smurferdigg put the classroom version: n ≈ 100,000 (everything stars) versus n = 99 (nothing does). The star is often a sample-size readout.
"You can get away with using just p-values when your findings don't actually matter. If I build a bridge and say 'yeah it's stronger than a pile of wet tissue paper, p<0.05' you need way more than that before anyone drives over it." (u/smbtuckma in r/AskStatistics, Apr 2026)
Clinical significance is the medical cousin: would a patient notice? Same split as practical vs statistical. For a UX test, stay on checkout, conversion, and engineering cost.
A 20% lift can be nonsignificant on a small sample; a 0.2% lift can be significant on a huge one. "Is 20% statistically significant?" is the wrong question. Ask whether the interval around the lift clears the change you would actually ship.
Benefits of Statistical Significance
A Shared Stop/Go Language
Without a pre-chosen α, every readout becomes a negotiation. Significance gives the PM, the designer, and finance one checkpoint: was this gap unusual under "nothing happened"? Usability testing still owns the "why"; this checkpoint owns whether the number is distinguishable from noise.
A Cap on False Alarms, If You Don't Peek
If H0 is true and you look once at a pre-registered horizon, α = 0.05 means about 1 in 20 false alarms. That is useful. Watch the p-value drip and stop at the first 0.05, and you destroy it.
It Forces the Money Question
A significant result is the prompt to look at magnitude, interval, and cost, not the end of the meeting. Skip that, and you ship wet tissue paper because it starred.
Challenges and Limitations
Peeking at Live P-Values
You watch the dashboard and stop when p first dips below 0.05. Classical p-values assume a stopping rule that does not depend on the data. Optional stopping breaks that.
Johari et al. showed that even with 10,000 samples, Type I error can easily increase fivefold. Always-valid p-values (mSPRT) were built to control Type I at any stopping time. This page will not teach that math.
VWO simulations put naive peeking in a 15% to nearly 40% false-positive band.
Plot of a p-value, calculated for a simple t-test, for sample sizes of 10 to 2010, when the null hypothesis is true. Because p-values are uniformly distributed, the p-value just wanders randomly between 0 and 1. This is why repeatedly analyzing data as it comes in is not good! https://t.co/80gz7uBaVj
Under a true null, the p-value wanders randomly between 0 and 1 as n grows. That wandering is why you do not re-analyze the same test as traffic arrives.
Always-valid, sequential, or Bayesian designs are what teams use instead of a live p-value as a stop button. Health checks (sample-ratio mismatch, guardrails) are not the same as calling the test.
Winner's Curse on Underpowered Wins
Low power does not produce honest small wins. The estimate had to clear the threshold, so it is inflated.
Jaksic et al. (2026): at power below 30%, relative bias among significant results exceeds 1.78. At 25% power, significant effects are about twice the true effect. At power ≥ 80%, relative bias stays under 1.13.
Kohavi et al. (Econ Journal Watch, March 2026) revisit a Journal of Consumer Research claim: rounded CTA buttons, +55% CTR, p = 0.037. High-powered replications came in about two orders of magnitude smaller, with 95% confidence intervals that include zero.
Kohavi's 20% power lesson: 8 of 10 tests will not be significant; the 2 that are suffer the winner's curse. "The p-value loses credibility if you're running in low power scenarios" (Ronny Kohavi in A/B Testing Replication Crisis, 13:18).
At Airbnb search, where Kohavi cites an 8% historical success rate, a p < 0.05 win carries about a 26% false-positive risk, not 5%. That is a different claim from Johari's fivefold Type I under peeking. Both still mean α is not the chance you are wrong after a star.
Nonsignificant Does Not Mean No Effect
Call a large p nonsignificant or a fail-to-reject, not "insignificant". GraphPad lists five explanations for a large p: no effect, a trivial effect plus small n, Type II error (1 − power), a design artifact, or optional stopping. "Didn't work" is only one of five.
A two-week test on a weak metric that "looks flat" is often a power problem, not a product truth. Peep Laja put the volume constraint in one line: most small sites cannot run even one statistically valid test per month. "Just A/B test it" is not a headline-sized task.
Segment fishing, HARKing, and month-A-then-month-B "tests" (seasonality stacked on the change) belong in the same misuse bucket as peeking. On r/AskStatistics, u/PandaJunk (Feb 2026) put the large-n edge: unless you have a proper null model, you can prove just about anything is statistically significant with enough data.
