How the p-value is calculated
The p-value is the probability, assuming the null hypothesis H₀ is true, of getting a test statistic at least as extreme as yours. "Extreme" depends on the tail you choose:
Right-tailed: p = P(T ≥ t)
Two-tailed (z, t): p = 2 × P(T ≥ |t|)
Two-tailed (χ², F): p = 2 × min(P(T ≤ t), P(T ≥ t))
The areas come from the exact distribution functions: the normal curve via the error function, t and F via the regularized incomplete beta function, and chi-square via the regularized incomplete gamma function. Each is computed with series and continued fractions to about 9 or more significant digits, so you don't need a printed table or to round df to a table row.
Worked examples
- z = 1.96, two-tailed: p = 2 × 0.025 = 0.05 (0.04999579 exactly), right on the usual cutoff.
- t = 2.1 with 15 df, two-tailed: p = 0.0531. That's just above 0.05, so you fail to reject H₀ at α = 0.05, though it's close.
- χ² = 10 with 4 df (a goodness-of-fit or independence test, which is right-tailed): p = 0.0404, so reject H₀ at α = 0.05.
- F = 4.26 with 2 and 9 df (one-way ANOVA, right-tailed): p = 0.0499.
Which tail should I use?
Use two-tailed when the alternative hypothesis is "not equal" (≠). Use left- or right-tailed only when you decided before seeing the data that only one direction matters. Chi-square tests of fit and independence, and ANOVA F tests, are right-tailed. Choosing a tail after looking at the data makes the p-value misleading.
What a p-value does and doesn't mean
The American Statistical Association's 2016 statement on p-values sets out these principles:
- A p-value shows how incompatible the data are with a specified model, such as the null hypothesis.
- It is not the probability that the null hypothesis is true, or that the result happened by chance alone.
- Conclusions shouldn't rest only on whether p crosses a threshold like 0.05. A p of 0.049 and one of 0.051 carry almost the same evidence.
- A p-value doesn't measure the size or importance of an effect. With a huge sample, a trivial difference can be "significant". Report an effect size and a confidence interval too.
- By itself, a p-value isn't a good measure of evidence for a model or hypothesis.
Limitations
- The p-value is only valid if the test's assumptions hold, such as independent observations and, for t and F tests, roughly normal data.
- Running many tests and reporting only the significant ones inflates false positives. Adjust α (for example with a Bonferroni correction) when testing many hypotheses.
- Degrees of freedom can be fractional (as in Welch's t-test) and go up to 1,000,000.
Sources
- Wasserstein and Lazar (2016), The ASA Statement on p-Values: Context, Process, and Purpose, The American Statistician 70(2)
- NIST/SEMATECH e-Handbook: Critical values and p-values
- NIST/SEMATECH e-Handbook: t distribution
- NIST/SEMATECH e-Handbook: Chi-square distribution
- NIST/SEMATECH e-Handbook: F distribution
- NIST Digital Library of Mathematical Functions §8.2: Incomplete gamma functions