Mathematical reasoning and assumptions
In a test, H0 and H1 are specified before analysis. A test statistic measures the discrepancy between data and the null model. A p-value is the chance of an at-least-as-extreme result assuming H0 and model conditions; it is not P(H0|data).
A Type I error rejects a true H0; in a calibrated procedure its probability is controlled by α. A Type II error fails to reject a particular false H0, with probability β. Power 1−β is the rejection probability at a specified alternative effect.
For independent normal observations with known σ, a two-sided mean z test uses z=(X̄−μ0)/(σ/√n). Sample size, true effect, noise and α influence power. A non-significant test does not establish the null; a wide interval or low power can leave meaningful effects plausible.
With multiple testing, distinguish exploratory findings from prespecified confirmatory hypotheses. A 0.05 threshold cannot substitute for effect size, interval estimates and methodological validity.
Formula
z=(X̄−μ0)/(σ/√n) p(two-sided)=2[1−Φ(|z|)] power=1−β
Worked example
- Fictional idealised example: H0: μ=70, known σ=10, n=25, observed sample mean X̄=74.
- SE=10/√25=2, z=(74−70)/2=2. Two-sided p=2[1−Φ(2)]≈0.0455.
- At prespecified α=0.05, the test rejects H0. This does not give the probability that H0 is true, the magnitude of the true effect or a guarantee of replication.
Independent exercises
- Under the same H0, σ and n, find z and approximate two-sided p for X̄=72.
- Define Type I error, Type II error and power in distinct sentences.
- With σ and true effect fixed, what usually happens to SE and power if n quadruples?
Instructor solutions are private.
Optional base R check
mu0 <- 70; sigma <- 10; n <- 25; xbar <- 74
z <- (xbar - mu0)/(sigma/sqrt(n))
p_two_sided <- 2*pnorm(abs(z), lower.tail=FALSE)
c(z=z, p_value=p_two_sided)
# Illustrative power if true mean were 74 at alpha = 0.05
se <- sigma/sqrt(n)
cut <- qnorm(0.975)*se
power <- pnorm(mu0-cut, mean=74, sd=se) +
pnorm(mu0+cut, mean=74, sd=se, lower.tail=FALSE)
power