Mathematical reasoning and assumptions
Covariance measures how deviations from two sample means move together: cov(X,Y)=Σ(x_i−x̄)(y_i−ȳ)/(n−1). It depends on units, whereas correlation r=cov(X,Y)/(s_X s_Y) standardises to between −1 and +1.
Simple linear regression ŷ=a+bx minimises the sum of squared residuals (y−ŷ)². The slope b=Σ(x_i−x̄)(y_i−ȳ)/Σ(x_i−x̄)² =cov(X,Y)/s_X², and intercept a=ȳ−bx̄. A slope cannot be fitted if X has no variation.
A residual e_i=y_i−ŷ_i records observation minus fitted value. R² describes the fraction of sample Y variation explained by the fitted model. In simple regression with an intercept, R²=r², a statement about sample fit, not causality.
A large slope or R² in observational data does not prove X causes Y. Inspect outliers, nonlinear patterns, changing residual spread and extrapolation beyond observed values.
Formula
b = Σ[(x−x̄)(y−ȳ)] / Σ[(x−x̄)²] a=ȳ−bx̄ e=y−ŷ
Worked example
- For fictional points (1,2), (2,4), (3,5), x̄=2 and ȳ=11/3≈3.667.
- Sum of cross-deviations is 3; sum of squared X deviations is 2. Therefore b=3/2=1.5, a=11/3−1.5×2=2/3≈0.667.
- At x=2, ŷ≈3.667 and residual=4−3.667≈+0.333. This fit describes the relationship; it cannot establish causality.
Independent exercises
- Find slope and intercept for points (1,2),(2,4),(3,6).
- For fitted ŷ=7 and observed y=9, find the residual and interpret its sign.
- For simple regression with an intercept and r=0.8, find R². Why is this not a causal percentage?
Instructor solutions are private.
Optional base R check
x <- c(1,2,3)
y <- c(2,4,5)
fit <- lm(y ~ x)
coef(fit)
residuals(fit)
c(covariance=cov(x,y), correlation=cor(x,y), r_squared=summary(fit)$r.squared)
# Association and fit do not prove causation.