Regression lines and predictions – Bivariate Statistics (Senior 5 Mathematics) | Ganzaa

Bivariate Statistics

Lessons

Senior 5 · Mathematics

Bivariate Statistics

4 lessons

Lesson 4 of 4

Regression lines and predictions

45 min

A line everyone agrees on

In lesson 1 each pupil drew a line of best fit by eye, and each line gave a slightly different prediction. Statisticians, economists and agronomists need one line that anyone gets from the same data. The answer, found by Legendre and Gauss around 1805, is the line that makes the squared errors as small as possible: the least-squares regression line. It is how a cooperative predicts its harvest from rainfall, or a school its exam results from mock marks.

Definition

The regression line of $y$ on $x$ is the line $y = a + b x$ that minimises the sum of the squared vertical distances (residuals) $\sum (y_i - a - bx_i)^2$ . It is used to predict $y$ from $x$ :

y - \bar{y} = \frac{\operatorname{cov}(X, Y)}{\operatorname{Var}(X)}\,(x - \bar{x}), \qquad b_{yx} = \frac{\operatorname{cov}(X, Y)}{\sigma_X^2} .

The regression line of $x$ on $y$ minimises the squared horizontal distances and is used to predict $x$ from $y$ :

x - \bar{x} = \frac{\operatorname{cov}(X, Y)}{\operatorname{Var}(Y)}\,(y - \bar{y}), \qquad b_{xy} = \frac{\operatorname{cov}(X, Y)}{\sigma_Y^2} .

Important

Why these lines minimise the squares. For a fixed slope $b$ , $S(a) = \sum (y_i - bx_i - a)^2$ is a quadratic in $a$ , smallest when $a$ is the mean of the numbers $y_{i} - b x_{i}$ : $a = \bar{y} - b\bar{x}$ . So the best line passes through $(\bar{x}, \bar{y})$ . Then

\frac{1}{n}\sum\left[(y_i - \bar{y}) - b(x_i - \bar{x})\right]^2 = \operatorname{Var}(Y) - 2b\operatorname{cov}(X, Y) + b^2\operatorname{Var}(X),

a quadratic in $b$ whose vertex is at $b = \frac{\operatorname{cov}(X, Y)}{\operatorname{Var}(X)}$ .

Properties. Both regression lines pass through the mean point $(\bar{x}, \bar{y})$ . Their slopes $b_{yx}$ and $b_{xy}$ have the same sign as $r$ , and

b_{yx} \times b_{xy} = \frac{\operatorname{cov}^2}{\sigma_X^2\sigma_Y^2} = r^2 .

The two lines coincide when $r = \pm 1$ , and the closer $∣ r ∣$ is to 1, the smaller the angle between them. $r^{2}$ , the coefficient of determination, is the fraction of the variation in $y$ that the line explains.

Worked example 1: the line of $y$ on $x$ and a prediction

Find the regression line of $y$ on $x$ for the revision data, and predict the mark of a pupil who revises 10 hours.

Solution. From lesson 2, $\bar{x} = 7$ , $\bar{y} = 30$ , $\operatorname{cov} = 29$ , $\operatorname{Var}(X) = 11.5$ . So $b_{yx} = \frac{29}{11.5} \approx 2.522$ and

y - 30 = 2.522(x - 7) \iff y = 2.52x + 12.35 .

At $x = 10$ : $y \approx 2.522 \times 10 + 12.35 \approx 37.6$ marks. The slope says each extra hour of revision goes with about 2.5 more marks, on average.

Worked example 2: predicting $x$ needs the other line

Estimate how long a pupil who scored 40 revised.

Solution. To predict $x$ , use the line of $x$ on $y$ : $b_{xy} = \frac{29}{77} \approx 0.377$ , so $x - 7 = 0.377 (y - 30)$ , that is $x = 0.377 y - 4.30$ . At $y = 40$ : $x \approx 10.8$ hours. (Solving the $y$ -on- $x$ line for $x$ would give $\frac{40 - 12.35}{2.52} \approx 11.0$ : close here because $r$ is close to 1, but in general different and not the least-squares estimate.)

Check. $b_{yx}b_{xy} = \frac{29}{11.5} \times \frac{29}{77} \approx 0.950 = r^2$ , and $\sqrt{0.950} \approx 0.97$ , the $r$ of lesson 3. ✓

Worked example 3: from two lines back to the data

The regression lines of a data set are $y = 0.8 x + 12$ (of $y$ on $x$ ) and $x = 0.45 y + 1$ (of $x$ on $y$ ). Find $\bar{x}$ , $\bar{y}$ and $r$ .

Solution. Both lines pass through the mean point. Substituting: $x = 0.45 (0.8 x + 12) + 1 = 0.36 x + 6.4$ , so $0.64 x = 6.4$ , $\bar{x} = 10$ and $\bar{y} = 20$ . Then $r^2 = 0.8 \times 0.45 = 0.36$ , and $r = + 0.6$ because both slopes are positive.

Worked example 4: when a prediction should not be trusted

Use the line of worked example 1 to predict the mark for 25 hours of revision, and comment.

Solution. $y \approx 2.522 \times 25 + 12.35 \approx 75.4$ , which is impossible for a test out of 50. The data cover 2 to 12 hours; predicting inside that range (interpolation) is reasonable, but far outside it (extrapolation) the straight-line pattern need not continue.

Warning

Predict $y$ from $x$ with the line of $y$ on $x$ , and $x$ from $y$ with the line of $x$ on $y$ .
$b_{yx}$ divides by $\operatorname{Var}(X)$ , not by $\sigma_X$ and not by $\operatorname{Var}(Y)$ .
Beware of extrapolation, and of predictions when $∣ r ∣$ is small: the line then explains little of the variation.

Tip

Given two regression lines but not told which is which, try both assignments: the right one gives $b_{yx}b_{xy} = r^2 \le 1$ .

ExamSolutions: Regression, Least Squares Method for y on x (about 6 min). A worked example of the regression line found from the column totals, then used for a prediction.

✦

Practice

Questions 1 to 3 are routine; 4 to 6 need more care; 7 asks for a judgement.

For $(1, 2), (2, 4), (3, 5), (4, 4), (5, 7)$ , with $\operatorname{cov} = 2$ , $\operatorname{Var}(X) = 2$ , $\operatorname{Var}(Y) = 2.64$ , find the regression line of $y$ on $x$ and estimate $y$ when $x = 3.5$ .
For the same data, find the regression line of $x$ on $y$ .
For the motorcycles of lesson 3 ( $\bar{x} = 8$ , $\bar{y} = 23$ , $\operatorname{cov} = -15.2$ , $\operatorname{Var}(X) = 8$ ), find the line of $y$ on $x$ and estimate the value of a 7-year-old motorcycle.
$r = 0.6$ , $\sigma_X = 2$ , $\sigma_Y = 5$ , $\bar{x} = 10$ , $\bar{y} = 30$ . Find the regression line of $y$ on $x$ and predict $y$ for $x = 12$ .
The two regression lines of a data set are $3 x + 2 y = 26$ and $6 x + y = 31$ . Find the mean point, decide which line is the regression of $y$ on $x$ , and find $r$ .
Show that when $r = 0$ the two regression lines are perpendicular, one horizontal and one vertical.
A school finds the line $y = 0.9 x + 8$ between mock marks $x$ and exam marks $y$ , from mocks between 30 and 80. Should it use the line to predict the exam mark of a pupil who scored 5 in the mock? Explain.

Show answersHide answers

$b_{yx} = 1$ : $y - 4.4 = x - 3$ , so $y = x + 1.4$ ; at $x = 3.5$ , $y = 4.9$ .
$b_{xy} = \frac{2}{2.64} \approx 0.758$ : $x - 3 = 0.758 (y - 4.4)$ , so $x \approx 0.758y - 0.333$ .
$b = - 1.9$ : $y = - 1.9 x + 38.2$ ; at $x = 7$ , $y = 24.9$ , about 2,490,000 FRW.
$b_{yx} = r\frac{\sigma_Y}{\sigma_X} = 1.5$ : $y = 1.5 x + 15$ ; at $x = 12$ , $y = 33$ .
Solving gives $(\bar{x}, \bar{y}) = (4, 7)$ . Taking $3 x + 2 y = 26$ as $y$ on $x$ : $b_{yx} = -1.5$ , $b_{xy} = -\frac{1}{6}$ , product $0.25 \le 1$ . The other choice gives $(-6) \times \left(-\frac{2}{3}\right) = 4 > 1$ , impossible. So $r = - 0.5$ (negative, like the slopes).
$r = 0$ means $\operatorname{cov} = 0$ , so the lines are $y = \bar{y}$ (horizontal) and $x = \bar{x}$ (vertical).
No: 5 is far outside the range 30 to 80 used to find the line, so the prediction $0.9 \times 5 + 8 = 12.5$ is an extrapolation and may be badly wrong.

1A line everyone agrees on

A line everyone agrees on