Linear Regression

What Is Linear Regression?

Linear regression is a method used to model the relationship between an independent variable (predictor) and a dependent variable (response).

It is used to predict continuous outcomes by finding a straight line that best fits the data - showing how one variable changes in response to another.

The Regression Equation

Linear regression uses a simple equation to make predictions:

y = mx + b
  • y: The outcome or dependent variable we want to predict.
  • x: The input or independent variable used to make predictions.
  • m: The slope - how much y changes for every one-unit increase in x.
  • b: The intercept - the value of y when x = 0.

Understanding the Slope and Intercept

The slope (m) tells us the strength and direction of the relationship between x and y. A positive slope means y increases as x increases. A negative slope means y decreases as x increases.

The intercept (b) is the starting value of y when x is zero - often giving context to predictions when x is near the lower end of the range.

Example: If a regression model gives the equation y = 5x + 2, it means:

  • For each 1 unit increase in x, y increases by 5.
  • When x = 0, the predicted value of y is 2.

Worked Example

NOTE: The learning expects you to "Explain how linear regression is used ...", not to calculate. However, this worked example will go through the entire process.

Scenario: Imagine you're predicting student exam scores based on hours studied.

You collect data from 5 students:

Hours Studied (x): 2, 4, 6, 8, 10
Exam Scores (y): 55, 65, 70, 80, 95

When you plot this data, it forms a fairly straight line pattern.

Data Points Graph

If you were to manually draw a best-fit line, you might get something like this:

Data Points Graph

  • The red line is the best-fit line, extended to show the y-intercept
  • The green dashed triangle shows how the slope is calculated:
    • Rise: Change in exam score (from 55 to 65 = 10)
    • Run: Change in hours studied (from 2 to 4 = 2)
    • So, slope = rise/run = 10/2 = 5
  • The intercept - where the line hits the y-axis - is around 45

This means that the formula, based on manually drawing a line of best fit is:

ŷ = 5x + 45

However, to properly calculate the best-fit line, we use a more complex technique.

We still need to end with:

ŷ = mx + b

But we start very differently!

We'll start with the formula for the slope:

m = Σ((x - x̄)(y - ȳ)) / Σ((x - x̄)²)

Step 1: Calculate mean values

  • x̄ (mean of hours) = (2 + 4 + 6 + 8 + 10) / 5 = 6
  • ȳ (mean of scores) = (55 + 65 + 70 + 80 + 95) / 5 = 73

Step 2: Fill in the formula with the values

x y x - x̄ y - ȳ (x - x̄)(y - ȳ) (x - x̄)²
255-4-187216
465-2-8164
6700-300
88027144
10954228816
Total 190 40

Step 3: Calculate the slope (m)

m = 190 / 40 = 4.75

Step 4: Use the slope and mean values to find the intercept (b)

Use the formula b = ȳ - m × x̄

b = 73 - (4.75 × 6) = 73 - 28.5 = 44.5

Final Regression Equation:

ŷ = 4.75x + 44.5

This line gives the predicted score (ŷ) for any number of hours studied.

Predicted scores:

If x = 2  → ŷ = 4.75×2  = 54
If x = 4  → ŷ = 4.75×4  = 63.5
If x = 6  → ŷ = 4.75×6  = 73
If x = 8  → ŷ = 4.75×8  = 82.5
If x = 10 → ŷ = 4.75×10 = 92

These predicted values are close to the real exam scores - a sign of a good fit!

Assessing Model Fit: R² Score

The R² score (called "R-squared") tells us how well the regression line fits the data. It represents the proportion of variance in the dependent variable that is explained by the independent variable.

In other words, R² shows how close the model’s predictions are to the actual results - the closer R² is to 1, the better.

How is R² calculated?

R² = 1 − (Sum of Squared Errors from the model / Total Sum of Squares from the mean)

Worked Example: Calculating R²

Given Test Data & Predicted Scores:

x (Hours Studied) y (Actual Scores) ŷ (Predicted Scores)
25554
46563.5
67073
88082.5
109592

Step 1: Mean of Actual Scores (ȳ):

ȳ = (55 + 65 + 70 + 80 + 95) ÷ 5 = 365 ÷ 5 = 73

Step 2: Total Sum of Squares (TSS)

TSS = Σ (y − ȳ)²
= (55−73)² + (65−73)² + (70−73)² + (80−73)² + (95−73)²
= 324 + 64 + 9 + 49 + 484 = 930

Step 3: Sum of Squared Errors (SSE)

SSE = Σ (y − ŷ)²
= (55−54)² + (65−63.5)² + (70−73)² + (80−82.5)² + (95−92)²
= 1 + 2.25 + 9 + 6.25 + 9 = 27.5

Step 4: Calculate R²

R² = 1 − (SSE ÷ TSS)
R² = 1 − (27.5 ÷ 930) ≈ 1 − 0.0296 = 0.970

Interpretation: This means the model explains about 97% of the variation in exam scores based on hours studied.

If the model is perfect, the top and bottom numbers match - then R² = 1. If the model is no better than guessing the average, R² = 0.

  • 97% of the variation in exam scores is explained by the number of study hours.
  • The remaining 3% could be due to other factors - like sleep, nutrition, or test anxiety.

R² Interpretation Guide

  • R² = 1: Perfect fit. All actual points lie on the prediction line.
  • R² = 0: The model doesn’t explain any of the variation.
  • Higher R² values indicate the model’s predictions are close to the real outcomes.

Python Example: Predicting House Prices

import numpy as np
import matplotlib.pyplot as plt
# scikit-learn (imported as sklearn)
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

# Sample data
X = np.array([500, 800, 1000, 1200, 1500]).reshape(-1, 1)
y = np.array([150000, 200000, 240000, 270000, 310000])

model = LinearRegression()
model.fit(X, y)
y_pred = model.predict(X)
r2 = r2_score(y, y_pred)

plt.scatter(X, y, label="Actual Prices")
plt.plot(X, y_pred, color="red", label="Regression Line")
plt.xlabel("Square Footage")
plt.ylabel("House Price")
plt.legend()
plt.show()

print(f"R² Score: {r2:.2f}")

Real-World Applications of Linear Regression

Finance

In finance, linear regression can be used to estimate future returns based on historical data. For example, an investment analyst might use past stock prices as the input variable (x) and future price changes as the output (y). The regression model helps determine whether there's a consistent trend that could support future predictions.

Healthcare

Healthcare professionals might use linear regression to predict a patient's blood pressure based on variables like age and weight. In this case, age and weight are the input variables (x), and blood pressure is the outcome (y). The model helps doctors understand how these factors contribute to health outcomes and make better-informed decisions.

Retail

In retail, marketing teams may use linear regression to forecast future sales based on advertising spending. Data on ad spend over several months (x) is compared with sales figures for the same periods (y). The model reveals whether higher marketing budgets typically lead to higher sales - helping businesses plan their investments.

Education

Educators can apply linear regression to explore the relationship between study hours and exam scores. For instance, student-reported study time (x) is compared with actual test results (y) to assess whether more time spent studying leads to better performance. This can inform teaching strategies and academic support.

Limitations of Linear Regression

  • Assumes a Linear Relationship: It cannot capture curved or complex patterns in data.
  • Sensitive to Outliers: Extreme data points can skew results.
  • Not Suitable for Categorical Inputs: Linear regression only works with numerical variables.

 Key Takeaways

  • Linear regression models a straight-line relationship between input and output variables.
  • The slope shows how much the outcome changes with the input.
  • The intercept is the predicted value when the input is zero.
  • The R² score measures how well the model explains the data.
  • It is widely used in areas like finance, healthcare, retail, and education.