10.3.1: Linear Regression

Linear Regression

Linear regression is a fundamental algorithm for predictive modeling of continuous numeric outcomes. It assumes a linear relationship between the input features (independent variables) and the target (dependent variable). The simplest form, simple linear regression, fits a straight line to data with one independent variable, often expressed as:

where is the input and   is the predicted output. Here is the intercept and the slope of the line. In multiple linear regression, the model extends to multiple features, but still maintains a linear form (a weighted sum of features). The model’s parameters (coefficients) are typically estimated using the least squares method, which finds the line (or hyperplane in higher dimensions) that minimizes the sum of squared errors between predictions and actual values.

Linear regression predicts a numeric value. For example, one could predict a house’s price based on its square footage, location, number of bedrooms, etc. If we plot known house prices against square footage, linear regression tries to draw a straight line through the data points that best approximates their relationship. Despite its simplicity, linear regression can be very effective for many tasks and provides an easily interpretable model (coefficients indicate how much the outcome changes with a feature, holding others constant). However, it is limited to linear relationships – it won’t capture complex nonlinear patterns.

Figure \(\PageIndex{1}\): Linear Regression — Line of Best Fit. A scatter plot showing data points with a fitted regression line, illustrating how linear regression minimizes the distance between observed values and the predicted line. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

From a predictive analytics standpoint, linear regression serves as a baseline model and a building block for more complex methods. It’s often one of the first models tried due to its interpretability and ease of training. If the relationship in the data is roughly linear, this model will perform well. If not, analysts might transform features or move to more complex models.

Python Implementation – Linear Regression Example: Using scikit-learn, we can quickly fit a linear regression model. In the snippet below, we assume X_train, X_test, y_train, y_test are prepared datasets (with X possibly multivariate features and y a continuous target):

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)               # Train the model on training data
predictions = model.predict(X_test)       # Predict on new (test) data
print(model.coef_, model.intercept_)      # Print learned coefficients and intercept

This code creates a linear regression model, fits it to the training data, and produces predictions for the test set. We could then evaluate these predictions (e.g., by computing mean squared error or   score on y_test). Linear regression models in scikit-learn automatically handle multiple features and include regularization variants like Ridge and Lasso for more complex scenarios.