10.3.2: Logistic Regression

Logistic Regression

Despite its name, logistic regression is actually a classification algorithm (supervised learning) commonly used for binary classification problems – that is, predicting one of two possible classes (e.g., spam vs not-spam, churn vs not-churn). Logistic regression models the probability that an input belongs to the “positive” class (conventionally labeled 1) using the logistic (sigmoid) function.

At its core, logistic regression learns a linear combination of the input features (like linear regression does), but then applies the sigmoid function to squeeze the output into the range [0,1], interpreting it as a probability. The model form for a single input feature is:

Figure \(\PageIndex{1}\): Logistic Regression — Sigmoid Function and Decision Boundary. The S-shaped sigmoid curve maps any input value to a probability between 0 and 1, with the 0.5 threshold serving as the classification decision boundary. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

Logistic regression is popular because it is efficient, highly interpretable (coefficients can be interpreted in terms of odds ratios for the outcome), and works well when the relationship between features and the log-odds of the outcome is linear. It’s widely used for problems like predicting if a customer will buy (yes/no) or if a patient has a disease (positive/negative) based on input features. Moreover, logistic regression can be extended to multi-class classification (using techniques like one-vs-rest or the softmax function for multinomial logistic regression).

One key output of logistic regression is the probability score. For instance, a logistic model might output that a certain email has a 0.92 (92%) probability of being spam. This is useful in practice because you can tune the decision threshold depending on how you balance precision and recall. Logistic regression models can also be regularized (using L1 or L2 penalties) to prevent overfitting, similar to linear regression.

Python Implementation – Logistic Regression Example: Using scikit-learn, we can fit a logistic regression for a binary classification task. For example, if we have a dataset of labeled examples (X, y) where y is 0 or 1:

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()           # By default, uses L2 regularization
clf.fit(X_train, y_train)            # Train on labeled data
probabilities = clf.predict_proba(X_test)  # Predicted probabilities for each class
predictions = clf.predict(X_test)         # Predicted class labels (0 or 1)

Here, predict_proba would give an array of probabilities for [class0, class1] for each sample (for binary problems, one probability and its complement). The .predict() uses a default 0.5 threshold on the positive class probability to assign a class label. We can then evaluate the predictions using metrics such as accuracy or more nuanced metrics like precision, recall (discussed in a later section).

Logistic regression provides a solid baseline for classification. It’s fast to train and often performs surprisingly well if the true relationship is roughly linear in the log-odds. It also has the advantage of providing calibrated probability estimates, which many more complex models do not output as directly.