10.4.3: Confusion Matrix and Classification Metrics

Confusion Matrix and Classification Metrics

For classification models (which predict discrete classes), overall accuracy is one metric, but it often doesn’t tell the full story, especially if classes are imbalanced. A confusion matrix is a better way to visualize and evaluate classification performance. It is a table that summarizes how many instances were predicted in each class vs. the actual classes. For a binary classification (two classes, positive and negative), the confusion matrix is 2x2, with entries: - True Positives (TP): cases where the model predicted Positive and the actual was Positive. - True Negatives (TN): model predicted Negative and actual was Negative. - False Positives (FP): model predicted Positive but actual was Negative (a “Type I error” or false alarm). - False Negatives (FN): model predicted Negative but actual was Positive (a “miss” or “Type II error”).

Figure \(\PageIndex{1}\): Confusion Matrix for Binary Classification. A 2x2 matrix showing True Positives, True Negatives, False Positives, and False Negatives, with derived metrics (accuracy, precision, recall, F1 score) annotated alongside. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

For multi-class classification, the confusion matrix extends to an N x N table (where N is number of classes), and precision/recall can be computed for each class (interpreting each class in turn as “positive” vs all others as “negative”) [8]. We then often report macro-averaged (simple average across classes) or weighted-average (average weighted by class frequency) precision, recall, F1, etc. Scikit-learn’s classification_report conveniently provides these metrics.

When to use which metric? It depends on the problem. For example, in a medical test, recall (sensitivity) might be critical – you don’t want to miss positive cases (disease), even if that means some false alarms, so you’d prioritize recall (and also consider precision or use specificity separately). In contrast, for a spam filter, precision might be more important – you don’t want to classify legitimate emails as spam (false positive) and accidentally drop them. F1 is useful when you seek a balance and the class distribution is uneven.

Python Implementation – Confusion Matrix & Metrics: Continuing from a trained classifier clf and test data:

from sklearn.metrics import confusion_matrix, classification_report
y_pred = clf.predict(X_test)
cm = confusion_matrix(y_test, y_pred, labels=[1,0])  # assuming 1=Positive, 0=Negative
print("Confusion Matrix:\n", cm)
print("Classification Report:\n", classification_report(y_test, y_pred))

The confusion matrix cm will be a 2x2 array like in the figure (with the specified ordering of labels). The classification_report will print precision, recall, F1 (and support counts) for each class and averages. For example it might output something like:

For regression models, different metrics are used (like Mean Squared Error, Mean Absolute Error, R² score), but the concept of splitting data and cross-validation for honest evaluation remains the same.