16: Classifier

Gather Data

We use the same cdc dataset as in the previous chapter. The original dataset contains 20,000 rows. Because KNN compares every testing row with every training row, classifying all 20,000 rows takes a long time. We will randomly select 3,000 rows so that the classifier runs more quickly while still using a large and varied sample of the data.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# read input data from file
url = "https://raw.githubusercontent.com/DeAnzaDataScience/CIS11/refs/heads/main/datasets_notes/cdc.csv"
health = pd.read_csv(url)
# random sample of 3000 rows
health = health.sample(n=3000, random_state=1)

Output?

Prepare Data

We take the same steps to prepare data as in Chapter 15: convert gender values to numbers, change height and weight to standard units.

# select only the features we want to use for classification and form a new dataset
data = health[['height', 'weight', 'gender']].copy()
# convert gender to numeric values
data['gender'] = data['gender'].map({'m': 0, 'f': 1})
# convert height and weight to standard units
data['height'] = (data['height'] - data['height'].mean()) / data['height'].std()
data['weight'] = (data['weight'] - data['weight'].mean()) / data['weight'].std()
    

Output?

Create Training and Testing Data

We use random sampling to create the two datasets. First we randomly shuffle all the rows of the dataset, in the same way that we randomly shuffle data for hypothesis testing in Chapter 10. Then we assign some of the rows to the training dataset and the remaining rows to the testing dataset. For this chapter we will use half of the rows for training and half for testing.

# shuffle all the rows
shuffled_data = data.sample(frac=1)
# take first half for training set
training_set = shuffled_data.iloc[:len(shuffled_data)//2].copy()
# rename the gender column to _class, a requirement for our classifier
training_set = training_set.rename(columns={'gender':'_class'})
# take second half for testing set
testing_set = shuffled_data.iloc[len(shuffled_data)//2:].copy()
# rename the gender column to _class, a requirement for our classifier
testing_set = testing_set.rename(columns={'gender':'_class'})

Output?

We confirm the training dataset by plotting the height vs weight.

plt.figure(figsize=(4,4))

groups = training_set.groupby('_class')
for name, group in groups:
    plt.scatter(group['height'], group['weight'], alpha = 0.4, label=name)
plt.xlabel('Height')
plt.ylabel('Weight')
plt.legend()
plt.title('Training Data')
plt.grid()
plt.show()

Output?

The scatterplot shows only the training data. When KNN makes a prediction, it searches only the training dataset for the nearest neighbors. The testing data are not used to make predictions because we want them to represent new, unseen data. After the predictions have been made, we compare them with the actual classes in the testing dataset to measure the classifier's accuracy.

Find Prediction Accuracy

We store the actual data and the predicted data in a DataFrame and print the first five rows.

results = pd.DataFrame({'Actual': actual, 'Predicted': predicted})
results.head(5)

Output?

Since the data is randomly shuffled before being divided into the training and testing sets, the results above may differ slightly with each run of this example. The predicted classes for individual data points may be different, but the overall accuracy should be similar.

We calculate the percent accuracy by finding the ratio of correct predictions over the total predictions.

# add all the rows where Actual is the same as Predicted
correct = (results.Actual == results.Predicted).sum()
# find the ratio of correct predictions over total predictions as a percentage
print("Accuracy:", round(correct / len(results) * 100), "%")

Output?

Accuracy: 84 %

The classifier correctly predicted the gender for a high percentage of the people in the testing dataset.

Looking at the female and male ratios in the starting dataset of 3000 people, we find the count and the percentage of f and m values:

display(health["gender"].value_counts())
display(health["gender"].value_counts(normalize=True) * 100)

Output?

gender
f    1570
m    1430
Name: count, dtype: int64

gender
f    52.333333
m    47.666667
Name: proportion, dtype: float64

url = "https://raw.githubusercontent.com/DeAnzaDataScience/CIS11/refs/heads/main/datasets_notes/cars.csv"
cars = pd.read_csv(url)
display(cars.head())
print("Number of rows:", cars.shape[0])

Output?

Number of rows: 392

car_data = cars[["horsepower", "weight", "cylinders", "mpg"]].copy()

Output?

We want to predict whether a car is fuel efficient. Since fuel efficiency is measured by mpg, we will use mpg to define our classes. However, a classifier can only predict categorical data, and mpg is continuous quantitative data, so we convert mpg into two categories:

• 0: mpg < 25, not fuel efficient
• 1: mpg >= 25, fuel efficient

# Convert mpg to binary classification: 0 if < 25, 1 if >= 25
car_data['mpg'] = (car_data['mpg'] >= 25).astype(int)
car_data.head()

Output?

In the previous example we used two features, height and weight, to predict gender. In this example, we will use three features: horsepower, weight, and cylinders to predict whether a car is fuel efficient. Even though the dataset and the number of features are different, the KNN algorithm does not change. The same classifier works without modifications on completely different datasets, even if they have a different number of features.

Although we first developed the algorithm using two features, the code is not limited to two dimensions. The distance_from_point function works with arrays of any length, so the same classifier can be used with datasets that have two features, three features, or many more.

We continue preparing the data:

• change the mpg label to _class as required by our classifier
• convert all the features to standard units

# rename the mpg column to _class
car_data = car_data.rename(columns={'mpg': '_class'})

Output?

# convert all the features to standard units
car_data['horsepower'] = (car_data['horsepower'] - car_data['horsepower'].mean()) / car_data['horsepower'].std()
car_data['weight'] = (car_data['weight'] - car_data['weight'].mean()) / car_data['weight'].std()
car_data['cylinders'] = (car_data['cylinders'] - car_data['cylinders'].mean()) / car_data['cylinders'].std()  
     

Output?

car_data.head()

Output?

We divide the data into the training and testing sets.

shuffled_data = car_data.sample(frac=1)
training_set = shuffled_data.iloc[:len(shuffled_data)//2].copy()
testing_set = shuffled_data.iloc[len(shuffled_data)//2:].copy()

Output?

We save the _class column from the testing_set. Then we remove the _class column from the testing data.

# save the actual target or `_class`, then drop `_class` from the testing_set
actual = testing_set._class
test_set = testing_set.drop(columns='_class')
test_set.head()

Output?

Now that the data has been prepared, we use the classifier to predict the fuel efficiency of the cars.

predicted = []
for index, row in test_set.iterrows():   
    predicted.append(classify(row, training_set, 5))

# combine the actual and predicted into a DataFrame, 
# and find the number of correct predictions
correct = (actual == predicted).sum()
print("Accuracy:", round(correct / len(actual) * 100))

Output?

Accuracy: 84

We see that the classifier has high accuracy on a completely different dataset. This demonstrates that the KNN algorithm is not tied to a particular dataset and can be reused to solve different classification problems.

Summary

In this chapter, we built a KNN classifier step by step by combining several functions into a single classifier that predicts the class of new data. We used training and testing datasets to measure its accuracy and saw that the same classifier can be applied to completely different datasets without changing the algorithm.

KNN is one of many machine learning algorithms. Machine learning is a foundation of modern AI because it enables computers to learn patterns from data and make predictions.