10.3.3: Decision Trees

Decision Trees

A decision tree is a versatile supervised learning algorithm used for both classification and regression tasks. It works by recursively partitioning the feature space into regions, based on a series of decision rules learned from the data. The model is structured as a tree: starting from a root node, which asks a question about one of the features, the data splits into branches based on the answer. Each branch leads to another node (with another question) or to a leaf node, which represents a final prediction (a class label for classification, or a numeric value for regression).

Think of a decision tree like an interview process: at each step you ask a question, and the next question depends on the answer. For example, a decision tree for a classification problem of whether to play tennis might start with “Is it sunny?”. If yes, go down one branch (then maybe ask about humidity), if no, go down another branch (maybe ask if it’s windy), and so on. Each question/node typically involves a feature threshold (e.g., “humidity > 75%?”). The tree algorithm chooses these splits by trying to maximize the separation of the data at each step according to some criteria. In classification, common criteria include Gini impurity or information gain (entropy) – measures of how “pure” (homogeneous) the resulting nodes are. In regression, typically variance reduction (e.g., minimizing mean squared error in the splits) is used.

Figure \(\PageIndex{1}\): Decision Tree Structure — Classification Example. A sample decision tree for the Iris dataset showing how sequential feature-based questions partition data into predicted classes at the leaf nodes. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

Decision trees have the advantage of producing models that are very interpretable: one can follow the path from root to leaf to see why a prediction was made (this transparency is valued in many applications). They can capture nonlinear relationships and handle feature interactions effectively. Trees also naturally handle mixed data types (numerical and categorical) and don’t require feature scaling. However, a single decision tree can easily overfit – it might keep branching to perfectly fit the training data noise. This is why trees are often pruned (cut back) or used in ensembles (next section). On their own, they may not generalize well if grown too deep.

In the figure above, we see a truncated decision tree for classifying Iris species. At the root, the tree splits on petal width. One branch leads to a leaf predicting setosa (with 100% purity for that class). The other branch (petal width > 0.8) leads to further splits to differentiate between versicolor and virginica based on petal length. This hierarchy of if-then rules is learned from the data: the tree algorithm found those particular thresholds most informative for splitting the species.

Python Implementation – Decision Tree Example: Using scikit-learn, training a decision tree is straightforward. For example, to fit a classification tree on the Iris dataset:

from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=3, random_state=0)  # limit depth to prevent overfitting
clf.fit(X_train, y_train)
preds = clf.predict(X_test)

Parameters like max_depth (maximum levels in the tree), min_samples_split (minimum samples to justify further splitting a node), and min_samples_leaf can be tuned to prune the tree and combat overfitting. We could visualize the tree using sklearn.tree.plot_tree or export it to text to interpret the structure (as illustrated in the figure above).

Decision trees serve as the building blocks for more sophisticated ensemble methods. On their own, they are fast and interpretable, but a single tree may not be the most accurate predictor if used in isolation, especially for complex datasets where a greedy splitting procedure might not find the global optimal tree. Nonetheless, they remain a cornerstone of machine learning – easy to understand, implement, and providing the intuition behind many advanced models.