5.7: Handling Outliers in EDA

Handling Outliers in EDA

Outliers are data points that deviate markedly from the rest of the dataset. They may be extreme values or just not follow the pattern of others. Identifying outliers is a critical part of EDA because outliers can skew analysis, affect summary statistics (like mean and standard deviation), and potentially indicate data quality issues or interesting rare events.

Detecting Outliers: EDA employs both graphical and quantitative methods to spot outliers. We have already mentioned visual tools: box plots naturally highlight outliers as individual points beyond the whiskers. A histogram or density plot might show outliers as isolated bars or a long tail. Scatter plots can reveal outliers, especially in relationships (e.g., one point far away from an apparent trend line). In addition, we can use statistical rules of thumb. Two common approaches are:

• The IQR rule: Points more than 1.5×IQR below the first quartile or above the third quartile are often labeled outliers (and “extreme” outliers sometimes defined at 3×IQR). This is the rule behind boxplot whiskers. It’s non-parametric and works well for many distributions.
• The Z-score or standard deviation rule: Compute how many standard deviations each point is from the mean. If a value has |Z| > 3 (for example) it could be considered an outlier (assuming a roughly normal distribution, 99.7% of data lie within ±3σ by the 68-95-99.7 rule). However, using Z-scores can be misleading if the distribution is not normal or if the mean/std themselves are pulled by outliers. A robust alternative is to use median and MAD (median absolute deviation) to compute robust Z-scores.

EDA might also incorporate more sophisticated outlier detection for multivariate data, like the Mahalanobis distance (for detecting outliers in a combination of variables) or clustering-based approaches. But often, simple 1D or 2D checks suffice to flag candidates.

Investigating Outliers:

When an outlier is identified, the first question is: Is this point a data error or a genuine observation? EDA is the time to dig into that. For example, if you have an outlier where a person's age is recorded as 300, that is clearly an error (perhaps meant 30.0 or a placeholder code). Such errors should be corrected or removed. On the other hand, if you have a value that is just very high but possible (e.g., a very high income in a dataset of salaries because a CEO is included), that's a legitimate outlier. During EDA, one should ideally trace outliers back to raw data if possible, to see if they might be typos, unit mismatches (feet vs meters giving an outlandish height, etc.), or truly exceptional cases.

Handling Outliers:

Once identified and deemed genuine or error, the strategy can vary:

• Exclude or Filter: If an outlier is a data entry error or not relevant to the analysis goal, it may be excluded. For example, if analyzing incomes of a general population, one or two billionaires might so dominate scale that one could choose to exclude them to focus on the typical range (noting that you did so).
• Transform: Sometimes a transformation can lessen the impact of outliers. For instance, applying a logarithmic transform to a right-skewed distribution will pull in high-end outliers and compress the scale, allowing the bulk of data to be seen in more detail. In EDA, one might take a log of a variable and re-plot a histogram to see if it reveals structure better (often it does for financial or population data).
• Analyze Separately: Outliers can be insights! It might be worth analyzing them on their own. For example, if a few specific days have extremely high web traffic due to a one-time event, they are outliers in a time series – but rather than drop them, an analyst would likely want to highlight those days and explain them, perhaps by adding a note ("this spike was Black Friday"). In EDA, noting the context (dates, categories, etc.) of outliers can be very informative.
• Impute or Cap: In some analyses, one might "winsorize" data – i.e., cap extreme values at a certain percentile. For instance, replace all values above the 99th percentile with the 99th percentile value. This keeps them in analysis but reduces their impact. This is a decision that EDA can inform: if you see a few points way off that are valid but problematic for modeling (like extremely high leverage points), you might consider capping them in the modeling stage. EDA would document that "the top 1% of values range from X to Y, which is a huge jump; for modeling, we might cap at X."

Figure \(\PageIndex{1}\): Outlier Detection and Decision Framework. This flowchart guides the analyst through the process of detecting outliers, investigating their origin, and choosing an appropriate handling strategy. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

The key principle is to ensure outliers do not unduly influence your interpretation without you realizing it. EDA brings them to light so that any decisions regarding them are explicit. One should report in the EDA or data cleaning report which outliers were removed or altered, to maintain transparency.

Example: In a dataset of test scores (0–100 range for most students), suppose one value is 900. EDA quickly shows this outlier in a histogram (a bar far to the right) and the mean is strangely inflated. Investigating the data, you find that 900 was a typo (extra zero). The EDA step would recommend correcting that to 90 (or treating it as missing if unsure). As another example, in a dataset of property prices, all houses might be under \$1 million except one that is \$20 million. That one sale might be a luxury mansion that is legitimately in the data but not typical. EDA might suggest doing analyses with and without that point to see its effect. The mean price with it might be much higher than without; the median might be unaffected. If your interest is in typical properties, you might exclude it going forward, but note its existence as a special case.

To quote an EDA best-practice: “Above all, do not blindly throw away outliers; understand them.” Sometimes outliers are the most interesting part of the dataset – they might represent novel phenomena or important exceptions. Other times, they are just noise or errors. EDA helps you discern which is which, using both statistical tools (like z-scores or IQR) and domain knowledge (would we expect such a value? what could cause it?).

EDA plays a pivotal role in the feature selection and data preprocessing stages of a project. Feature selection is about identifying which variables (features) will be useful predictors or explanatory variables in modeling, and which might be dropped or transformed. Data preprocessing involves cleaning and transforming raw data into a form that can be effectively used by models. The insights gained during EDA directly inform these decisions.

Identifying Redundant or Irrelevant Features:

Through EDA, we often find that some features carry similar information. For example, you might have both "Age" and "Year of Birth" in a dataset – they are perfectly correlated (once Year is converted to Age). EDA would catch this through a correlation matrix or scatter plot; seeing a correlation of -0.99, you'd recognize one is redundant. Another case: two survey questions might overlap (e.g., one question about happiness on scale 1–5 and another about satisfaction on 1–5 could turn out nearly identical in responses). Such redundancy suggests you can remove one of the features without losing much information, simplifying your model (and reducing multicollinearity). EDA helps support the selection of appropriate variables by revealing these interdependencies.

EDA also flags irrelevant features – those that show little to no variation or no relationship with the target of interest. For instance, if you're doing a predictive task and one feature is almost constant (say, 95% of entries have the same value), that feature likely has no predictive power and can be dropped (plus it might indicate a data collection quirk). If a feature shows no correlation or differing distribution across any other variables or the outcome, it may not be useful. However, one must be careful: lack of simple correlation does not always mean a feature is useless (it could have non-linear effects or be important in combination with others). Still, EDA can hint that some features are strong candidates for removal or for further investigation.

Feature Transformations:

EDA often suggests certain transformations of features to improve analysis or modeling. For example:

• Skewed distributions: If EDA shows a variable is highly skewed (e.g., income, population counts, etc.), a log transform or square-root transform might normalize it. Many statistical models perform better with approximately normal or symmetric inputs, so EDA would prompt such a transform. The decision comes from seeing the histogram and possibly a Q-Q plot in EDA.
• Categorical encoding: EDA of categorical variables might show some categories have very few observations. The analyst might decide to combine rare categories or drop them. For instance, a "Country" variable might have 90% "USA" and a long tail of many countries with 1 or 2 records each. EDA reveals this distribution, and the preprocessing decision could be to group those tiny categories into an "Other" bucket to avoid noise.
• Interactions: Sometimes EDA reveals that a combination of features is important, suggesting creation of a new feature. For example, maybe neither "Age" nor "Income" alone correlates strongly with a target, but EDA might show that young age & high income together is a segment with special behavior. That could lead you to create an interaction feature or at least be aware to include an interaction term in a model.

Handling of missing and outliers: As already discussed, EDA guides how to impute missing values and treat outliers. These are key preprocessing steps. For instance, if EDA finds a certain variable has 30% missing, you might choose to drop that variable entirely (feature selection) if it’s not critical, rather than imputing. Or if an outlier is determined to be an error, you’ll remove or fix it in the preprocessing stage. All these decisions stem from the EDA findings.

Feature Scaling: EDA will show the ranges of variables. If one variable ranges from 1 to 1000 and another from 0 to 1, one may decide to scale features (normalize or standardize) before modeling, especially for algorithms that are sensitive to scale. For example, if you're using k-nearest neighbors or gradient descent-based models, features should typically be on comparable scales. EDA ensures you're aware of these differences. You might see from df.describe() that one attribute has a mean of 5 and std of 2, while another has mean 500 and std 300 – a flag that scaling should be considered. Sometimes a quick min-max normalization in EDA (just to see all features on [0,1] range) can also help visualize multivariate relationships without one dominating due to scale.

Feature Selection for Modeling:

If the analysis is headed towards a predictive model, EDA helps narrow down the features to feed into that model. One might use EDA to perform a rough feature importance analysis by exploring how each predictor relates to the outcome. For classification, that could be comparing distributions of a feature across classes (does it separate classes well?). For regression, seeing scatter plots or calculating correlations with the target can hint at predictive power. For example, if one variable shows no difference between cases that had an event vs those that didn't, it likely won't contribute to a classifier. Conversely, if another variable shows a big separation, it's a prime candidate to include. These are informal assessments but very useful to guide model building. In some cases, analysts use techniques like stepwise plots or partial dependence even in EDA to see potential influence of features on outcomes.

EDA-driven feature selection is not a formal test – it's possible to be misled by randomness (especially in small data). It’s also during EDA that you might detect multicollinearity (as mentioned) which would inform you to not use two redundant features together in a model.

Example: Imagine we’re building a model to predict house prices. EDA might reveal: - Square footage has a strong positive relationship with price (scatter plot upward trend) – definitely include this feature. - Year built has a moderate trend (newer houses slightly pricier), but EDA also shows year built is correlated with square footage (newer houses tend to be larger). We might still include both, but be cautious of multicollinearity, or combine them in a single feature (like price per square foot vs age). - "Number of bathrooms" correlates with price, but it’s almost entirely because bigger houses have more bathrooms. Maybe the combination of [bathrooms, bedrooms, size] needs to be considered together. We might decide to use an engineered feature like bathrooms per square foot if EDA shows that density matters. - "ZIP code" (location) might show dramatic price differences – EDA would have us treat ZIP as a categorical feature with potentially high importance. - A feature like "Has pool (yes/no)" might show in EDA that houses with pools are on average 20% more expensive. That insight means this binary feature is valuable. - EDA also might show that one feature, say "school quality rating", has a lot of missing values in this dataset. If we can’t easily get those, maybe we drop it to avoid complexity.

In short, EDA informs a blueprint for feature engineering: which features to keep, which to drop, which to transform, and which new features to create. This is why EDA is often considered as having a close interplay with data preprocessing – the findings in EDA directly translate to actions in cleaning and preparing the data.