5.6: Handling Missing Data in EDA

Handling Missing Data in EDA

Real-world data is often messy, and one common issue is missing data. EDA must address missing values for two reasons: (1) to accurately characterize the dataset (e.g., how much data is missing and whether the missingness has any pattern), and (2) to decide how to handle missing values in subsequent analysis. Ignoring missing data can lead to biased results or effective loss of information, so it’s an important part of exploratory analysis to identify and account for missingness.

Detecting Missing Values:

In the early stage of EDA, an analyst will typically quantify missing data. Tools like df.info() or df.isnull().sum() in Python can quickly show how many nulls are in each column. It’s good practice to compute the percentage of missing values per variable, as well as check if there are entire observations (rows) with a lot of missing fields. Sometimes a pattern emerges: e.g., one particular column might be 90% missing, or a combination of two columns might have complementary missingness (when one is filled, the other is not). Visualization can help here: a matrix plot of missing data (with a dot or colored cell for each missing value in a grid of rows vs columns) can reveal if missing values concentrate in certain rows or columns. There are also dedicated packages (like missingno in Python) that produce heatmaps or bar charts of missing data by column. For example, you might see that most missing entries occur in later years of a dataset, suggesting a data collection issue after a certain date.

A critical part of EDA is to consider whether data are Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR), because that affects how you handle them. EDA can provide hints: if missingness correlates with other variables (e.g., all missing ages are for people with high income), then it's not random and one must be careful not to simply ignore those values. Sometimes, missing values themselves carry information (for instance, a missing value for "credit score" might indicate the person has no credit history; thus "missing" is actually a category in itself).

Figure \(\PageIndex{1}\): Missing Data Mechanisms and Handling Strategies. This diagram categorizes the three types of missing data mechanisms (MCAR, MAR, MNAR) alongside the common strategies for handling them. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

Strategies for Missing Data:

Once we’ve assessed missingness, we need to decide on a handling strategy, and this often involves some exploratory trial-and-error. Common approaches include:

• Deletion: If only a small percentage of data is missing and it’s randomly distributed, one might choose to omit those records (listwise deletion) or drop those specific features. E.g., if 2% of values in a column are missing, perhaps dropping those rows is acceptable. However, if the missingness is significant or not random, deletion can bias the analysis.
• Imputation: Replacing missing values with a reasonable estimate. Simple methods are mean imputation (fill with the mean of that variable) or median/mode imputation for numeric and categorical respectively. These preserve the dataset size but may underestimate variability (since all imputed values are the same). More sophisticated methods include predictive imputation (using regression or KNN to predict missing values from other variables), or multiple imputation which involves creating several imputed datasets with random variations and averaging results. During EDA, one might try a quick imputation to see how it affects distributions – e.g., fill NAs with median and then redraw a histogram to see if the shape looks reasonable. If the imputed values create odd spikes or distortions, a different approach might be needed.
• Flagging: In some cases, especially for categorical data, one can treat "missing" as its own category (e.g., Unknown). Or create an additional binary flag variable indicating missingness. This way, the fact that data was missing is not lost – a model can potentially learn if "missingness" itself was informative. In EDA, one might explore whether the missing-flag correlates with other outcomes.

The best practice is to be strategic and cautious with imputation. During EDA, an analyst might do a quick fill to allow certain analyses (for instance, filling missing values in order to compute a correlation matrix), but should remember that these are provisional. It’s important to note how many and which values were imputed, and perhaps try multiple approaches. For example, if replacing NAs with the mean, one should be aware that this can shrink variance and might mask the true uncertainty – so one might also try an alternative like median or a basic model-based imputation and compare results. The EDA phase should document how sensitive any insights are to missing data handling. If, say, an important correlation only appears after you impute some values, you should check whether those values being missing in a particular pattern could have been the real reason for the correlation.

Example: Imagine a dataset of housing prices with some houses missing the square footage field. EDA reveals 10% of homes have no square footage listed, and these tend to be older houses in rural areas. Simply dropping those houses could bias any analysis of price vs size (because it drops many rural homes). If we impute square footage by average, we might be overestimating those rural homes' sizes or underestimating variance. EDA might involve trying a group-wise imputation (e.g., fill missing square footage with the median for rural homes of that era) and then seeing if the price vs size relationship holds consistent. The EDA conclusion might be: rural older homes often lack recorded size, and they also tend to have lower prices; the missing data is not random. We should either acquire that information if possible, or use a carefully considered imputation (perhaps a separate model for square footage in rural homes) and include a flag for imputed values. This kind of reasoning ensures that when we move to modeling, we won’t be caught off guard by the missing data issue.

In summary, handling missing data in EDA means making the problem visible (quantifying and understanding it) and experimenting with solutions in a reversible way. One should always document how many points were dropped or altered and consider the impact. As a rule of thumb: do not let missing values silently distort your analysis. Address them upfront during EDA either by cleaning or explicitly accounting for them.