4.7: Data Distribution and Skewness

Data Distribution and Skewness

Beyond concrete rules and constraints, a more exploratory aspect of data preparation is understanding the distribution of your data and identifying any skewness or anomalies in it. Skewness refers to asymmetry in the distribution of values – a skewed dataset has a long tail on one side. While distribution shape is technically more of an analysis topic than a "data quality" dimension, it is highly relevant to data preparation: a very skewed distribution might indicate the presence of outliers or data entry errors, and often suggests the need for transformations (e.g., log transform) before analysis or modeling.

Why look at distributions? When preparing data, especially for statistical analysis or machine learning, knowing how the values are distributed can alert you to potential problems:

- Outliers: Extremely high or low values that are valid but rare can heavily skew the distribution. Some outliers are legitimate (e.g., a one-time huge purchase), but some may be errors (an extra zero turning $1,000 into $10,0000). Plotting or computing distribution statistics helps spot these.
- Skewness: If a variable is highly right-skewed (a long tail to the right), the mean will be much larger than the median. For example, income data is typically right-skewed: a small number of very high incomes raise the average above the median. In data cleaning, recognizing skewness might not mean removing it (since it could be real data), but it might influence how you handle the data. For instance, you might decide to report median instead of mean, or apply a transformation. Also, if the skewness is due to a few outliers that look suspect, you might investigate those specific points.
- Uniformity vs. clustering: Sometimes distribution analysis shows that data is clustered in certain ranges or categories with big gaps elsewhere. This might reveal missing categories or ranges that should have been present. For example, if you expect data across all months of the year but see none in July and August, it could indicate an issue in data collection for those months (not exactly skewness, but a distribution anomaly).

Using visuals: A histogram or box plot is extremely useful in data preparation. Even before formal analysis, plotting each numeric field’s histogram can quickly show if something is off. Many data prep tools (including Pandas in Python, R, or Dataprep UI) can generate profile charts. Figure 4.2 illustrates an example of a right-skewed distribution for a variable (say, sales amounts). The long tail to the right shows a few very large values. The mean is much higher than the median in such a case, which is a hallmark of positive skew.

Dealing with skewed data: If data is highly skewed but valid, sometimes a transformation is applied to make it more amenable to analysis. A common approach is a log transformation (for positive-valued data) which can compress the right tail. This isn’t exactly “cleaning” in the sense of fixing an error, but it is preparation to ensure analyses (like regression models that assume normality) perform better. Another approach is to bin the data (create categories) to reduce the impact of extreme values – for example, group all incomes above a certain threshold into a single top-coded category.

Skewness as a quality check: In some cases, the expected distribution is known and any major deviation could signal issues. For instance, if you have a sensor that should output values that follow (approximately) a normal distribution around some value, a highly skewed output might mean the sensor is malfunctioning or the data pipeline is truncating values on one end. As another example, consider a test score dataset that should be roughly bell-curved – if you see it skewed heavily or bi-modal, it might indicate some students have incomplete scores or the data combined two different scoring systems. Thus, examining distribution can reveal hidden data issues like mixed data sources or truncated data.

Identifying skew with summary stats: Even without plots, one can gauge skew by comparing mean and median, or using the skewness coefficient. If mean >> median, it’s right-skewed; if mean << median, left-skewed. A quick SQL check: SELECT AVG(value), MEDIAN(value), MIN(value), MAX(value) FROM Table; (note: MEDIAN might not be a built-in in all SQLs, but percentile can be used). Additionally, looking at the top 1% and bottom 1% of values can highlight outliers: e.g., SELECT value FROM Table ORDER BY value DESC LIMIT 10; to see the top 10 values.

Example: Suppose we are cleaning a dataset of household incomes for a region. We expect a skewed distribution (incomes often are), but when we chart it, we notice an extremely long tail with a handful of values an order of magnitude larger than the rest. Investigating, we find those few values are in the billions – likely data entry errors (maybe someone entered net worth instead of annual income, or added extra digits). These points would massively skew any average income calculation. In cleaning, we might decide to remove or Winsorize them (cap them to a lower value) because they are not representative of actual incomes. If they were legitimate (say the data included a few billionaires by design), then we keep them but report median incomes as more representative for many analyses, or separate that segment in reporting. The key is we wouldn’t have caught this issue without looking at the distribution.

In summary, understanding data distribution and skewness is a bridge between pure data cleaning and exploratory data analysis. It helps ensure that the dataset not only passes formal rules, but also behaves as expected statistically. Significant skewness or strange distribution patterns often merit a closer look to ensure they are explainable by real phenomena and not artifacts of data problems. This step gives confidence that the data is ready for analysis, or provides one more opportunity to apply transformations to make the data suitable for the intended analytical methods.