5.2: Historical Background and Importance

Historical Background and Importance

Today, EDA is considered an essential step in any data-driven investigation. It’s often said (perhaps hyperbolically) that data scientists spend 80% of their time on data cleaning and EDA, and only 20% on modeling or algorithm development. While the exact percentage may vary, this adage underscores the reality that a major portion of analytical effort is devoted to understanding and preparing the data. Without EDA, one risks building models on faulty assumptions or missing key insights. Many mistakes in analysis (or even project failures) can be traced to inadequate exploratory analysis: for instance, unnoticed data quality issues or overlooked relationships can lead to incorrect conclusions. EDA’s importance in the data analysis pipeline therefore cannot be overstated – it ensures that “the results produced are valid and applicable to any desired business outcomes and goals”, as one practitioner notes. By investing time in EDA, analysts validate that they are asking the right questions of the data and that subsequent steps (like statistical modeling or hypothesis tests) are based on a sound understanding of the dataset.

Exploratory vs. Confirmatory Analysis:

It is useful here to distinguish EDA from confirmatory data analysis (CDA). In confirmatory analysis, one begins with a clear hypothesis or model and uses statistical tests to confirm or refute that hypothesis (for example, performing a t-test or fitting a predefined regression model). Confirmatory analysis relies on probability models and assesses evidence against a formal null hypothesis. EDA, in contrast, is more open-ended – it does not start with a definitive hypothesis, and it makes no strict assumptions about the underlying data distribution or generative model. As one EDA textbook analogizes, exploring data is like detective work: the analyst is looking for clues and patterns in the data (the “evidence”) without being sure what they will find. There is no notion of statistical significance or p-values in pure EDA; instead, the focus is on uncovering what is interesting in the data at hand. In practice, a complete data analysis project will involve both EDA and confirmatory analysis: typically, one uses EDA to discover patterns or to frame hypotheses, and then uses confirmatory analysis (formal statistical tests or models) to verify those insights on the same or new data. This interplay ensures that when we finally communicate results, we are highlighting insights that have both exploratory evidence and confirmatory support.

Exploratory vs. Explanatory Visualization:

A related distinction is between exploratory analysis and explanatory analysis when it comes to data visualization [5]. In exploratory visualization, we create plots for ourselves, as analysts, to reveal patterns and check anomalies during EDA. These plots prioritize learning from the data over polish – they can be quick, numerous, and even rough, as long as they help us iterate and refine our understanding. In explanatory (or communicative) visualization, by contrast, we create graphics for an audience to convey a specific finding or story derived from the data. In an exploratory plot, you might not worry about every detail of formatting; you might try many chart types quickly. In an explanatory plot (say, for a report), you would highlight particular insights, label and color carefully, and remove extraneous details to tell a clear story. It’s important to keep this distinction in mind: during EDA we are in “discovery mode,” but afterwards, when communicating, we switch to “presentation mode.” EDA results often must be refined and filtered before presentation. In fact, one best practice is to perform thorough EDA before presenting findings – this gives confidence that when you focus your audience on a particular statistic or trend, you’ve ruled out other confounding factors or patterns through your exploratory work. In summary, EDA is about exploring and finding the truth in data, while explanatory analysis is about explaining and illustrating that truth effectively.

Figure \(\PageIndex{1}\): Exploratory vs. Confirmatory vs. Explanatory Analysis. This comparison table contrasts the three modes of data analysis by their goals, methods, audience, and outputs. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)