5.1: Definition and Goals of EDA

Definition and Goals of EDA

Exploratory Data Analysis (EDA) is a philosophy and set of techniques for first looking at data to discover its underlying structure and patterns, without initially resorting to formal modeling or hypothesis testing [1]. In statistics, EDA is defined as an approach to analyzing data sets that summarizes their main characteristics—often employing visual methods and simple quantitative techniques. The primary goal of EDA is to let the data speak for itself: to uncover unexpected phenomena, patterns, or anomalies in the data, suggest interesting hypotheses, and check assumptions, all before formal modeling. In other words, EDA is a preparatory step in the data analysis pipeline that helps analysts make informed decisions about which statistical tools or models may be appropriate and whether additional data might be needed.

Several key objectives of EDA can be highlighted:

• Summarize main characteristics of the dataset in a clear form (e.g. through descriptive statistics and plots).
• Discover patterns, trends, or relationships that were not evident initially (EDA encourages unexpected discoveries in data).
• Check assumptions required for later statistical inference or modeling (e.g. normality, linearity, independence).
• Support selection of appropriate techniques for further analysis (guiding which models or tests make sense next).
• Formulate hypotheses about causes of observed phenomena, which can later be tested with confirmatory analysis.
• Guide further data collection if needed (for example, noticing a data gap or the need for additional variables).

EDA thus serves as a foundation for subsequent analysis. It falls under descriptive analytics, focusing on what the data shows, as opposed to inferential or predictive analytics which focus on proving or predicting something. A well-conducted EDA ensures that any modeling or hypothesis testing that follows is based on a thorough understanding of the data's properties and potential quirks.

Figure \(\PageIndex{1}\): EDA in the Data Analysis Pipeline. This flowchart illustrates where Exploratory Data Analysis sits within the broader data analysis workflow, from raw data ingestion through to confirmatory analysis and reporting. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)