5: Exploratory Data Analysis (EDA)
- Page ID
- 48056
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\dsum}{\displaystyle\sum\limits} \)
\( \newcommand{\dint}{\displaystyle\int\limits} \)
\( \newcommand{\dlim}{\displaystyle\lim\limits} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\(\newcommand{\longvect}{\overrightarrow}\)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\ket}[1]{\left| #1 \right>}\)
\(\newcommand{\bra}[1]{\left< #1 \right|}\)
\(\newcommand{\braket}[2]{\left< #1 \vphantom{#2} \right| \left. #2 \vphantom{#1} \right>}\)
\(\newcommand{\braopket}[3]{\left< #1 \vphantom{#2}\vphantom{#3} \right| #2 \vphantom{#1}\vphantom{#3} \left| #3 \vphantom{#1}\vphantom{#2} \right>}\)
\(\newcommand{\qmvec}[1]{\mathbf{\vec{#1}}}\)
\(\newcommand{\op}[1]{\hat{\mathbf{#1}}}\)
\(\newcommand{\expect}[1]{\langle #1 \rangle}\)
\(\newcommand{\dfn}[1]{\emph{\textbf{#1}}}\)
- Define exploratory data analysis (EDA) and explain its goals in the data analysis process.
- Distinguish between exploratory analysis and confirmatory analysis.
- Differentiate exploratory visualizations from explanatory visualizations.
- Explain the historical importance of John Tukey and the development of EDA.
- Describe how EDA helps analysts summarize data, discover patterns, check assumptions, and generate hypotheses.
- Apply univariate analysis techniques to examine individual variables.
- Use summary statistics such as mean, median, quartiles, range, standard deviation, and IQR to describe numeric data.
- Interpret histograms, box plots, bar charts, and frequency tables to understand distributions and categorical patterns.
- Apply bivariate analysis techniques to examine relationships between two variables.
- Use scatter plots, grouped summaries, box plots, contingency tables, and correlation measures to explore pairwise relationships.
- Recognize how categorical and numerical variables can be compared in EDA.
- Apply multivariate analysis techniques to explore relationships among three or more variables.
- Use tools such as scatter plot matrices, correlation heatmaps, faceting, clustering, and principal component analysis (PCA) for multivariate exploration.
- Detect, evaluate, and handle missing data during EDA.
- Detect, investigate, and appropriately handle outliers using both visual and statistical methods.
- Use EDA findings to guide feature selection, feature transformation, and preprocessing decisions.
- Explain how EDA supports data preparation for later modeling and predictive analysis.
- Identify common tools and workflows used for EDA, including notebooks, Python/R libraries, and automated profiling tools.
- Apply best practices for EDA, including documentation, reproducibility, iteration, and balancing visual and quantitative analysis.
- Recognize common mistakes in EDA, such as rushing, confirmation bias, ignoring data quality issues, and confusing correlation with causation.
- 5.1: Definition and Goals of EDA
- Exploratory Data Analysis (EDA) is an approach to analyzing data sets to summarize their main characteristics and uncover patterns, anomalies, and insights, often using visual methods. It serves as a crucial preparatory step in the data analysis pipeline to inform which statistical models to use and to formulate hypotheses before formal testing.
- 5.2: Historical Background and Importance
- Today, EDA is considered an essential first step in any data investigation. It's often said that data scientists spend the vast majority of their time (e.g., 80%) on data cleaning and EDA. This highlights its importance in understanding and preparing data, which helps prevent building models on faulty assumptions, missing key insights, or producing invalid results.
- 5.3: Univariate Analysis Techniques
- This chapter explains that univariate analysis is the first step in Exploratory Data Analysis (EDA), where each variable is examined individually to understand its distribution, typical values, and outliers. It details the use of summary statistics and, more importantly, visualizations like histograms and box plots, which are crucial for revealing patterns and anomalies not apparent from statistics alone.
- 5.4: Bivariate Analysis Techniques
- This chapter explains how to perform bivariate analysis by examining relationships between pairs of variables using techniques like scatter plots for numeric data, box plots for categorical versus numeric data, and contingency tables for two categorical variables.
- 5.5: Multivariate Analysis Techniques
- This chapter on Multivariate Analysis Techniques explains how to examine relationships among three or more variables in Exploratory Data Analysis (EDA). It details various methods, including advanced visualization (like faceted plots), dimensionality reduction (such as PCA), clustering, and using tools like correlation heatmaps to uncover complex patterns and structures within data.
- 5.6: Handling Missing Data in EDA
- This chapter details the process of handling missing data in Exploratory Data Analysis (EDA), covering how to detect and quantify missing values, understand the nature of the missingness (e.g., MCAR, MAR, MNAR), and apply appropriate strategies like deletion, imputation, or flagging. The text emphasizes that carefully addressing missing data is crucial to avoid introducing bias and ensure the validity of any subsequent analysis.
- 5.7: Handling Outliers in EDA
- This section focuses on the critical process of identifying, understanding, and managing outliers in a dataset. Outliers are data points that significantly differ from other observations and can affect statistical summaries and model performance.
- 5.8: Tools and Workflow for EDA
- This chapter outlines the tools and workflow for Exploratory Data Analysis (EDA), covering programming languages like Python and R, their associated libraries, and various software environments. It also details a typical, iterative EDA workflow that progresses from data understanding and cleaning through univariate, bivariate, and multivariate exploration to a final synthesis of findings.
- 5.9: Best Practices in EDA
- This chapter details a thoughtful and methodological approach to Exploratory Data Analysis (EDA), emphasizing that it's a systematic yet flexible process requiring a balance of visualization and quantitative analysis. Key practices include understanding the problem's context, not rushing, documenting for reproducibility, being aware of biases, and knowing when to stop the exploratory phase.
- 5.10: Common Mistakes in EDA and How to Avoid Them
- This chapter outlines ten common pitfalls in Exploratory Data Analysis (EDA), such as rushing the process, confirmation bias, misinterpreting correlation as causation, and using poor visualizations. It advises analysts to be mindful of these errors to improve the quality of their analysis, leading to more robust and credible conclusions.