2: Data Types, Data Sources, and Data Quality
- Page ID
- 48023
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\dsum}{\displaystyle\sum\limits} \)
\( \newcommand{\dint}{\displaystyle\int\limits} \)
\( \newcommand{\dlim}{\displaystyle\lim\limits} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\(\newcommand{\longvect}{\overrightarrow}\)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\ket}[1]{\left| #1 \right>}\)
\(\newcommand{\bra}[1]{\left< #1 \right|}\)
\(\newcommand{\braket}[2]{\left< #1 \vphantom{#2} \right| \left. #2 \vphantom{#1} \right>}\)
\(\newcommand{\braopket}[3]{\left< #1 \vphantom{#2}\vphantom{#3} \right| #2 \vphantom{#1}\vphantom{#3} \left| #3 \vphantom{#1}\vphantom{#2} \right>}\)
\(\newcommand{\qmvec}[1]{\mathbf{\vec{#1}}}\)
\(\newcommand{\op}[1]{\hat{\mathbf{#1}}}\)
\(\newcommand{\expect}[1]{\langle #1 \rangle}\)
\(\newcommand{\dfn}[1]{\emph{\textbf{#1}}}\)
Understanding, defining, and categorizing the scope of data types, including distinctions between structured vs. unstructured data, qualitative vs. quantitative data, and the classic levels of measurement (nominal, ordinal, interval, ratio) will be what we start with first. Next, we will review various data sources—ranging from internal databases to edge devices such as external sensors and open datasets—illustrating how organizations acquire data. The next section we will cover will delve into data quality, examining key quality dimensions (accuracy, completeness, consistency, timeliness, validity, etc.) and why each of these dimensions are critical for reliable analysis. Finally, we will discuss the roles of metadata, data governance, and data stewardship in maintaining data quality and integrity. Throughout, real-world examples and case studies from multiple industries (finance, healthcare, retail, manufacturing, etc.) will highlight the practical importance of these concepts. Scholarly and industry sources are cited to provide an academic foundation for the discussion.
- Classify data as structured, semi-structured, or unstructured and explain how each type impacts storage, processing, and analysis approaches.
- Distinguish between quantitative and qualitative data, providing examples of each and explaining why this distinction matters when selecting analysis methods.
- Identify and apply the four levels of measurement — nominal, ordinal, interval, and ratio — and determine which statistical operations are appropriate for each level.
- Compare internal and external data sources, describing common examples of each (e.g., CRM systems, public datasets, IoT sensors, social media) and explaining how integrating both enriches analytics.
- Define the key dimensions of data quality — accuracy, completeness, consistency, timeliness, validity, uniqueness, and integrity — and explain why each is critical for reliable analysis.
- Analyze real-world consequences of poor data quality and describe techniques organizations use to prevent and correct quality issues (e.g., validation rules, data profiling, deduplication).
- Explain the role of metadata in making data discoverable, understandable, and manageable, and identify different types of metadata (descriptive, structural, administrative, technical).
- Differentiate between data governance and data stewardship, describing how governance sets policies and strategy while stewardship handles day-to-day enforcement and quality management.
- 2.1: Data Types
- This chapter introduces the fundamental ways to classify data, which is essential for choosing the correct analysis methods and tools.
- 2.2: Data Sources
- This chapter explores the various data sources that are fundamental to data analytics, categorizing them broadly as either internal or external.
- 2.3: Data Quality
- This chapter introduces the fundamental concept of Data Quality, emphasizing that for data to be useful, it must be of high quality. It explains that poor data quality can lead to significant negative consequences, such as incorrect insights, bad decisions, and lost opportunities. The chapter then defines and elaborates on the key dimensions used to measure and evaluate data quality.
- 2.4: Metadata, Data Governance, and Data Stewardship
- In this section, we shift from the data itself to the practices and information that support the management of data. We discuss metadata (often described as "data about data"), and the roles of data governance and data stewardship in an organization. These concepts are crucial for maintaining data quality, ensuring proper use of data, and adhering to policies and regulations. They provide the necessary infrastructure and oversight so that data remains an asset rather than becoming a liability.
- 2.5: Chapter Summary and Industry Examples
- In this chapter, we cover the essentials of data types, data sources, and data quality, along with the governance structures that support reliable data analytics.