3.10: Detailed Figure Captions - Chapter 3

Data Collection and Ingestion — Figure Specifications

Overview

• Chapter: 3 — Data Collection and Ingestion
• Total Figures Proposed: 8
• Figure Types Used: Comparison Table, Flowchart, Architecture Diagram, Conceptual Illustration, Infographic, Network Diagram

---

Figure 3.1.1

Title: Data Collection vs. Data Ingestion — Conceptual Boundary 

Detailed Caption: Two color-coded zones sit side by side, separated by a gold dashed "Raw Data" boundary line. The left zone, navy "DATA COLLECTION," holds six gray source tiles in a 2×3 grid (Surveys, Sensors/IoT, Web Scraping, APIs, Databases, Logs) above a Characteristics list — Gathering from sources, Source-focused, Manual or automated, Raw form. The right zone, teal "DATA INGESTION," shows a horizontal pipeline of four teal tiles (Parse → Validate → Transform → Load) above a row of three navy targets (Data Warehouse, Data Lake, Stream Platform), with its own characteristics list — Transporting to target, Target-focused, Automated pipelines, Prepared for analysis. The deliberate visual handoff at the gold "Raw Data" boundary makes the chapter's distinction concrete: collection ends when raw data exists somewhere capturable, while ingestion begins the moment that data starts moving toward an analytics destination — and confusing the two often leads teams to over-automate collection or under-engineer ingestion. 

Figure 3.1.2

Title: Data Collection vs. Data Ingestion — Conceptual Boundary 2

Detailed Caption: The diagram is a side-by-side comparison divided into two distinct panels, illustrating the difference between collecting raw data from various sources and moving that data into a central system for analysis. Key takeaway: Data Collection is about gathering raw data from diverse sources (source-focused, possibly manual), while Data Ingestion is about processing and loading that data into structured target systems via automated pipelines so it's ready for analysis.                                                             

Figure 3.2.1

Title: Common Data Collection Methods — Overview 

Detailed Caption: A horizontal gradient bar runs across the top labeled "MANUAL" in warm gold on the left and "AUTOMATED" in teal on the right. Beneath it, seven evenly spaced cards line up along the spectrum, each connected to its position by a dashed line. Reading left to right: Manual Data Entry (gold-bordered, "e.g., digitizing paper forms"), Surveys & Questionnaires (gold, "e.g., customer feedback forms"), Transaction Exports (gray, "e.g., POS system data dumps"), Web Scraping (gray, "e.g., price monitoring"), APIs (teal, "e.g., REST/GraphQL data feeds"), Logs & Machine Data (teal, "e.g., server logs, app telemetry"), and Sensors & IoT (teal, "e.g., temperature, GPS tracking"). Each card carries a navy header and an italic example below it. The deliberate spectrum encoding — color cooling from gold to teal as automation rises — makes the chapter's pedagogical point visible: collection methods aren't a flat menu but a continuum where higher automation buys scale and consistency at the cost of flexibility, and most enterprise pipelines combine methods from across the bar rather than choosing one. 

Figure 3.3.1

Title: Batch vs. Streaming Ingestion — Processing Model Comparison 

Detailed Caption: Three stacked swim lanes share a parallel timeline structure to make the latency contrast immediate. The top navy "BATCH INGESTION" lane shows a dashed Collection Window (24 hrs) accumulating event dots, then a thick "Bulk Load" arrow into a navy Warehouse tile, with the latency annotation "High Latency (hours)." The middle warm-gold "MICRO-BATCH" lane shows two smaller dashed groupings labeled "Every 5 min" each emitting an arrow into a gold Storage tile, with "Near Real-Time (minutes)." The bottom teal "STREAMING INGESTION" lane shows a continuous trail of teal dots flowing through a Processor tile and on to a Storage tile, with "Low Latency (seconds)." By holding the timeline metaphor constant and varying only the grouping pattern, the figure isolates the single variable that distinguishes the three approaches — collection-window size — and shows why latency, complexity, and throughput trade off the way they do: bigger windows pack more efficient bulk loads at the cost of freshness, while smaller windows shorten latency but multiply infrastructure overhead. 

Figure 3.3.2

Title: ETL vs. ELT — Data Flow Comparison

Detailed Caption: Two parallel horizontal flows stack vertically, separated by a thin dashed line. The top flow, navy "ETL (Extract, Transform, Load)," runs Extract → Transform (Staging Server / ETL Engine) → Load (Data Warehouse), with the footer "Transform before load — only refined, schema-conforming data in warehouse." The bottom flow, teal "ELT (Extract, Load, Transform)," runs Extract → Load (Data Lake — Raw Data As-Is) → Transform (In-Place Using Target Compute), with the footer "Load first, transform later — raw data preserved for flexibility." Warm-gold arrows connect the stages in both rows. The mirrored layout — same three steps, only their order swapped — is the pedagogical anchor: it makes clear that ETL and ELT aren't different operations but the same operations sequenced differently, and the choice of sequence has architectural consequences (purpose-built ETL engines vs. powerful target compute, refined-only storage vs. raw-data flexibility) that ripple through the entire pipeline. 

Figure 3.3.3

Title: Change Data Capture (CDC) — Mechanism and Data Flow 

Detailed Caption: A navy "Source Database" cylinder sits on the left, with an attached "Transaction Log" panel listing three sample entries — INSERT row 101, UPDATE row 42, DELETE row 77. A teal "CDC Tool (e.g., Debezium)" arrow reads from the log and feeds a warm-gold "Event Stream · Kafka Topic" tile in the center. From this hub, three branching gold arrows fan rightward to three navy downstream targets — Analytics Warehouse, Search Index, and Cache / Microservice — each labeled to indicate that all three apply the same change events. A teal-bordered callout reads "Key Benefit: Only deltas transferred — no full table scans." The figure makes vivid why CDC is the right tool for high-volume incremental sync: by reading the database's own write log rather than scanning tables, the technique transforms what would be a recurring full-table read into a continuous trickle of only the rows that actually changed, which scales gracefully even when the source database holds billions of rows. 

Figure 3.4.1

Title: Data Ingestion Challenges and Best Practices 

Detailed Caption: A 2×2 grid of light-gray quadrants frames the figure, with a teal "Robust Data Pipeline" badge anchoring the center where the four meet. Each quadrant pairs a navy challenge header with three bulleted mitigations: top-left Schema Evolution lists Schema Registry, Backward Compatibility, Flexible Formats (JSON, Parquet); top-right Scalability lists Distributed Systems, Cloud Auto-scaling, Partitioning & Parallelism; bottom-left Latency lists Streaming over Batch, Incremental Processing, Edge Computing; and bottom-right Data Quality lists Validation Rules, Deduplication, Profiling & Monitoring. Warm-gold borders frame each quadrant. The four-quadrant geometry communicates that resilient ingestion requires investment on all four fronts — solving any single challenge isn't sufficient because the four interact (high scalability without quality controls amplifies bad data; low latency without schema evolution support breaks downstream consumers when sources change) — and the central badge reminds students that "robust pipeline" is the integrated outcome these four practices produce together, not any one of them in isolation. 

Figure 3.5.1

Title: Lambda vs. Kappa Architecture — Structural Comparison 

Detailed Caption: Two architecture panels sit side by side. The left panel, navy "LAMBDA ARCHITECTURE," shows a single Data Source feeding two parallel paths — a navy Batch Layer (MapReduce / Spark) above a teal Speed Layer (Flink / Storm) — both converging into a warm-gold Serving Layer / Query, footnoted "Two codebases, high accuracy + low latency." The right panel, teal "KAPPA ARCHITECTURE," shows the Data Source feeding a teal Event Log (Kafka), then a single Stream Processor, then a gold Serving Layer / Materialized Views, with a dashed "Replay for Reprocessing" arc curving back from the Stream Processor to the Event Log; footer "One codebase, simpler but requires replay capability." The mirrored composition — same input source, same serving output, but very different middles — captures the essential trade-off: Lambda's two pipelines deliver belt-and-suspenders correctness at the cost of maintaining duplicate logic, while Kappa replaces that complexity with a single replay-capable log, simpler to operate but requiring infrastructure (and discipline) to reprocess history when business logic changes.

Figure 3.6.1

Title: Cross-Industry Data Ingestion Patterns — Case Study Summary 

Detailed Caption: A six-column matrix runs beneath a navy header row (Industry, Key Data Sources, Ingestion Approach, Architecture, Domain Standard, Key Outcome), with five industry rows alternating between white and pale-mint backgrounds. Healthcare combines EHR/Claims/Registries with CDC + Batch and ELT/Data Lake under HL7/FHIR for a 360° Patient View; Finance pairs Transactions and Fraud DBs with Streaming/CDC over an Event-Driven architecture and ISO 8583 to deliver $1M/mo fraud prevented; Retail uses POS/Inventory/Stores via Streaming + Batch on Cloud ELT for a 40% cost reduction; IoT/Mfg streams Sensors/PLCs/SCADA over MQTT in a Lambda (Edge+Cloud) pattern with MQTT/OPC-UA standards for predictive maintenance; and Education runs SIS/LMS/Engagement through nightly Batch into an ETL Warehouse under FERPA, lifting retention from 84% to 89%. The Key Outcome column is rendered in warm gold to draw the eye to the measurable result of each pattern. The matrix collapses five long case-study sections into one scannable reference and shows that the same canonical ingestion building blocks recombine differently across industries — domain standards and outcomes vary, but the underlying choices (batch vs. stream vs. CDC, ETL vs. ELT) repeat with predictable trade-offs.