3.2: Data Collection Methods

Data Collection Methods

Organizations collect data through a wide variety of methods, depending on the nature of the information and the sources available. Here we discuss several common data collection methods, highlighting how each works and in what scenarios it is used.

Figure \(\PageIndex{1}\): This infographic lines up seven ways to collect data — from entering it by hand to fully automatic sensors — showing how each method works and when it is used. (CC BY 4.0; Felix Amoruwa) Access to detailed descriptions of Figure \(\PageIndex{1}\)

Each method has its own considerations: manual data entry and surveys rely on human participation and thus can be slow and error-prone (typos, respondent bias), whereas sensors and logs can generate huge volumes of data automatically, raising issues of storage and filtering. Web scraping and APIs are powerful for accessing external data but may face limitations (rate limits, need for cleaning, or legal restrictions on scraped content). Transaction system exports often require working with structured, relational data and ensuring consistency (especially if data comes from multiple systems – integrating these can overlap with data ingestion tasks).

In practice, organizations often use multiple collection methods in tandem. For example, a retail company might collect data via surveys (customer satisfaction), sensors (foot traffic counters in stores), transaction systems (sales records), and web scraping (competitor prices). A holistic data strategy will define how each source is collected and how frequently. Data governance policies also come into play at the collection stage – ensuring that personal data is collected with consent and stored securely, for instance.

Let us examine some of these methods in more detail:

3.2.1 Manual Data Entry

Manual data entry is one of the oldest and simplest forms of data collection. It involves humans entering data into a system, typically by reading from a source and typing it into a computer application. Sources might include paper forms, written notes, or any non-digital format that needs to be digitized. In other cases, manual entry is used when the volume of data is low or when an immediate digital interface is not available – for instance, a field researcher might jot observations in a notebook and later manually enter them into a spreadsheet.

Use Cases: Manual entry is common in small businesses and situations where automation is not feasible. Examples include: entering business card info from a networking event into a contacts database; keying in survey responses from paper questionnaires; hospital staff manually inputting patient vitals into an electronic record when devices aren’t integrated. It is also used to correct or update data – e.g. a data steward manually edits a few records that were flagged as errors.

Advantages: This method requires minimal technical setup – often just a person and a spreadsheet or form interface. It can handle unstructured or nuanced data that might be difficult to capture via automated means (humans can interpret messy handwriting or ambiguous info better than a basic scanner, for example). Manual entry also allows for real-time quality control by the person entering (they might notice an outlandish value and double-check it immediately).

Challenges: The obvious downsides are time and accuracy. Humans work slowly relative to machines and are prone to mistakes (transposition errors, typos). Data quality is a major concern – errors can propagate if not caught. There is also a lack of scalability: a single person can only enter so many records per hour, so this method doesn’t scale well for large datasets. For these reasons, organizations try to minimize manual data collection and replace it with automated methods whenever possible. However, some scenarios (like one-off small data gathering, or capturing subjective human judgments) will always require a manual element.

Best practices for manual entry include double data entry (two people independently enter the same data and results are compared to catch errors), providing training and clear guidelines to data entry personnel, and using user-friendly interfaces that reduce the chance of mistakes (drop-down menus, validation rules on fields, etc.).

3.2.2 Surveys and Questionnaires

Surveys are a research and data-gathering tool that involve asking a standard set of questions to a target group. They can be conducted in various modes: paper forms, telephone interviews, face-to-face interviews, or (very commonly now) online via web survey platforms. Surveys can produce quantitative data (e.g., Likert scale ratings 1–5, yes/no answers) and qualitative insights (open-ended responses).

Use Cases: Surveys are used in customer feedback (product satisfaction surveys, Net Promoter Score questionnaires), market research (consumer preference surveys), academic research (sociological or psychological questionnaires), employee engagement surveys, etc. Educational institutions might survey students on their learning experience; healthcare providers might survey patients about service quality.

Advantages: Surveys allow collection of specific information directly from the source (people) that might not be available in existing databases. They are flexible – you can design questions to gather exactly the data you need. Modern online survey tools can automatically record responses in a structured format and even enforce answer validation (ensuring required questions are answered, etc.). Surveys also scale reasonably well: an online survey can be sent to thousands of people, and the data comes back digitally.

Challenges: The quality of survey data depends on the survey design and respondent behavior. Poorly worded questions can bias answers or cause confusion. Low response rates are common challenges – many people may ignore the survey invitation, leading to potential response bias (those who do respond might not be representative of the whole group). There’s also honesty and accuracy – respondents might provide inaccurate information due to memory lapses or social desirability bias. Data from open-ended questions requires further processing or human reading to analyze. Finally, privacy is a concern: surveys often collect personal or sensitive data, so researchers must obtain consent and handle the data securely.

In data collection terms, surveys produce a dataset (each row might be one respondent’s answers, with columns for each question). If on paper, those then require manual data entry to digitize (which is a hybrid scenario: collection via survey, then ingestion via manual entry). Online surveys jump directly to digital form, which blurs the line between collection and ingestion (the survey platform might directly feed responses into a database).

3.2.3 Sensors and IoT Data

Sensors are devices that automatically detect or measure physical properties and produce data, often in real-time. Examples include temperature sensors, accelerometers, GPS sensors, RFID readers, and more. When we talk about IoT (Internet of Things) devices, we mean everyday objects instrumented with sensors and network connectivity, allowing them to send data to servers continuously. Collecting data from sensors typically involves the sensor pushing data to a gateway or cloud service, or a system polling the sensor at intervals.

Use Cases: Sensors are prevalent in manufacturing and industry (for equipment monitoring, where vibration or pressure sensors can detect machine maintenance needs), in smart cities (air quality sensors, traffic sensors), in healthcare (wearable health monitors, smart insulin pumps logging usage), and in consumer tech (smart home thermostats, fitness trackers). An IoT example is a fleet of trucks each equipped with GPS and engine sensors, streaming location and performance data to a central system.

Advantages: Sensor data collection is automated and can be real-time. It enables an “always-on” data stream. Because sensors collect objective measurements, the data is typically high frequency and granular (e.g., a reading every second). This enables use cases like real-time alerts (if a temperature exceeds a threshold, an alarm can be triggered) and massive longitudinal datasets for trend analysis or machine learning. IoT devices can collectively generate big data that yields insights through aggregation (for example, analyzing energy usage patterns from thousands of smart meters).

Challenges: The volume, velocity, and variety of sensor data are challenges. Volume/Velocity: even one sensor producing a reading per second generates over 31 million data points per year; with many sensors, the data can overwhelm networks or databases if not managed. Real-time data also implies issues of latency – ensuring the data is collected and transmitted fast enough for timely use. Data quality: sensors can drift or malfunction, leading to erroneous readings (which may need filtering out). There’s also often a need for schema evolution – if a new sensor type is added or a firmware update changes the data format (more on schema evolution in Section 3.4). Additionally, integration can an issue: sensor data might need to be joined with other business data (e.g., linking a machine sensor ID to the machine’s maintenance history from an enterprise system).

Connectivity and power can also constrain data collection. IoT devices might rely on battery power and intermittent connectivity, requiring strategies like edge collection (data buffered locally until connection resumes). For critical applications, reliability of sensor data collection is paramount – losing data due to network outages or device errors might mean losing visibility into important events.

3.2.4 Web Scraping

Web scraping refers to programmatically extracting data from websites not provided via an API. Essentially, a web scraper fetches the HTML of pages and parses it to find relevant information. For example, a scraper might pull all product names and prices from an e-commerce site’s pages.

Use Cases: Common uses of scraping include: price monitoring (e.g., an app that aggregates prices of flights or hotels by scraping multiple sites), content aggregation (news or blog aggregators), competitive intelligence (scraping competitor websites for product listings or job postings), and academic research (e.g., scraping social media or forums to analyze text). When an official API is unavailable or restricted, scraping is often the fallback to collect the needed data.

Advantages: Scraping can unlock access to vast amounts of publicly available data. If it’s on the web, you can often scrape it. It can also be automated to run regularly, keeping a dataset updated (e.g., scraping every day to see changes over time). With scraping, one can gather unstructured data (like text, images from web pages) and convert it into structured datasets for analysis.

Challenges: There are several challenges: legality and ethics – scraping might violate a website’s terms of service, and indiscriminate scraping can overload sites (which is why well-behaved scrapers respect robots.txt and rate limits). Anti-scraping measures like CAPTCHAs or dynamic content loading (via JavaScript) can complicate collection. Data cleanliness: HTML pages are meant for human viewing, not data tables, so scraped data often requires substantial cleaning and parsing to extract just the needed pieces. If the website’s structure changes, the scraper code breaks – this maintenance overhead is significant.

Scraping is also not real-time in the sense that one typically scrapes in batches or intervals, and one must be careful not to get IP-blocked by the host server.

Overall, web scraping is a powerful method but one that should be used responsibly. Where possible, using an official API is preferable; scraping is the method of last resort when no direct data interface is provided.

3.2.5 APIs (Application Programming Interfaces)

APIs allow software to communicate and exchange data. Many online services expose APIs to allow developers to retrieve data in a structured format. Collecting data via an API involves writing scripts that make requests (often HTTP requests) to the service’s API endpoints, usually authenticating with an API key, and then receiving data (often in JSON or XML) in response.

Use Cases: The range is huge: social media APIs (Twitter, Facebook) for collecting posts or metrics; financial APIs (e.g., Bloomberg, Alpha Vantage) for stock and economic data; weather APIs for climate data; Google Maps API for geographical and location data; public open data APIs from governments for things like census data or health statistics. In enterprise contexts, internal systems often have APIs too – e.g., a company’s SaaS CRM provides an API to fetch customer records.

Advantages: APIs are often the most efficient and reliable way to collect data from a platform, as they are designed for data interchange. You can usually filter and query an API to get exactly the data needed (for example, requesting only records updated since a certain date). Data from APIs is typically already well-structured (no messy parsing like HTML). Additionally, using an API is generally within the allowed usage of the service, provided you respect rate limits and usage policies; it's a sanctioned method to get the data. Automation is straightforward – one can schedule API calls to run regularly and pull new data incrementally (many APIs support querying “new since last ID” etc.).

Challenges: APIs come with rate limits (e.g., you can only make X requests per minute/hour), which means if you need a large volume quickly, you might be throttled. Some APIs require authentication and permissions – getting access might involve fees (e.g., commercial data APIs) or approval processes. The API could change or be versioned over time, requiring updates to your collection scripts (though changes are usually communicated by the provider). Also, integrating data from multiple APIs can be challenging if they have different formats or conceptual schemas. For example, linking data from a social media API with data from an internal sales system requires aligning keys like usernames to customer IDs, etc., which is non-trivial.

Another consideration is that API data collection bleeds into data ingestion: often, when we call an API, we immediately insert the results into our database or data lake (i.e., we are collecting and ingesting in one go). For instance, a pipeline might daily call a CRM API to get new customers and directly load them into a data warehouse table – this is essentially an ELT process focusing on an API as the extract mechanism.

3.2.6 Transaction Systems and Operational Databases

Most organizations have systems that record transactions – whether financial transactions, sales orders, inventory changes, or user actions. The data in these operational databases (often relational databases behind applications like ERP, CRM, POS, etc.) is a goldmine for analytics. Collecting data from such systems can be done in several ways:

• Exporting or dumping data tables (e.g., a full or incremental dump of a SQL table to a CSV or backup file).

• Running ETL jobs that query the database for new or updated records (perhaps every night).

• Tapping into the system’s change logs or using Change Data Capture (CDC) techniques (discussed in Section 3.3.3) to stream out changes in real-time.

Use Cases: Virtually every industry uses this: a retail company extracting sales and inventory data from its store databases to analyze trends; a bank pulling data from its transaction processing system to feed a risk or fraud analytics system; a university exporting student enrollment data from its registration system for analysis in a reporting database. Anytime you hear about a “data warehouse,” it likely gets its content by collecting data from operational systems of record on a scheduled basis.

Advantages: Transactional data is often well-structured (since it resides in databases with defined schemas) and high-value (it directly reflects business events like sales, deposits, shipments, etc.). Collecting from internal systems means you have full control over the data schema and content (unlike web data that can be unstructured or externally controlled). There are also mature tools for this kind of collection, from database replication software to ETL platforms designed to pull from SQL databases. If done through backups or dumps, it can capture the entire dataset as a snapshot.

Challenges: One major challenge is minimizing disruption to the source systems. Running heavy queries or exports on a production database can slow it down, affecting users. This is why techniques like CDC or replication are used – to offload the data without impacting the live system. Data consistency is another issue: if you export orders at 2 AM, what about an order that was partially completed at that time? Ensuring you capture a consistent cut of data (often via transactions or using timestamps) is important. Schema changes in source systems (e.g., a new column added to a table) can break the export or the downstream process if not handled (we’ll discuss schema evolution later). Security and privacy are also key – operational data often includes personal or sensitive information, so collection processes must secure the data in transit and at rest (e.g., encrypting files, restricting access).

Another challenge is that internal systems might be siloed. Collecting data in isolation from each doesn’t automatically make it integrated – for example, a customer might be represented differently in the sales system vs. the support ticket system. A lot of data integration work (beyond just collection) might be needed to merge such datasets.

3.2.7 Logs and Machine-Generated Data

Logs are a somewhat special category. They are generated by software and machines to record what’s happening. Collecting logs is crucial for fields like IT operations, cybersecurity, and web analytics. Unlike structured databases, logs are often semi-structured text (like lines of text with timestamp and message). Tools like ELK Stack (Elasticsearch-Logstash-Kibana), Splunk, or cloud logging services are dedicated to collecting and analyzing log data.

Use Cases: Web analytics – e.g., analyzing web server logs to understand user behavior on a website (each log line could be seen as analogous to an entry in a “website visits” dataset). Security – collecting authentication logs, system event logs, firewall logs into a SIEM (Security Information and Event Management) system to detect intrusions or anomalies. Application performance monitoring – devops teams collect application logs and error traces to debug issues or monitor performance metrics. IoT devices might also produce log-like telemetry streams.

Advantages: Logs provide a detailed, chronological record of events. They are often the only source for understanding what happened in a complex system (e.g., if a server crashes, the logs leading up to it tell the story). From a collection standpoint, logs are append-only and can be tailed (new entries collected continuously) which suits streaming ingestion well. Many logs follow standard formats (like the common combined log format for HTTP servers) which makes parsing easier. Modern logging frameworks tag logs with metadata (like severity, service name) which helps in filtering and collecting relevant subsets.

Challenges: The volume of logs can be enormous (a busy server can produce thousands of log lines per second). Storing and indexing that data can be expensive, so organizations often have to decide how long to retain logs and at what granularity. Logs may also contain unstructured text (error messages, stack traces) which are harder to analyze than numeric data – advanced techniques (like text search or machine learning) might be needed for full insight.

From a collection perspective, distributed systems produce distributed logs – collecting them means aggregating from multiple sources. Time synchronization becomes important (to merge logs by time across servers). There’s also sensitivity: logs can inadvertently contain sensitive data (e.g., a user’s email or password might appear in a log if not handled properly), so one must scrub or protect logs as needed.

Another complication is schema: logs are often schema-less or have a very flexible schema (new log types can appear as code is updated). So log collection systems need to handle schema on read (interpreting whatever fields exist in each log entry when analyzing). This again touches on schema evolution, which we’ll revisit.

In summary, data collection methods are diverse, and the choice of method depends on where the data originates and how it can be accessed. Often, a combination of methods is used to build a complete data repository. Regardless of method, a few key best practices for data collection include ensuring accuracy (through validation and verification), respecting privacy and ethical considerations (especially for personal data from surveys or logs), and documenting the collection process (knowing the provenance of your data is critical for later analysis trust). Once data is collected, the next step is to ingest it properly into an analytical environment – which is our next topic.