Data Science Module 2: Data Collection

Introduction to Data Collection

Data collection is the foundational step in any data science project. It involves gathering relevant information from various sources to address specific research questions or business problems. The quality and relevance of collected data directly impact the success of subsequent analysis.

Effective data collection requires careful planning to ensure data is accurate, complete, and representative of the phenomenon being studied.

Knowledge Check: Data Types

Select the correct data type for each example below:

"Customer satisfaction rating: Very Satisfied, Satisfied, Neutral"

"Temperature in Celsius: 23.5, 25.0, 18.2"

Data Sources and Collection Methods

Data can be collected from primary sources (first-hand collection) or secondary sources (pre-existing data). Modern data science leverages both approaches.

APIs

Application Programming Interfaces allow programmatic access to data from services like Twitter, Google, or financial markets.

Real-time

Web Scraping

Extracting data from websites using tools like BeautifulSoup, Scrapy, or Selenium for large-scale data collection.

Ethical considerations

Databases

Structured data storage systems (SQL, NoSQL) containing organized records from business operations or research.

Structured

API Simulation

Try making a simulated API request to understand how data is fetched programmatically.

GET https://api.learn.geniusjr.org/v1/weather?city=London&units=metric

{ "city": "London", "temperature": 18.5, "conditions": "Partly Cloudy", "humidity": 65, "wind_speed": 12.3, "timestamp": "2023-10-15T14:30:00Z", "forecast": [ {"day": "Monday", "high": 19, "low": 12, "condition": "Sunny"}, {"day": "Tuesday", "high": 17, "low": 11, "condition": "Rainy"}, {"day": "Wednesday", "high": 20, "low": 13, "condition": "Cloudy"} ] }

Sampling Techniques

When collecting data from large populations, sampling is essential. Different sampling methods have different strengths and biases.

Method	Description	When to Use	Bias Risk
Simple Random	Every member has equal chance of selection	Homogeneous populations	Low
Stratified	Population divided into subgroups, sample from each	Ensuring subgroup representation	Low
Cluster	Population divided into clusters, random clusters selected	Geographically dispersed populations	Medium
Systematic	Select every kth member from a list	When list is randomly ordered	Medium
Convenience	Sample readily available subjects	Preliminary research only	High

Sampling Visualization

Adjust the sample size and see how it affects representation:

Sampling visualization will appear here

Sample Size: 30%

Sampling Method

Hands-On: Collecting Data with Python

In this exercise, you'll write Python code to collect data from a simulated API and store it in a pandas DataFrame.

Data Collection Exercise

# Data Collection Exercise
import pandas as pd
import numpy as np

# Simulated API response (in real scenario, you'd use requests.get())
def fetch_user_data():
    """Simulates fetching user data from an API"""
    # Simulated API response
    data = [
        {"id": 1, "name": "Alice", "age": 28, "country": "USA", "score": 85},
        {"id": 2, "name": "Bob", "age": 34, "country": "Canada", "score": 92},
        {"id": 3, "name": "Charlie", "age": 22, "country": "UK", "score": 78},
        {"id": 4, "name": "Diana", "age": 29, "country": "Australia", "score": 88},
        {"id": 5, "name": "Eve", "age": 31, "country": "USA", "score": 95}
    ]
    return data

# TODO: Complete the following tasks:
# 1. Call fetch_user_data() to get the data
# 2. Convert the data to a pandas DataFrame
# 3. Calculate the average score
# 4. Find the user with the highest score

# Your code here:

# Step 1: Get data
users_data = fetch_user_data()

# Step 2: Convert to DataFrame
df = None  # Replace None with your code

# Step 3: Calculate average score
average_score = 0  # Replace 0 with your code

# Step 4: Find user with highest score
top_user = None  # Replace None with your code

print("DataFrame shape:", df.shape if df is not None else "Not created")
print("Average score:", average_score)
print("Top user:", top_user)

Solution:

# Solution
import pandas as pd

users_data = fetch_user_data()

# Convert to DataFrame
df = pd.DataFrame(users_data)

# Calculate average score
average_score = df['score'].mean()

# Find user with highest score
top_user = df.loc[df['score'].idxmax(), 'name']

print("DataFrame shape:", df.shape)
print("Average score:", average_score)
print("Top user:", top_user)
print("\nFull DataFrame:")
print(df)

Module 2 Quiz: Data Collection

Test your understanding of data collection concepts. Select the best answer for each question.

1. Which of the following is a primary data collection method?

Analyzing existing company sales records

Using government census data

Conducting customer surveys

Downloading stock prices from a financial website

2. When collecting data via web scraping, what is the MOST important ethical consideration?

Using the fastest scraping tool available

Respecting robots.txt and website terms of service

Collecting as much data as possible

Avoiding JavaScript-based websites

3. Which sampling method ensures proportional representation of subgroups in the population?

Simple random sampling

Convenience sampling

Cluster sampling

Stratified sampling

4. What is the main advantage of using APIs for data collection compared to web scraping?

APIs provide structured, authorized access to data

APIs are always free to use

APIs work faster than web scraping

APIs don't require programming knowledge

5. In the context of data collection, what does "data provenance" refer to?

The size of the dataset

The accuracy of the data

The origin and history of the data

The storage format of the data

Additional Resources

Expand your knowledge of data collection with these resources:

Data Collection Checklist

Use this checklist for your data collection projects:

Define clear objectives and data requirements

Identify appropriate data sources

Ensure legal and ethical compliance

Plan for data storage and organization

Document data provenance and collection methods