Data Science Module 4: Exploratory Data Analysis

Introduction to Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is an approach to analyzing datasets to summarize their main characteristics, often with visual methods. It was pioneered by John Tukey in the 1970s with the philosophy that data should be allowed to "speak for themselves" before formal modeling.

Goals of EDA:

Maximize Insight: Understand the underlying structure of the data
Uncover Patterns: Detect anomalies, outliers, and trends
Check Assumptions: Verify statistical assumptions for modeling
Generate Hypotheses: Formulate questions for further analysis
Data Quality Assessment: Validate cleaning decisions

Data Understanding

Univariate Analysis

Bivariate Analysis

Multivariate Analysis

Hypothesis Testing

Univariate Analysis

Univariate analysis examines one variable at a time. It's the simplest form of analysis where the data isn't analyzed for relationships or differences.

Numerical Variables:

0.00

Mean (μ)

0.00

Std Dev (σ)

0.00

Median

0.00

Range

Distribution Types:

Normal Distribution

Bell-shaped, symmetric

Uniform Distribution

Constant probability

Exponential

Right-skewed

Bimodal

Two peaks

Distribution Explorer

Adjust the parameters to see how different statistical measures change:

Distribution Type

Mean (μ): 0

Std Dev (σ): 1

Skewness: 0

Bivariate Analysis

Bivariate analysis examines the relationship between two variables. It helps understand if and how variables are related.

Correlation Analysis:

Correlation (r)	Strength	Direction
0.00 to ±0.19	Very weak	None
±0.20 to ±0.39	Weak	Slight
±0.40 to ±0.59	Moderate	Moderate
±0.60 to ±0.79	Strong	Substantial
±0.80 to ±1.00	Very strong	Nearly perfect

Correlation Matrix:

Age

Income

Experience

Age

1.00

0.75

0.82

Income

0.75

1.00

0.63

Experience

0.82

0.63

1.00

Color intensity indicates correlation strength

Scatter Plot Explorer

Explore different relationship patterns between variables:

Relationship Type

Correlation (r): 0.80

Noise Level: 0.2

Show Regression Line

Show Confidence Interval

0.80

Pearson's r

0.64

R² Value

0.001

p-value

Multivariate Analysis

Multivariate analysis examines relationships among multiple variables simultaneously. This includes dimensionality reduction techniques and advanced visualizations.

Dimensionality Reduction:

Linear transformation that identifies orthogonal directions of maximum variance.

Eigenvectors: Directions of maximum variance
Eigenvalues: Amount of variance explained
Scree Plot: Visualizes variance per component

Non-linear technique preserving local structure, ideal for visualization.

Perplexity: Balances local/global structure
Learning Rate: Step size for optimization
Preserves: Local neighborhoods in high-D space

Manifold learning preserving both local and global structure.

n_neighbors: Local neighborhood size
min_dist: Minimum distance in low-D space
Fast: Scales better than t-SNE

3D Scatter Plot:

Parallel Coordinates Plot

Visualize multivariate data by plotting each variable on a separate vertical axis:

Variables:

Age

Income

Experience

Education

Satisfaction

Color by Group

Hypothesis Testing in EDA

Formal hypothesis testing allows us to make statistically sound inferences about populations based on sample data.

Common Statistical Tests:

t-Test

For means

Compare means between two groups. Assumes normality and equal variances.

H₀: μ₁ = μ₂ H₁: μ₁ ≠ μ₂

ANOVA

For 3+ groups

Compare means across three or more groups. Extends t-test.

H₀: All μ equal H₁: At least one μ differs

Chi-Square Test

For independence

Test independence between categorical variables.

H₀: Variables independent H₁: Variables dependent

A/B Test Simulation:

Group A Mean

Group B Mean

Sample Size per Group: 100

Comprehensive EDA Exercise

In this exercise, you'll perform a complete EDA on a real-world dataset of housing prices. You'll need to explore the data, generate insights, and answer specific questions.

Housing Price EDA Exercise

# COMPREHENSIVE EXPLORATORY DATA ANALYSIS
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import warnings
warnings.filterwarnings('ignore')

# Set plot style
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_palette("husl")

# ========== 1. LOAD AND EXPLORE DATA ==========
print("=== STEP 1: DATA LOADING AND INITIAL EXPLORATION ===")

# Load the housing dataset
data = {
    'price': [350000, 450000, 550000, 650000, 750000, 425000, 525000, 625000, 375000, 475000,
              575000, 675000, 325000, 425000, 525000, 625000, 725000, 395000, 495000, 595000],
    'sqft': [1500, 1800, 2200, 2400, 2800, 1650, 2000, 2300, 1450, 1750,
             2100, 2500, 1400, 1700, 1900, 2250, 2700, 1550, 1850, 2150],
    'bedrooms': [3, 3, 4, 4, 5, 3, 4, 4, 3, 3, 4, 5, 3, 3, 4, 4, 5, 3, 4, 4],
    'bathrooms': [2, 2.5, 3, 3.5, 4, 2, 2.5, 3, 1.5, 2, 2.5, 3.5, 2, 2, 2.5, 3, 3.5, 2, 2.5, 3],
    'age': [15, 10, 5, 3, 1, 12, 8, 4, 20, 18, 7, 2, 25, 22, 9, 6, 1, 17, 11, 8],
    'location': ['Urban', 'Suburban', 'Urban', 'Suburban', 'Urban', 'Suburban', 'Urban', 'Suburban',
                 'Urban', 'Suburban', 'Urban', 'Suburban', 'Urban', 'Suburban', 'Urban', 'Suburban',
                 'Urban', 'Suburban', 'Urban', 'Suburban']
}

df = pd.DataFrame(data)

print(f"Dataset shape: {df.shape}")
print("\nFirst 5 rows:")
print(df.head())
print("\nDataset information:")
print(df.info())
print("\nBasic statistics:")
print(df.describe())

# ========== 2. UNIVARIATE ANALYSIS ==========
print("\n\n=== STEP 2: UNIVARIATE ANALYSIS ===")
# TODO: Analyze each variable individually
# 2.1 Calculate summary statistics for numerical variables
# 2.2 Create histograms for each numerical variable
# 2.3 Create boxplots to identify outliers
# 2.4 Analyze categorical variable distribution

# Your code here:

# ========== 3. BIVARIATE ANALYSIS ==========
print("\n\n=== STEP 3: BIVARIATE ANALYSIS ===")
# TODO: Analyze relationships between variables
# 3.1 Create scatter plot of price vs sqft
# 3.2 Calculate correlation matrix
# 3.3 Create heatmap of correlations
# 3.4 Analyze price by location (boxplot or violin plot)

# Your code here:

# ========== 4. MULTIVARIATE ANALYSIS ==========
print("\n\n=== STEP 4: MULTIVARIATE ANALYSIS ===")
# TODO: Analyze multiple variables simultaneously
# 4.1 Create pair plot of all numerical variables
# 4.2 Analyze interaction between sqft and bedrooms on price
# 4.3 Create 3D scatter plot (price, sqft, bedrooms)

# Your code here:

# ========== 5. HYPOTHESIS TESTING ==========
print("\n\n=== STEP 5: HYPOTHESIS TESTING ===")
# TODO: Perform statistical tests
# 5.1 Test if price differs significantly between urban and suburban locations
# 5.2 Test correlation between price and sqft
# 5.3 Test if number of bedrooms affects price

# Your code here:

# ========== 6. INSIGHTS AND RECOMMENDATIONS ==========
print("\n\n=== STEP 6: KEY INSIGHTS ===")
# TODO: Summarize your findings
# 6.1 What are the strongest predictors of price?
# 6.2 Are there any outliers that need investigation?
# 6.3 What recommendations would you make to stakeholders?

# Your analysis here:

print("\nEDA Completed Successfully!")

Complete Solution:

# COMPLETE SOLUTION
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

# Load data
df = pd.DataFrame(data)

# 1. Univariate Analysis
print("=== UNIVARIATE ANALYSIS ===")
print("Summary Statistics:")
print(df.describe())

# Histograms
fig, axes = plt.subplots(2, 3, figsize=(15, 8))
df['price'].hist(ax=axes[0,0], bins=10)
axes[0,0].set_title('Price Distribution')
df['sqft'].hist(ax=axes[0,1], bins=10)
axes[0,1].set_title('Square Feet Distribution')
df['bedrooms'].hist(ax=axes[0,2], bins=5)
axes[0,2].set_title('Bedrooms Distribution')
df['bathrooms'].hist(ax=axes[1,0], bins=5)
axes[1,0].set_title('Bathrooms Distribution')
df['age'].hist(ax=axes[1,1], bins=10)
axes[1,1].set_title('Age Distribution')
fig.delaxes(axes[1,2])
plt.tight_layout()
plt.show()

# 2. Bivariate Analysis
print("\n=== BIVARIATE ANALYSIS ===")
print("Correlation Matrix:")
print(df.corr())

# Scatter plot
plt.figure(figsize=(10,6))
plt.scatter(df['sqft'], df['price'], alpha=0.7)
plt.xlabel('Square Feet')
plt.ylabel('Price')
plt.title('Price vs Square Feet')
plt.show()

# Heatmap
plt.figure(figsize=(8,6))
sns.heatmap(df.corr(), annot=True, cmap='coolwarm', center=0)
plt.title('Correlation Heatmap')
plt.show()

# 3. Multivariate Analysis
sns.pairplot(df, hue='location', diag_kind='hist')
plt.suptitle('Pair Plot of All Variables', y=1.02)
plt.show()

# 4. Hypothesis Testing
print("\n=== HYPOTHESIS TESTING ===")
urban_prices = df[df['location'] == 'Urban']['price']
suburban_prices = df[df['location'] == 'Suburban']['price']
t_stat, p_value = stats.ttest_ind(urban_prices, suburban_prices)
print(f"t-test for price by location: t={t_stat:.3f}, p={p_value:.4f}")

# Correlation test
corr_coef, p_value_corr = stats.pearsonr(df['sqft'], df['price'])
print(f"Correlation between price and sqft: r={corr_coef:.3f}, p={p_value_corr:.4f}")

# 5. Insights
print("\n=== KEY INSIGHTS ===")
print("1. Square footage is the strongest predictor of price (r=0.95)")
print("2. Urban houses are slightly more expensive on average")
print("3. Most houses have 3-4 bedrooms")
print("4. Age shows negative correlation with price")
print("5. No significant outliers detected in the data")

Module 4 Quiz: Exploratory Data Analysis

Test your understanding of EDA concepts, statistical methods, and visualization techniques.

1. What is the primary purpose of Exploratory Data Analysis (EDA)?

To build predictive models

To understand data patterns and generate hypotheses

To clean and preprocess data

To deploy machine learning models

2. Which visualization is most appropriate for showing the distribution of a continuous variable?

Scatter plot

Bar chart

Histogram

Heatmap

3. A correlation coefficient of -0.85 indicates:

No relationship

Weak negative relationship

Strong positive relationship

Strong negative relationship

4. In hypothesis testing, what does a p-value of 0.03 typically indicate?

Statistically significant result (reject H₀)

Non-significant result (fail to reject H₀)

Inconclusive result

Strong evidence for H₀

5. Which statistical test is appropriate for comparing means between three or more groups?

t-test

ANOVA

Chi-square test

Pearson correlation

6. What does the Shapiro-Wilk test assess?

Homogeneity of variances

Independence of observations

Normality of distribution

Correlation between variables

7. Which of these is NOT a dimensionality reduction technique?

PCA (Principal Component Analysis)

t-SNE (t-Distributed Stochastic Neighbor Embedding)

UMAP (Uniform Manifold Approximation and Projection)

KNN (K-Nearest Neighbors)

8. What is the main advantage of using a box plot?

Shows exact data points

Visualizes distribution shape and outliers

Shows correlation between variables

Works well with categorical data only

EDA Resources & Tools

Python Libraries for EDA

pandas: df.describe(), df.corr(), df.groupby()
matplotlib: Comprehensive 2D plotting
seaborn: Statistical data visualization
plotly: Interactive visualizations
scipy.stats: Statistical tests and distributions
pandas-profiling: Automated EDA reports

EDA Checklist

Use this comprehensive checklist for your EDA projects:

Understand data structure and types

Calculate summary statistics

Check for missing values and outliers

Visualize distributions (histograms, box plots)

Analyze relationships (scatter plots, correlations)

Create correlation matrix/heatmap

Test statistical hypotheses

Document insights and generate hypotheses

Sample Datasets for Practice

Housing Data

Predict housing prices based on features

Medical Data

Patient data with medical measurements

E-commerce Data

Customer purchase behavior and demographics

Introduction to Exploratory Data Analysis (EDA)

Goals of EDA:

Univariate Analysis

Numerical Variables:

Distribution Types:

Normal Distribution

Uniform Distribution

Exponential

Bimodal

Distribution Explorer

Bivariate Analysis

Correlation Analysis:

Correlation Matrix:

Scatter Plot Explorer

Multivariate Analysis

Dimensionality Reduction:

PCA (Principal Component Analysis)

t-SNE (t-Distributed Stochastic Neighbor Embedding)

UMAP (Uniform Manifold Approximation)

3D Scatter Plot:

Parallel Coordinates Plot

Variables:

Hypothesis Testing in EDA

Common Statistical Tests:

t-Test

ANOVA

Chi-Square Test

A/B Test Simulation:

Result:

Comprehensive EDA Exercise

Analysis Questions:

Visualizations Required:

Exercise Controls:

Hints

Complete Solution:

Module 4 Quiz: Exploratory Data Analysis

1. What is the primary purpose of Exploratory Data Analysis (EDA)?

2. Which visualization is most appropriate for showing the distribution of a continuous variable?

3. A correlation coefficient of -0.85 indicates:

4. In hypothesis testing, what does a p-value of 0.03 typically indicate?

5. Which statistical test is appropriate for comparing means between three or more groups?

6. What does the Shapiro-Wilk test assess?

7. Which of these is NOT a dimensionality reduction technique?

8. What is the main advantage of using a box plot?

Your Score: 0/8

Detailed Explanations:

EDA Resources & Tools

EDA Checklist

Sample Datasets for Practice

Housing Data

Medical Data

E-commerce Data

Dataset Loaded: