Data Handling & Analysis

In AI and Machine Learning, data is more important than algorithms.
A simple model trained on clean, meaningful data often outperforms a complex model trained on poor data.
This module focuses on understanding, preparing, and analyzing data so that ML models can learn correctly and efficiently.

1. Understanding Datasets

A dataset is a collection of data points used to train, validate, and test machine learning models.

Components of a Dataset

• Rows (Records / Samples)
  Each row represents one observation or example.
• Columns (Features / Attributes)
  Each column represents a characteristic or property of the data.

Example:

• Dataset of students
  • Rows → individual students
  • Columns → age, marks, attendance, result

Types of Variables in a Dataset

Numerical Variables

• Continuous (salary, temperature)
• Discrete (number of purchases)

Categorical Variables

• Nominal (city, color)
• Ordinal (rating: low, medium, high)

Understanding variable types is crucial for choosing correct preprocessing techniques.

2. Structured vs Unstructured Data

Structured Data

Structured data is organized in a fixed format, usually in tables.

Examples:

• CSV files
• Excel sheets
• SQL databases

Characteristics:

• Easy to analyze
• Well-defined schema
• Used in most classical ML problems

Example:
Customer data with columns like age, gender, income

Unstructured Data

Unstructured data has no predefined format.

Examples:

• Text (emails, reviews)
• Images
• Audio
• Video

Characteristics:

• Requires preprocessing
• High dimensional
• Used in NLP and Computer Vision

Example:
Social media posts or medical images

Why This Matters in ML

• Structured data → traditional ML algorithms
• Unstructured data → deep learning techniques

3. Data Collection Techniques

Good models start with good data sources.

Common Data Sources

Databases

• SQL, NoSQL systems

APIs

• Social media APIs
• Weather APIs

Web Scraping

• Extracting data from websites

Sensors & IoT

• Temperature sensors
• Wearable devices

User-Generated Data

• Click logs
• Search history

Data Collection Challenges

• Incomplete data
• Biased samples
• Noisy measurements

Poor data collection leads to biased or inaccurate models.

4. Data Cleaning

Data cleaning is the process of fixing or removing incorrect, incomplete, or irrelevant data.

Common Data Quality Issues

• Missing values
• Duplicate records
• Inconsistent formats
• Typographical errors

Data Cleaning Steps

1. Remove duplicates
2. Fix incorrect values
3. Standardize formats (dates, text)
4. Validate ranges (age cannot be negative)

Clean data leads to stable and reliable ML models.

5. Handling Missing Values

Missing data is unavoidable in real-world datasets.

Why Data Goes Missing

• User skipped input
• Sensor failure
• Data corruption

Strategies to Handle Missing Values

1. Remove Rows or Columns

• Suitable when missing data is small
• Risk of losing information

2. Mean / Median Imputation

• Numerical data
• Median preferred when outliers exist

3. Mode Imputation

• Categorical data

4. Forward / Backward Fill

• Time-series data

5. Model-Based Imputation

• Predict missing values using ML

Choice of method impacts model bias and accuracy.

6. Outlier Detection

Outliers are extreme values that differ significantly from other observations.

Causes of Outliers

• Measurement error
• Data entry error
• Genuine rare events

Why Outliers Matter

• Can distort mean and variance
• Affect model learning
• Reduce accuracy

Common Outlier Detection Techniques

1. Z-Score Method

• Based on standard deviation

2. IQR (Interquartile Range)

• Uses quartiles
• More robust

3. Visualization

• Box plots
• Scatter plots

Outliers should be analyzed, not blindly removed.

7. Feature Scaling & Normalization

ML models perform better when features are on similar scales.

Why Scaling is Important

• Prevents dominance of large-scale features
• Improves convergence speed
• Essential for distance-based models

Feature Scaling Techniques

1. Normalization (Min-Max Scaling)

• Scales values to range [0, 1]

Used when:

• No extreme outliers
• Neural networks

2. Standardization (Z-score Scaling)

• Mean = 0, Std = 1

Used when:

• Data follows normal distribution
• Algorithms like SVM, Logistic Regression

8. Feature Encoding

ML models require numerical input, but real data often contains categories.

Label Encoding

• Assigns numbers to categories

Example:

• Male → 0
• Female → 1

Problem:

• Introduces unintended order

Used when:

• Categories are ordinal

One-Hot Encoding

• Creates binary columns for each category

Example:
City → Delhi, Mumbai, Chennai

Used when:

• Categories are nominal
• No inherent order

Encoding Impact

Wrong encoding can mislead models and reduce accuracy.

9. Exploratory Data Analysis (EDA)

EDA is the process of understanding data before modeling.

Goals of EDA

• Identify patterns
• Detect anomalies
• Understand relationships
• Validate assumptions

Common EDA Techniques

Summary Statistics

• Mean, median, variance

Visualization

• Histograms
• Box plots
• Scatter plots
• Correlation heatmaps

Feature Relationships

• Correlation analysis
• Pair plots

EDA helps answer:

• Which features matter?
• Are variables correlated?
• Is data skewed?

Why Data Handling is Critical in AI/ML

• 70–80% of ML time is spent on data
• Clean data improves performance more than complex algorithms
• Good EDA prevents wrong modeling decisions