Deep Learning Fundamentals

Deep Learning is a subfield of Machine Learning inspired by the human brain. It uses structures called Artificial Neural Networks (ANNs) to learn complex patterns from large amounts of data.

Deep Learning powers many modern technologies such as:

• Image recognition
• Speech recognition
• Chatbots
• Recommendation systems
• Autonomous vehicles

At its heart, deep learning is about learning representations automatically instead of manually crafting features.

---

1. Neural Networks Basics

What Is a Neural Network?

A neural network is a computational model made up of interconnected units called neurons, organized in layers. These neurons process input data, transform it, and produce output predictions.

A neural network tries to mimic how biological neurons work, but in a mathematical way.

---

Structure of a Neural Network

A basic neural network consists of:

1. Input Layer
   • Receives raw data (features)
   • Each neuron represents one feature
2. Hidden Layers
   • Perform transformations
   • Learn patterns and representations
   • “Deep” learning means multiple hidden layers
3. Output Layer
   • Produces final predictions
   • Could be a number, class label, or probability

---

Why Neural Networks Are Powerful

Traditional algorithms require manual feature engineering.
Neural networks learn features automatically from data.

Example:

• In image recognition:
  • Early layers learn edges
  • Middle layers learn shapes
  • Deep layers learn objects

---

2. Perceptron

What Is a Perceptron?

The perceptron is the simplest form of a neural network and the building block of deep learning.

It represents a single artificial neuron.

---

How a Perceptron Works

A perceptron:

1. Takes multiple inputs
2. Multiplies each input by a weight
3. Adds a bias
4. Passes the result through an activation function
5. Produces an output

Mathematically:

Output = Activation(Σ (input × weight) + bias)

---

Real-World Example

Imagine deciding whether to approve a loan:

• Input features: income, credit score, age
• Each feature has importance (weight)
• Bias represents baseline decision

The perceptron combines these to output:

• Approve (1)
• Reject (0)

---

Limitation of a Single Perceptron

• Can only solve linearly separable problems
• Cannot handle complex patterns

This limitation led to multi-layer neural networks.

---

3. Activation Functions

Why Activation Functions Are Needed

Without activation functions:

• Neural networks become simple linear models
• No matter how many layers you add, the model remains linear

Activation functions introduce non-linearity, allowing networks to learn complex patterns.

---

Common Activation Functions

---

1. Sigmoid

• Output range: 0 to 1
• Interpreted as probability

Used in:

• Binary classification output layers

Limitation:

• Vanishing gradient problem

---

2. Tanh

• Output range: -1 to 1
• Zero-centered

Better than sigmoid but still suffers from vanishing gradients.

---

3. ReLU (Rectified Linear Unit)

f(x) = max(0, x)

Advantages:

• Fast computation
• Reduces vanishing gradient
• Most widely used in hidden layers

---

4. Softmax

• Converts outputs into probabilities
• Sum of outputs = 1

Used in:

• Multi-class classification

---

Choosing Activation Functions

• Hidden layers → ReLU
• Binary output → Sigmoid
• Multi-class output → Softmax

---

4. Forward & Backpropagation

These two processes are the core learning mechanism of neural networks.

---

Forward Propagation

Forward propagation is the process of:

• Passing input data through the network
• Calculating outputs step by step
• Producing predictions

Flow:

Input → Hidden Layers → Output

Each neuron:

• Computes weighted sum
• Applies activation function
• Passes result forward

---

Backpropagation

Backpropagation is the process of learning from mistakes.

Steps:

1. Compare predicted output with actual output
2. Calculate error using a loss function
3. Propagate error backward
4. Update weights using gradients

This is done using calculus (chain rule).

---

Why Backpropagation Is Important

Without backpropagation:

• The network cannot learn
• Weights remain random
• Predictions never improve

Backpropagation allows the network to:

• Minimize error
• Improve accuracy over time

---

5. Loss Functions

What Is a Loss Function?

A loss function measures how wrong the model’s predictions are.

It provides a numerical value representing prediction error.

The goal of training:

Minimize the loss function

---

Common Loss Functions

---

1. Mean Squared Error (MSE)

Used for:

• Regression problems

Measures average squared difference between predicted and actual values.

---

2. Mean Absolute Error (MAE)

Less sensitive to outliers than MSE.

---

3. Binary Cross-Entropy

Used for:

• Binary classification

Measures how close predicted probabilities are to actual labels.

---

4. Categorical Cross-Entropy

Used for:

• Multi-class classification

---

Why Loss Functions Matter

Choosing the wrong loss function:

• Slows learning
• Produces incorrect results
• Causes poor convergence

---

6. Optimizers

What Is an Optimizer?

An optimizer decides how the network updates its weights to minimize loss.

It controls:

• Learning speed
• Stability
• Convergence quality

---

Why Optimizers Are Needed

Backpropagation calculates gradients, but optimizers:

• Decide how much to change weights
• Prevent overshooting minima
• Speed up training

---

Common Optimizers

---

1. Gradient Descent

• Updates weights using full dataset
• Slow for large data

---

2. Stochastic Gradient Descent (SGD)

• Updates weights using small batches
• Faster and more practical

---

3. Momentum

• Adds memory of past gradients
• Reduces oscillations

---

4. RMSprop

• Adjusts learning rate dynamically
• Good for non-stationary problems

---

5. Adam (Adaptive Moment Estimation)

• Combines momentum + RMSprop
• Most widely used optimizer

Adam is popular because:

• Fast convergence
• Minimal tuning
• Works well in most cases