Reinforcement Learning

Reinforcement Learning (RL) is a unique branch of Machine Learning where an “agent” learns to make decisions by performing actions in an environment to maximize a cumulative reward. Unlike Supervised Learning, there is no “answer key.” The agent learns from trial and error.

1. Core RL Concepts

To understand RL, you must understand the interaction loop between these components:

• Agent: The learner or decision-maker (e.g., a robot, a game character).
• Environment: Everything the agent interacts with (e.g., a chess board, a physical room).
• State ($s$): The current situation of the agent (e.g., coordinates on a map).
• Action ($a$): What the agent does (e.g., move left, jump, buy a stock).
• Reward ($r$): The immediate feedback from the environment (positive for good actions, negative for mistakes).

2. Reward Systems & Policy

The goal of the agent is not just to get the immediate reward, but the Maximum Cumulative Reward over time.

• Policy ($\pi$): The agent’s “strategy” or mapping from states to actions. It tells the agent what to do in any given situation.
• Discount Factor ($\gamma$): A value between 0 and 1 that determines how much the agent cares about future rewards vs. immediate rewards. (0 = short-sighted, 1 = visionary).

3. Markov Decision Process (MDP)

MDP is the mathematical framework used to describe the RL environment. It assumes the Markov Property: “The future is independent of the past, given the present.”

Essentially, you don’t need the history of how the agent got to the current state; the current state $s$ contains all the information needed to make the next decision.

4. Q-Learning (The Foundation)

Q-Learning is a “Value-Based” algorithm. It uses a Q-Table to store the “Quality” (Q-value) of an action taken in a specific state.

• The Q-Table: A lookup table where rows are states and columns are actions.
• The Logic: The agent looks at the table, finds the state it’s in, and picks the action with the highest Q-value.
• The Update: Every time the agent takes an action, it updates the table using the Bellman Equation:$$Q(s, a) = Q(s, a) + \alpha [R + \gamma \max Q(s’, a’) – Q(s, a)]$$(Where $\alpha$ is the learning rate).

5. Deep Q Networks (DQN)

In the real world, the number of possible states is often too large for a table (e.g., a video game screen has millions of pixel combinations).

DQN replaces the Q-Table with a Neural Network.

• The Neural Network takes the State as input and predicts the Q-values for all possible actions as output.
• Experience Replay: The agent stores its past experiences in a “buffer” and randomly samples from them to train the network. This prevents the model from only learning from its most recent (and potentially biased) actions.

6. Applications

Gaming

• AlphaGo: Used RL to beat the world champion in Go, a game with more possible moves than atoms in the universe.
• Atari Games: DQNs famously learned to play Breakout by realizing that digging a tunnel through the bricks was the most efficient way to get a high score.

Robotics

• Industrial Arms: Learning how to pick up fragile objects without breaking them through thousands of simulated trials.
• Legged Locomotion: Teaching four-legged robots (like those from Boston Dynamics) to walk over uneven terrain by rewarding “forward progress” and punishing “falling down.”

Finance

• Automated Trading: An agent learns to buy/sell stocks to maximize profit while minimizing risk (negative reward for losses).