Agents & Reinforcement

What Is Reinforcement Learning?

Reinforcement Learning (RL) is a type of machine learning where an agent learns by interacting with an environment, taking actions that lead to rewards or penalties. The goal is to learn a policy that maximises the cumulative reward over time.

The agent explores, receives feedback in the form of rewards, and updates its strategy accordingly.

Agent–Environment Interaction

Reinforcement learning follows a repeating loop where the agent learns from feedback:

  • State (S): The agent observes the current situation.
  • Action (A): It chooses an action based on its current policy.
  • Reward (R): The environment responds with a reward.
  • Next State (S’): The environment transitions to a new state.

Example 1: In a maze, a robot observes its location (state), moves (action), receives feedback for correct moves (reward), and reaches a new cell (next state).

Example 2: In a customer support chatbot, the state is the current dialogue, the action is a response, and the reward is positive if the issue is resolved.

The Principle of Cumulative Reward

Unlike supervised learning, RL is driven by delayed rewards. The agent’s objective is to:

  • Maximise not just immediate rewards, but the total rewards it can collect over time.
  • Balance short-term gains with long-term strategy.

Example 1: In chess, early moves may not yield reward, but they set up a winning position later in the game.

Example 2: In warehouse automation, a robot sacrifices speed at first to learn a safer, more efficient long-term route.

Exploration vs. Exploitation Trade-Off

A core tension in RL is deciding between:

  • Exploration: Trying new or less known actions to discover potentially better rewards.
  • Exploitation: Leveraging known successful actions to maximise current rewards.

The agent must balance the two to learn effectively - too much exploration wastes time, too much exploitation misses out on better strategies.

Example 1: A delivery drone tries a new route (exploration) vs using the fastest known path (exploitation).

Example 2: A stock-trading bot must decide whether to stick with a consistently profitable stock or test a new one with uncertain returns.

Common Strategies to Balance the Trade-Off

ε-greedy (epsilon-greedy)
Most of the time pick the best action you currently know; with a small chance ε (say 10%), pick a random action to explore. You can shrink ε over time (e.g. from 0.3 → 0.1 → 0.01) so the agent explores a lot early on and settles down later.

Why use it? Very simple baseline; easy to implement and reason about.

UCB (Upper Confidence Bound)
Choose the action with the best balance of current average reward plus an uncertainty bonus. Actions tried less often have bigger bonuses, so they get tested; as evidence grows, bonuses shrink and the policy focuses on proven winners.

Why use it? Systematic exploration when rewards are fairly stable; prioritises actions you’re unsure about.

Thompson Sampling
Keep a belief (a simple probability model) about how good each action is; randomly draw a score from each belief and pick the action with the highest draw. Good actions tend to win more often, but uncertain actions still get chances, especially early on.

Why use it? Naturally balances exploration and exploitation; often performs well in practice without many knobs to tune.

Real-World Applications

Robotics

Robots use reinforcement learning to master tasks like walking, grasping, or avoiding obstacles. The agent (robot) interacts with its environment (e.g. a room), receives feedback (like balance or object detection), and adjusts its movements to maximise success. Over time, it learns an optimal policy through cumulative rewards. Balancing exploration (trying new paths) and exploitation (using stable ones) is critical to fine-tuning robot behaviour.

Game AI (Chess/Go)

Agents in games like chess or Go make decisions based on board states. Through trial and error, the AI explores various strategies, and over many episodes, it refines its moves to win more often. The environment is the game itself, with rewards tied to wins or achieving sub-goals. Cumulative reward and strategic exploration drive performance improvements against human or AI opponents.

Finance

Trading agents observe market states and decide whether to buy, sell, or hold. Each decision yields a profit or loss (reward), encouraging the agent to learn profitable strategies over time. The market acts as the environment, and the agent’s actions directly affect its outcomes. A good balance between exploiting known profitable patterns and exploring new strategies helps adapt to shifting market trends.

Healthcare

Personalised treatment agents suggest medication plans based on patient state data. The environment is patient response over time, and rewards are tied to improved outcomes or recovery. The system learns which treatments yield the best long-term results. Exploration helps discover new treatment paths, while exploitation focuses on what already works well.

Manufacturing

An industrial robotic arm uses RL to learn the most efficient way to assemble parts. The environment includes object positions and timing conditions. Success (e.g. correctly attaching a part) is rewarded, while failure (e.g. dropping it) incurs penalties. Over time, the agent optimises its movements. It explores new assembly sequences and exploits those that have proven successful.

Gaming (Video)

RL agents in video games evolve their strategy in real time by interacting with human players. The environment is the game scenario, actions are the agent’s moves, and rewards come from performance (e.g. surviving, scoring). The agent constantly tests new tactics (exploration) while refining successful ones (exploitation), aiming to win more often by learning from player behaviour.

Challenges in Reinforcement Learning

  • Long Training Times: RL often requires thousands or millions of interactions to learn effectively.
  • Reward Engineering: Defining a good reward function can be complex.
  • Exploration-Exploitation Balance: Too much of either can hurt learning.

Example: In autonomous racing, poor reward design can make a car drive in circles to collect easy points.

 Key Takeaways

  • Reinforcement learning trains agents to act in environments by maximising cumulative rewards.
  • It involves a cycle of state → action → reward → next state.
  • The exploration vs. exploitation balance is critical for long-term success.
  • RL is already transforming fields like robotics, games, finance, and healthcare.