A Markov Decision Process (MDP) is a mathematical framework used to model sequential decision-making problems where the outcome of an action is uncertain. It is widely used in Artificial Intelligence (AI), Machine Learning, Robotics and Reinforcement Learning to help an agent choose actions that maximize long-term rewards while interacting with an environment.
- An MDP is represented by the tuple (S, A, P, R, γ), where S denotes the set of states, A the available actions, P the transition probabilities, R the reward function and γ the discount factor.
- Using these components, an agent learns an optimal strategy (policy) that determines the best action to take in every state.
Components
1. States (S): A state is a situation or condition the agent can be in. For example, A position on a grid like being at cell (1,1).
2. Actions (A): An action is something the agent can do. For example, Move UP, DOWN, LEFT or RIGHT. Each state can have one or more possible actions.
3. Transition Model (T): The model tells us what happens when an action is taken in a state. It’s like asking: “If I move RIGHT from here, where will I land?” Sometimes the outcome isn’t always the same that’s uncertainty. For example:
- 80% chance of moving in the intended direction
- 10% chance of slipping to the left
- 10% chance of slipping to the right
This randomness is called a stochastic transition.
4. Reward (R): A reward is a number given to the agent after it takes an action. If the reward is positive, it means the result of the action was good. If the reward is negative it means the outcome was bad or there was a penalty help the agent learn what’s good or bad. Examples:
- +1 for reaching the goal
- -1 for stepping into fire
- -0.1 for each step to encourage fewer moves
5. Policy (π): A policy is the agent’s plan. It tells the agent: “If you are in this state, take this action.” The goal is to find the best policy that helps the agent earn the highest total reward over time.
Let’s consider a 3x4 grid world. The agent starts at cell (1,1) and aims to reach the Blue Diamond at (4,3) while avoiding Fire at (4,2) and a Wall at (2,2). At each state the agent can take one of the following actions: UP, DOWN, LEFT or RIGHT
1. Movement with Uncertainty (Transition Model)
The agent’s moves are stochastic (uncertain):
- 80% chance of going in the intended direction.
- 10% chance of going left of the intended direction.
- 10% chance of going right of the intended direction.
2. Reward System
- +1 for reaching the goal.
- -1 for falling into fire.
- -0.04 for each regular move (to encourage shorter paths).
- 0 for hitting a wall (no movement or penalty).
3. Goal and Policy
- The agent’s objective is to maximize total rewards.
- It must find an optimal policy: the best action to take in each state to reach the goal quickly while avoiding danger.
4. Path Example
- One possible optimal path is: UP → UP → RIGHT → RIGHT → RIGHT
- But because of randomness the agent must plan carefully to avoid accidentally slipping into fire.
Applications
- Robots use MDPs to decide how to move safely and efficiently in places like factories or warehouses and avoid obstacles.
- In board games or video games MDPs help characters to choose the best moves to win or complete tasks even when outcomes are not certain.
- Doctors can use it to plan treatments for patients, choosing actions that improve health while considering uncertain effects.
- Self-driving cars or delivery vehicles use it to find safe routes and avoid accidents on unpredictable roads.
- Stores and warehouses use MDPs to decide when to order more stock so they don’t run out or keep too much even when demand changes.
