Reinforcement Learning (RL) is a paradigm in Machine Learning, as the name suggests it is simply learning from interaction and reinforcement. The idea arrives from neuroscience, suggesting that humans and animals learn directly from neurochemicals such as dopamine, oxytocin, etc as feedback elements. In RL these feedback elements are summed under a term called reward. Similar to what theories suggest, that humans learn by maximizing this reinforcement, the RL objective is to maximize the reward.
In RL, we basically care about learning tasks that a human can perform without explicit information of the task. We call the entity which has this goal of learning by maximizing reward, an 'agent'. RL sees tasks as Sequential Decision Making, hence the agent has to decide what decision to take at each step. The RL framework assumes a simple setting. The agent is in a given environment, the agent can observe and interact with it by acting upon its decisions, to which the the environment changes its state and a feedback is recieved by the agent. Here is a digram that depicts this framework.
addimageThe agent observes the current state of the environment at time \( t \), denoted by \( s_t \), makes a decision on the state and takes an action \( a_t \). The environment responds to the action with a respective change in state, \( s_{t+1} \), and a reward to the agent for taking that action, \(r_t\). Then again, the agent observes the state \(s_{t+1}\) and takes an action on it, this continuous action-feedback loop continues until the task is completed or a terminal state is reached (generally represented by \(s_{goal}\) or \(s_T\)), here we say, an episode has ended and call the collected cummulative reward as the 'return', denoted by \( \mathcal{G} = \sum_t r_t\), which the agent aims to maximize. By interacting with environment across episodes, the agent learns sequences of actions at each state (\(a_t|s_t\)) that maximizes the cummulative reward. We call this sequence of choice of actions in a given state, a 'policy', given by \( \pi(a_t|s_t)\). In a setting where is no terminal state and agent continues interacting with environment indefinitely, the goal remains the same, to maximize the cummulative reward. We call such setting as non-episodic reinforcement learning.
Next, we will discuss RL in simpler settings. This series will largely focus on fundamentals of RL than its applications. If required, we may discuss benefits and drawbacks of the RL formulation in a separate post.
7 Feb 2025.
Next in series: Bandits.