Now, suppose the same bandits could give us an additional information about the machine, e.g. what specific scheme running on the bandit machine, according to scheme the returns might change and this information could be useful in making optimal decision. Such information encompasing the complete description of an environment at an instance is a state, represented by \(s\). Hence, now the reward, action value and policy will also depend on the state of the environment and will be represented as \(r(s,a), q(s,a)\) and \(\pi(a|s)\) respectively.
To be Updated