What happened
You sampled three options with hidden, drifting reward probabilities. Midway, the best option silently changed. How quickly your choices tracked the reversal measures your learning rate; your switching pattern reveals your exploration policy.
Why it happened
Each outcome generates a prediction error, the gap between expected and received reward, which nudges the option's value estimate. High learning rates adapt fast but chase noise; low rates are stable but slow after reversals. There is no free lunch, only a tradeoff.
Experimental design
Research protocol
Within-subjects; multi-armed bandit with non-stationary rewards
- Independent variable (IV)
- Choice option (arm) and trial position (including reversal at midpoint)
- Dependent variable(s) (DV)
- Choice allocation per arm
- Post-reversal adaptation rate (trials to shift preference)
- Total reward earned
- Measured constructs
- Reinforcement learningReward sensitivityBehavioral flexibility
- Operational definitions
Prediction error
δ = received reward − expected reward; drives associative learning (Rescorla & Wagner, 1972)
Explore–exploit tradeoff
Balance between sampling uncertain options and choosing the current best estimate
Variable-ratio schedule
Reinforcement on unpredictable trials; produces high, persistent response rates (Skinner, 1938)
- Key terms
Multi-armed bandit
Sequential choice among options with unknown payoff distributions; formal model of exploration.
Rescorla–Wagner rule
ΔV = αβ(λ − ΣV): associative strength updates proportionally to surprise.
Reversal learning
Contingency switch requiring extinction of old associations and acquisition of new ones.
The original experiment
Year
1938
Researchers
B. F. Skinner · Robert Rescorla · Allan Wagner
Sample
Foundational animal-learning programs, later formalized computationally
Key finding
Behavior tracks reinforcement probability; learning speed is proportional to prediction error, and variable schedules resist extinction.
Skinner, B. F. (1938). The Behavior of Organisms. / Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning. In Classical Conditioning II, 64–99.
Where this shows up in the world
A/B testing and recommendation systems are industrial bandit problems
Habit formation and extinction in behavior therapy
Why variable-ratio rewards make slot machines and feeds compulsive
Continue
Return to the facility