Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning
Explaining outcome-level mode collapse and correcting it with inverse probability scaling.
Authors: Abhijeet Sinha, Sundari Elango, and Dianbo Liu
Venue: International Conference on Machine Learning (ICML), 2026
Expected-return objectives can concentrate reinforcement-learning policies on a narrow set of outcomes, even when many diverse high-reward outcomes exist. This work explains that outcome-level mode collapse and introduces inverse probability scaling to improve coverage while preserving high reward.