Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning over a given time horizon to compute an approximately optimal policy for a hypothesized reward function and then match this policy with expert demonstrations. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. Interestingly, an effective time horizon shorter than the ground-truth value often produces better results faster. This work formally analyzes this phenomenon and provides an explanation: the time horizon controls the complexity of an induced policy class and mitigates overfitting with limited data. This analysis serves as a guide for the principled choice of the effective horizon for IRL. It also prompts us to re-examine the classic IRL formulation: it is more natural to learn jointly the reward and the effective horizon rather than the reward alone with a given horizon. To validate our findings, we implement a cross-validation extension and the experimental results confirm the theoretical analysis.
We present a formal analysis showing that, with limited expert data, using a shorter discount factor or time horizon improves the generalization of the learned reward function to unseen states. In IRL settings where the effective horizon varies, the performance gap between the induced policy and the expert policy arises from two sources: (i) reward estimation error due to limited data during IRL, and (ii) policy optimization error from using an effective horizon shorter than the ground-truth. We prove that the effective horizon controls the complexity of the approximated policy class during IRL. As the horizon increases, reward estimation error grows—overfitting occurs because we estimate policies from a more complex class using limited expert data. On the other hand, as the horizon approaches the ground-truth, the policy optimization error decreases. These opposing errors suggest that an intermediate effective horizon balances this trade-off and produces the most expert-like policy.
We empirically examine our theoretical findings by exploring how discount factors influence IRL performance with varying amounts of expert data. We adapt Linear Programming IRL (LP-IRL) and Maximum Entropy IRL (MaxEnt-IRL) to accommodate different discount factor settings and expert data sizes. Specifically, We evaluate the performance of LP-IRL and MaxEnt-IRL on four Gridworld and Objectworld tasks with varying reward complexity. To jointly learn the reward-horizon pair, we incorporate cross-validation into our modified IRL methods to optimize the discount factor. By evaluating policies across a range of discount factors/horizons, cross-validation allows us to directly observe how varying b𝛾 affects IRL performance with different amounts of expert data.
The error counts measure the number of states for which a policy's action selection deviates from the expert's actions. Each task displays the ground-truth value function (column 1), reward function (column 2), expert policy (column 3), error count curves for different amount of expert data in a single instance (columns 4–8), and the error count curve summary for a batch of 10 MDPs across varying amount of expert data (column 9). In all four tasks, ground-truth discount factor is 0.99. The optimal discount factor is smaller than the ground-truth value for varying amount of expert data.
For each task, we select the optimal discount factors for all 10 sampled environments through cross-validation. The orange curves illustrate how the optimal discount factor changes with the amount of expert data, while the green curves show the corresponding error counts. The ground-truth value (0.99) is depicted in grey, with its error counts displayed in blue. As the amount of expert data increases, the optimal discount factor initially decreases sharply and then gradually increases, indicating that overfitting is prominent when expert data is scarce.
For each task, we present the ground-truth value function (column 1), ground-truth reward function (column 2), expert policy (column 3), error count curves in all states for different amount of expert data for a single instance of the task (columns 4–8), and finally, a summary of the error curves for a batch of 10 MDPs (column 9). In all four tasks, the ground-truth horizon is 20. The optimal horizon is smaller than the ground-truth value for varying amount of expert data.
We select the optimal horizon values for each of 10 sampled task environments using cross-validation, based on the amount of expert data. Orange curves show how optimal horizon value changes with the amount of expert data, while green curves display corresponding error counts. The ground-truth horizon (20) is depicted by a grey line, with corresponding error lines in blue. The trends are consistent with LP-IRL.
@inproceedings{xu2025effective,
title={On the Effective Horizon of Inverse Reinforcement Learning},
author={Xu, Yiqing and Doshi-Velez, Finale and Hsu, David},
booktitle={International Conference on Autonomous Agents and Multiagent Systems},
year={2025}
}