Correcting robot manipulation policies through human intervention holds great promise for real-world deployment, yet human operators are inherently imperfect in both the actions they provide and the timing of their intervention signals. While the former has been extensively discussed in reinforcement learning (RL), the latter remains underexplored. At high control frequencies, human intervention signals are often delayed and inconsistent across time and state space. In this work, we present GAINS, a framework for leveraGing inconsistent humAn InterventioN Signals in reinforcement learning. At the core of GAINS, we employ distributional RL with quantile Q-networks to explicitly model uncertainty in intervention signals, capturing the inherent variability in human feedback. Building on this distributional representation, we introduce a pessimistic exploration strategy that ensures safe and sample-efficient learning under human corrections. We evaluate GAINS on four diverse simulated manipulation tasks and three challenging real-world scenarios against state-of-the-art intervention-based methods. GAINS achieves a 22% higher task success rate than RLIF and improves recovery success by up to 44% in failure scenarios. These results suggest that explicitly modeling and adapting to human imperfection is crucial for deploying intervention-based learning in real-world robotic systems.
GAINS leverages truncated distributional Q estimation to capture mixed learning signals arising from inconsistent interventions and sparse task rewards. Building on this distributional representation, we further introduce a reward-scaled Huber loss and a pessimistic exploration strategy, enabling safe yet efficient online learning.
GAINS provides a unified real-sim HIL-RL training system. Human operators can intervene through different devices in either simulation or real-world tasks, while the training pipeline is unified under an asynchronous actor--critic learning framework built on LeRobot. This modular design decouples teleoperation, environment interaction, data collection, and policy learning, making the system easy to extend to new robots, methods, tasks, and intervention interfaces.
Pressing the space bar triggers human intervention in simulation. In the illustration, red indicates active human intervention, while green denotes autonomous robot behavior.
System Structure in GAINS
Intervention in HIL-RM
The video below demonstrates the training process in HIL-RM. The green bounding box indicates autonomous robot actions, while the red bounding box indicates periods of human intervention. ✓ indicates task success, and ✗ indicates task failure.
Pick-Cube success case
Pick-Cube failure case
Open-Cabinet success case
Open-Cabinet failure case
Arrange-Boxes success case
Arrange-Boxes failure case
Straighten-Rope success case
Straighten-Rope failure case
We evaluate GAINS on two challenging real-world manipulation tasks: Pick-Spoon on a UR5 robot and Insert-USB On a Franka Emika robot. The videos below demonstrate the human-in-the-loop training process on real robotic platforms. As training progresses, the frequency of human interventions gradually decreases, while the task success rate steadily improves, indicating that the robot becomes increasingly capable of completing the tasks autonomously.
Pick-Spoon on UR5
Insert-USB on Franka Emika
We evaluate GAINS on Pick-Spoon and Insert-USB. The videos below demonstrate the evaluation results on real robotic platforms.
Pick-Spoon on UR5
Insert-USB on Franka Emika
In this work we propose GAINS, a human-in-the-loop reinforcement learning (HIL-RL) framework that leverages inconsistent human interventions via distributional Q estimation. Unlike prior methods that rely on scalar Q-functions and thus fail to fully capture the multimodal nature of human feedback, our approach provides a more expressive modeling of human interventions. We demonstrate that distributional RL with a reward-scaled Huber loss and a pessimistic exploration strategy significantly improves success rates, and can even surpass expert-level performance, enabling efficient and safe online learning. To further substantiate our approach, we also evaluated its effectiveness on two real-world robotic systems. Beyond these results, we introduce HIL-RM, a benchmark with built-in expert policies, to facilitate fair and reproducible evaluation, comprising four challenging sparse-reward manipulation tasks, with the potential to support broader research on HIL-RL.