The most disruptive innovation of the R1-V project is its cost-benefit ratio. According to the paper's data, the special reinforcement learning training strategy used in the project enables a 2B-scale model to outperform a 72B-scale conventional model that requires tens of times the computational resources by consuming only $2.62 in training costs (8*A100 GPUs*30 minutes).
The key to achieving this breakthrough lies in three technical optimizations: first, a sample-efficient reward computation module is designed to increase the training sample utilization by 80%; second, a gradient accumulation strategy is adopted to effectively reduce the GPU memory occupation by 90%; and third, a dynamic course learning algorithm is developed to enable the model to automatically adjust the learning focus in different training phases. These technological innovations make the amount of information in each parameter update reach 5-8 times of the traditional method.
The project's open source code shows that the whole training system contains 17 core optimizer components and supports mixed-accuracy training and distributed computation, which makes it easy for small and medium-sized organizations to reproduce the paper's results. Comparison data shows that to achieve the same task accuracy, the computational energy consumption of the R1-V solution is only 1/47 of the Transformer baseline.
This answer comes from the articleR1-V: Low-Cost Reinforcement Learning for Visual Language Model Generalization CapabilitiesThe
