Unleashing Large-Scale Video Generative Pre-training
for Visual Robot Manipulation

Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner . Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation.

We perform extensive end-to-end real robot experiments to evaluate how GR-1 works in the real world. We first evaluate on object transportation in three settings: Seen Objects, Unseen Instances (unseen object instances of seen categories), and Unseen Categories (unseen objects of unseen categories). Example instructions include "put the broccoli onto the plate". This task is challenging as the robot needs to ground the correct object in the scene and successfully grasp and place it in the correct area. We also perform experiments on contact-rich articulated object manipulation. Example instructions include "open the drawer". Results are shown in the below figure.

1. Seen Objects
In this setting, there are three objects, i.e. a broccoli, an eggplant, and a bell pepper. The robot is trained to perform transporting one of the three objects from the plate to the table or vice versa. We evaluate the performance on transporting these three objects in 1) training scenes 2) scenes with distrators and 3) scenes with background changes and distrators. In the scenes with distractors, we introduce three unseen objects, i.e. a corn, a tomato, and a yellow peach. In the scenes with background changes and distrators, we futher add a wooden board and an unseen bowl. GR-1 outperforms comparing baseline methods and achieve a high success rate in this setting.

2. Unseen Instances & Unseen Categories
In these two settings, we challenge the robot to transport unseen objects. There are five objects in the scene, i.e. a broccoli, an eggplant, a bell pepper, a yellow peach and a tomato. While "broccoli", "eggplant", and "bell peper" are seen in the languages during finetuning, these three instances are unseen in the robot training data. In Unseen Instances, we command the robot to transport these three instances. Both "yellow peach" and "tomato" are unseen in the languages of the robot training data. In Unseen Categories, the robot is instructed to transport these two objects. The performance of GR-1 drops when evaluted on unseen objects. However, it is surprising that the drop for Unseen Instances is modest. This demostrate GR-1 possesses strong zero-shot generalization to unseen instances. In the most challenging setting of unseen categories, a typical failure mode is that GR-1 sometimes mixes up the bell pepper with the peach which has a similar color.

3. Articulated Object Manipulation
In this experiment, we aim to evaluate GR-1 on handling contact-rich articulated object manipulation. We train GR-1 to open and close a drawer. GR-1 outperforms the comparing baseline methods. Typical failure modes of GR-1 include 1) failing to completely close the drawer in the closing task and 2) failing to engage with the drawer handle when pulling it out in the opening task.

CALVIN is a challenging benchmark focusing on learning language-conditioned policy for long-horizon robot manipulation. It contains 34 manipulation tasks and features unconstrained language instructions. The environment contains a Franka Emika Panda robot with a parallel-jaw gripper and a desk with a sliding door, a drawer that can be opened or closed, blocks with different colors, an LED, and a light bulb that can be turned on or off. CALVIN contains 4 different environments A, B, C, and D. These four environments are different in desk colors and object configurations. We perform experiments on CALVIN in four settings.

In this paper, we propose to leverage large-scale video generative pre-training for enhancing visual robot manipulation learning. We present GR-1, a strightforward GPT-style transformer that takes as input a language instruction, a sequence of observation images and robot states, and outputs actions and future images in an end-to-end manner. GR-1 is first pre-trained on language-conditioned video prediction with a large-scale video dataset. Owing to a flexible design, it can then be seamlessly finetuned on robot data to predict actions and future frames. GR-1 shows strong performance in both simulation and real robot experiments. By incorporating large-scale video data, we showcase that GR-1 is able to perform robustly in scenes which are disturbed heavily from those in the training data. More importantly, GR-1 is able to generalize to unseen object instances and categories in a zero-shot manner. In the future, we hope to incorporate more video data in training to further enhance the robustness and generalization capability of GR-1. In addition, we plan to scale up the robot data by increasing both the number of robot trajectories in diverse environments and the number of manipulation skills.

@misc{wu2023unleashing,
      title={Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation}, 
      author={Hongtao Wu and Ya Jing and Chilam Cheang and Guangzeng Chen and Jiafeng Xu and Xinghang Li and Minghuan Liu and Hang Li and Tao Kong},
      year={2023},
      eprint={2312.13139},
      archivePrefix={arXiv},
      primaryClass={cs.RO}}