sFyTZEqmUY.pdf.json

{
    "abstractText": "Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different axes (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, UniSim can emulate how humans and agents interact with the world by simulating the visual outcome of both high-level instructions such as \u201copen the drawer\u201d and low-level controls such as \u201cmove by x, y\u201d from otherwise static scenes and objects. There are numerous use cases for such a real-world simulator. As an example, we use UniSim to train both high-level vision-language planners and low-level reinforcement learning policies, each of which exhibit zero-shot real-world transfer after training purely in a learned real-world simulator. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience in UniSim, opening up even wider applications. Video demos can be found at anonymous-papers-submissions.github.io.",
    "authors": [],
    "id": "SP:85565efbff8f2acb90a284bd0efd37d92e953e3c",
    "references": [
        {
            "authors": [
                "Alessandro Achille",
                "Stefano Soatto"
            ],
            "title": "A separation principle for control in the age of deep learning",
            "venue": "Annual Review of Control, Robotics, and Autonomous Systems,",
            "year": 2018
        },
        {
            "authors": [
                "Peter Anderson",
                "Qi Wu",
                "Damien Teney",
                "Jake Bruce",
                "Mark Johnson",
                "Niko S\u00fcnderhauf",
                "Ian Reid",
                "Stephen Gould",
                "Anton Van Den Hengel"
            ],
            "title": "Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments",
            "venue": "In Proceedings of the IEEE conference on computer vision and pattern recognition,",
            "year": 2018
        },
        {
            "authors": [
                "Marcin Andrychowicz",
                "Filip Wolski",
                "Alex Ray",
                "Jonas Schneider",
                "Rachel Fong",
                "Peter Welinder",
                "Bob McGrew",
                "Josh Tobin",
                "Pieter Abbeel",
                "Wojciech Zaremba"
            ],
            "title": "Hindsight experience replay",
            "venue": "In Advances in Neural Information Processing Systems",
            "year": 2017
        },
        {
            "authors": [
                "Rohan Anil",
                "Andrew M Dai",
                "Orhan Firat",
                "Melvin Johnson",
                "Dmitry Lepikhin",
                "Alexandre Passos",
                "Siamak Shakeri",
                "Emanuel Taropa",
                "Paige Bailey",
                "Zhifeng Chen"
            ],
            "title": "Palm 2 technical report",
            "venue": "arXiv preprint arXiv:2305.10403,",
            "year": 2023
        },
        {
            "authors": [
                "Andreas Blattmann",
                "Robin Rombach",
                "Huan Ling",
                "Tim Dockhorn",
                "Seung Wook Kim",
                "Sanja Fidler",
                "Karsten Kreis"
            ],
            "title": "Align your latents: High-resolution video synthesis with latent diffusion models",
            "venue": "arXiv preprint arXiv:2304.08818,",
            "year": 2023
        },
        {
            "authors": [
                "Zal\u00e1n Borsos",
                "Rapha\u00ebl Marinier",
                "Damien Vincent",
                "Eugene Kharitonov",
                "Olivier Pietquin",
                "Matt Sharifi",
                "Dominik Roblek",
                "Olivier Teboul",
                "David Grangier",
                "Marco Tagliasacchi"
            ],
            "title": "Audiolm: a language modeling approach to audio generation",
            "venue": "IEEE/ACM Transactions on Audio, Speech, and Language Processing,",
            "year": 2023
        },
        {
            "authors": [
                "Anthony Brohan",
                "Noah Brown",
                "Justice Carbajal",
                "Yevgen Chebotar",
                "Joseph Dabis",
                "Chelsea Finn",
                "Keerthana Gopalakrishnan",
                "Karol Hausman",
                "Alex Herzog",
                "Jasmine Hsu"
            ],
            "title": "Rt-1: Robotics transformer for real-world control at scale",
            "venue": "arXiv preprint arXiv:2212.06817,",
            "year": 2022
        },
        {
            "authors": [
                "Anthony Brohan",
                "Noah Brown",
                "Justice Carbajal",
                "Yevgen Chebotar",
                "Xi Chen",
                "Krzysztof Choromanski",
                "Tianli Ding",
                "Danny Driess",
                "Avinava Dubey",
                "Chelsea Finn"
            ],
            "title": "Rt-2: Vision-language-action models transfer web knowledge to robotic control",
            "venue": "arXiv preprint arXiv:2307.15818,",
            "year": 2023
        },
        {
            "authors": [
                "Pablo Samuel Castro"
            ],
            "title": "Scalable methods for computing state similarity in deterministic markov decision processes",
            "venue": "In Proceedings of the AAAI Conference on Artificial Intelligence,",
            "year": 2020
        },
        {
            "authors": [
                "Angel Chang",
                "Angela Dai",
                "Thomas Funkhouser",
                "Maciej Halber",
                "Matthias Niessner",
                "Manolis Savva",
                "Shuran Song",
                "Andy Zeng",
                "Yinda Zhang"
            ],
            "title": "Matterport3d: Learning from rgb-d data in indoor environments",
            "venue": "arXiv preprint arXiv:1709.06158,",
            "year": 2017
        },
        {
            "authors": [
                "Chang Chen",
                "Yi-Fu Wu",
                "Jaesik Yoon",
                "Sungjin Ahn"
            ],
            "title": "Transdreamer: Reinforcement learning with transformer world models",
            "venue": "arXiv preprint arXiv:2202.09481,",
            "year": 2022
        },
        {
            "authors": [
                "Xi Chen",
                "Xiao Wang",
                "Soravit Changpinyo",
                "AJ Piergiovanni",
                "Piotr Padlewski",
                "Daniel Salz",
                "Sebastian Goodman",
                "Adam Grycner",
                "Basil Mustafa",
                "Lucas Beyer"
            ],
            "title": "Pali: A jointly-scaled multilingual language-image model",
            "venue": "arXiv preprint arXiv:2209.06794,",
            "year": 2022
        },
        {
            "authors": [
                "Xi Chen",
                "Josip Djolonga",
                "Piotr Padlewski",
                "Basil Mustafa",
                "Soravit Changpinyo",
                "Jialin Wu",
                "Carlos Riquelme Ruiz",
                "Sebastian Goodman",
                "Xiao Wang",
                "Yi Tay"
            ],
            "title": "Pali-x: On scaling up a multilingual vision and language model",
            "venue": "arXiv preprint arXiv:2305.18565,",
            "year": 2023
        },
        {
            "authors": [
                "Yung-Yu Chuang",
                "Dan B Goldman",
                "Ke Colin Zheng",
                "Brian Curless",
                "David H Salesin",
                "Richard Szeliski"
            ],
            "title": "Animating pictures with stochastic motion textures",
            "venue": "In ACM SIGGRAPH 2005 Papers,",
            "year": 2005
        },
        {
            "authors": [
                "\u00d6zg\u00fcn \u00c7i\u00e7ek",
                "Ahmed Abdulkadir",
                "Soeren S Lienkamp",
                "Thomas Brox",
                "Olaf Ronneberger"
            ],
            "title": "3d u-net: learning dense volumetric segmentation from sparse annotation",
            "venue": "19th International Conference,",
            "year": 2016
        },
        {
            "authors": [
                "Dima Damen",
                "Hazel Doughty",
                "Giovanni Maria Farinella",
                "Sanja Fidler",
                "Antonino Furnari",
                "Evangelos Kazakos",
                "Davide Moltisanti",
                "Jonathan Munro",
                "Toby Perrett",
                "Will Price"
            ],
            "title": "Scaling egocentric vision: The epic-kitchens dataset",
            "venue": "In Proceedings of the European conference on computer vision (ECCV),",
            "year": 2018
        },
        {
            "authors": [
                "Sudeep Dasari",
                "Frederik Ebert",
                "Stephen Tian",
                "Suraj Nair",
                "Bernadette Bucher",
                "Karl Schmeckpeper",
                "Siddharth Singh",
                "Sergey Levine",
                "Chelsea Finn"
            ],
            "title": "Robonet: Large-scale multi-robot learning",
            "year": 1910
        },
        {
            "authors": [
                "Danny Driess",
                "Fei Xia",
                "Mehdi SM Sajjadi",
                "Corey Lynch",
                "Aakanksha Chowdhery",
                "Brian Ichter",
                "Ayzaan Wahid",
                "Jonathan Tompson",
                "Quan Vuong",
                "Tianhe Yu"
            ],
            "title": "Palm-e: An embodied multimodal language model",
            "venue": "arXiv preprint arXiv:2303.03378,",
            "year": 2023
        },
        {
            "authors": [
                "Yilun Du",
                "Mengjiao Yang",
                "Bo Dai",
                "Hanjun Dai",
                "Ofir Nachum",
                "Joshua B Tenenbaum",
                "Dale Schuurmans",
                "Pieter Abbeel"
            ],
            "title": "Learning universal policies via text-guided video generation, 2023",
            "venue": "URL https://arxiv. org/abs/2302.00111,",
            "year": 2023
        },
        {
            "authors": [
                "Yuqing Du",
                "Olivia Watkins",
                "Zihan Wang",
                "C\u00e9dric Colas",
                "Trevor Darrell",
                "Pieter Abbeel",
                "Abhishek Gupta",
                "Jacob Andreas"
            ],
            "title": "Guiding pretraining in reinforcement learning with large language models",
            "venue": "arXiv preprint arXiv:2302.06692,",
            "year": 2023
        },
        {
            "authors": [
                "Gabriel Dulac-Arnold",
                "Daniel Mankowitz",
                "Todd Hester"
            ],
            "title": "Challenges of real-world reinforcement learning",
            "venue": "arXiv preprint arXiv:1904.12901,",
            "year": 2019
        },
        {
            "authors": [
                "Frederik Ebert",
                "Yanlai Yang",
                "Karl Schmeckpeper",
                "Bernadette Bucher",
                "Georgios Georgakis",
                "Kostas Daniilidis",
                "Chelsea Finn",
                "Sergey Levine"
            ],
            "title": "Bridge data: Boosting generalization of robotic skills with cross-domain datasets",
            "venue": "arXiv preprint arXiv:2109.13396,",
            "year": 2021
        },
        {
            "authors": [
                "Norm Ferns",
                "Prakash Panangaden",
                "Doina Precup"
            ],
            "title": "Metrics for finite markov decision processes",
            "venue": "In UAI,",
            "year": 2004
        },
        {
            "authors": [
                "Raghav Goyal",
                "Samira Ebrahimi Kahou",
                "Vincent Michalski",
                "Joanna Materzynska",
                "Susanne Westphal",
                "Heuna Kim",
                "Valentin Haenel",
                "Ingo Fruend",
                "Peter Yianilos",
                "Moritz Mueller-Freitag"
            ],
            "title": "The\u201d something something\u201d video database for learning and evaluating visual common sense",
            "venue": "In Proceedings of the IEEE international conference on computer vision,",
            "year": 2017
        },
        {
            "authors": [
                "Kristen Grauman",
                "Andrew Westbury",
                "Eugene Byrne",
                "Zachary Chavis",
                "Antonino Furnari",
                "Rohit Girdhar",
                "Jackson Hamburger",
                "Hao Jiang",
                "Miao Liu",
                "Xingyu Liu"
            ],
            "title": "Ego4d: Around the world in 3,000 hours of egocentric video",
            "venue": "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,",
            "year": 2022
        },
        {
            "authors": [
                "Danijar Hafner",
                "Timothy Lillicrap",
                "Mohammad Norouzi",
                "Jimmy Ba"
            ],
            "title": "Mastering atari with discrete world models",
            "venue": "arXiv preprint arXiv:2010.02193,",
            "year": 2020
        },
        {
            "authors": [
                "Danijar Hafner",
                "Jurgis Pasukonis",
                "Jimmy Ba",
                "Timothy Lillicrap"
            ],
            "title": "Mastering diverse domains through world models",
            "venue": "arXiv preprint arXiv:2301.04104,",
            "year": 2023
        },
        {
            "authors": [
                "Zekun Hao",
                "Xun Huang",
                "Serge Belongie"
            ],
            "title": "Controllable video generation with sparse trajectories",
            "venue": "In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,",
            "year": 2018
        },
        {
            "authors": [
                "Jonathan Ho",
                "Tim Salimans"
            ],
            "title": "Classifier-free diffusion guidance",
            "venue": "arXiv preprint arXiv:2207.12598,",
            "year": 2022
        },
        {
            "authors": [
                "Jonathan Ho",
                "Ajay Jain",
                "Pieter Abbeel"
            ],
            "title": "Denoising diffusion probabilistic models",
            "venue": "In Advances in Neural Information Processing Systems,",
            "year": 2020
        },
        {
            "authors": [
                "Jonathan Ho",
                "William Chan",
                "Chitwan Saharia",
                "Jay Whang",
                "Ruiqi Gao",
                "Alexey Gritsenko",
                "Diederik P Kingma",
                "Ben Poole",
                "Mohammad Norouzi",
                "David J Fleet"
            ],
            "title": "Imagen video: High definition video generation with diffusion models",
            "venue": "arXiv preprint arXiv:2210.02303,",
            "year": 2022
        },
        {
            "authors": [
                "Jonathan Ho",
                "Tim Salimans",
                "Alexey Gritsenko",
                "William Chan",
                "Mohammad Norouzi",
                "David J Fleet"
            ],
            "title": "Video diffusion models",
            "venue": "arXiv preprint arXiv:2204.03458,",
            "year": 2022
        },
        {
            "authors": [
                "Chao Jia",
                "Yinfei Yang",
                "Ye Xia",
                "Yi-Ting Chen",
                "Zarana Parekh",
                "Hieu Pham",
                "Quoc Le",
                "Yun-Hsuan Sung",
                "Zhen Li",
                "Tom Duerig"
            ],
            "title": "Scaling up visual and vision-language representation learning with noisy text supervision",
            "venue": "In International conference on machine learning,",
            "year": 2021
        },
        {
            "authors": [
                "Leslie Pack Kaelbling",
                "Tom\u00e1s Lozano-P\u00e9rez"
            ],
            "title": "Hierarchical task and motion planning in the now",
            "venue": "In 2011 IEEE International Conference on Robotics and Automation,",
            "year": 2011
        },
        {
            "authors": [
                "Leslie Pack Kaelbling",
                "Michael L Littman",
                "Anthony R Cassandra"
            ],
            "title": "Planning and acting in partially observable stochastic domains",
            "venue": "Artificial intelligence,",
            "year": 1998
        },
        {
            "authors": [
                "George Konidaris",
                "Andrew Barto"
            ],
            "title": "Skill discovery in continuous reinforcement learning domains using skill chaining",
            "venue": "Advances in neural information processing systems,",
            "year": 2009
        },
        {
            "authors": [
                "Ranjay Krishna",
                "Kenji Hata",
                "Frederic Ren",
                "Li Fei-Fei",
                "Juan Carlos Niebles"
            ],
            "title": "Dense-captioning events in videos",
            "venue": "In Proceedings of the IEEE international conference on computer vision,",
            "year": 2017
        },
        {
            "authors": [
                "Timoth\u00e9e Lesort",
                "Natalia"
            ],
            "title": "D\u0131\u0301az-Rodr\u0131\u0301guez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview",
            "venue": "Neural Networks,",
            "year": 2018
        },
        {
            "authors": [
                "Tsung-Yi Lin",
                "Michael Maire",
                "Serge Belongie",
                "James Hays",
                "Pietro Perona",
                "Deva Ramanan",
                "Piotr Doll\u00e1r",
                "C Lawrence Zitnick"
            ],
            "title": "Microsoft coco: Common objects in context",
            "venue": "In Computer Vision\u2013ECCV",
            "year": 2014
        },
        {
            "authors": [
                "Corey Lynch",
                "Pierre Sermanet"
            ],
            "title": "Language conditioned imitation learning over unstructured data",
            "venue": "arXiv preprint arXiv:2005.07648,",
            "year": 2020
        },
        {
            "authors": [
                "Corey Lynch",
                "Ayzaan Wahid",
                "Jonathan Tompson",
                "Tianli Ding",
                "James Betker",
                "Robert Baruch",
                "Travis Armstrong",
                "Pete Florence"
            ],
            "title": "Interactive language: Talking to robots in real time",
            "venue": "IEEE Robotics and Automation Letters,",
            "year": 2023
        },
        {
            "authors": [
                "Ajay Mandlekar",
                "Yuke Zhu",
                "Animesh Garg",
                "Jonathan Booher",
                "Max Spero",
                "Albert Tung",
                "Julian Gao",
                "John Emmons",
                "Anchit Gupta",
                "Emre Orbay"
            ],
            "title": "Roboturk: A crowdsourcing platform for robotic skill learning through imitation",
            "venue": "In Conference on Robot Learning,",
            "year": 2018
        },
        {
            "authors": [
                "Vincent Micheli",
                "Eloi Alonso",
                "Fran\u00e7ois Fleuret"
            ],
            "title": "Transformers are sample efficient world models",
            "venue": "arXiv preprint arXiv:2209.00588,",
            "year": 2022
        },
        {
            "authors": [
                "Antoine Miech",
                "Dimitri Zhukov",
                "Jean-Baptiste Alayrac",
                "Makarand Tapaswi",
                "Ivan Laptev",
                "Josef Sivic"
            ],
            "title": "Howto100m: Learning a text-video embedding by watching hundred million narrated video clips",
            "venue": "In Proceedings of the IEEE/CVF international conference on computer vision,",
            "year": 2019
        },
        {
            "authors": [
                "Volodymyr Mnih",
                "Koray Kavukcuoglu",
                "David Silver",
                "Alex Graves",
                "Ioannis Antonoglou",
                "Daan Wierstra",
                "Martin Riedmiller"
            ],
            "title": "Playing atari with deep reinforcement learning",
            "venue": "arXiv preprint arXiv:1312.5602,",
            "year": 2013
        },
        {
            "authors": [
                "Volodymyr Mnih",
                "Koray Kavukcuoglu",
                "David Silver",
                "Andrei A Rusu",
                "Joel Veness",
                "Marc G Bellemare",
                "Alex Graves",
                "Martin Riedmiller",
                "Andreas K Fidjeland",
                "Georg Ostrovski"
            ],
            "title": "Human-level control through deep reinforcement learning",
            "year": 2015
        },
        {
            "authors": [
                "Mathew Monfort",
                "SouYoung Jin",
                "Alexander Liu",
                "David Harwath",
                "Rogerio Feris",
                "James Glass",
                "Aude Oliva"
            ],
            "title": "Spoken moments: Learning joint audio-visual representations from video descriptions",
            "venue": "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,",
            "year": 2021
        },
        {
            "authors": [
                "Michael Montemerlo",
                "Sebastian Thrun"
            ],
            "title": "FastSLAM: A scalable method for the simultaneous localization and mapping problem in robotics, volume",
            "year": 2007
        },
        {
            "authors": [
                "Ofir Nachum",
                "Shixiang Shane Gu",
                "Honglak Lee",
                "Sergey Levine"
            ],
            "title": "Data-efficient hierarchical reinforcement learning",
            "venue": "Advances in neural information processing systems,",
            "year": 2018
        },
        {
            "authors": [
                "Colin Raffel",
                "Noam Shazeer",
                "Adam Roberts",
                "Katherine Lee",
                "Sharan Narang",
                "Michael Matena",
                "Yanqi Zhou",
                "Wei Li",
                "Peter J Liu"
            ],
            "title": "Exploring the limits of transfer learning with a unified text-to-text transformer",
            "venue": "J. Mach. Learn. Res.,",
            "year": 2020
        },
        {
            "authors": [
                "Santhosh Kumar Ramakrishnan",
                "Aaron Gokaslan",
                "Erik Wijmans",
                "Oleksandr Maksymets",
                "Alexander Clegg",
                "John M Turner",
                "Eric Undersander",
                "Wojciech Galuba",
                "Andrew Westbury",
                "Angel X Chang",
                "Manolis Savva",
                "Yili Zhao",
                "Dhruv Batra"
            ],
            "title": "Habitat-matterport 3d dataset (HM3d): 1000 largescale 3d environments for embodied AI",
            "venue": "In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,",
            "year": 2021
        },
        {
            "authors": [
                "Aditya Ramesh",
                "Prafulla Dhariwal",
                "Alex Nichol",
                "Casey Chu",
                "Mark Chen"
            ],
            "title": "Hierarchical textconditional image generation with clip latents",
            "venue": "arXiv preprint arXiv:2204.06125,",
            "year": 2022
        },
        {
            "authors": [
                "Paulo Rauber",
                "Avinash Ummadisingu",
                "Filipe Mutz",
                "J\u00fcrgen Schmidhuber"
            ],
            "title": "Hindsight policy gradients",
            "venue": "In International Conference on Learning Representations,",
            "year": 2019
        },
        {
            "authors": [
                "Andrei A Rusu",
                "Matej Ve\u010der\u0131\u0301k",
                "Thomas Roth\u00f6rl",
                "Nicolas Heess",
                "Razvan Pascanu",
                "Raia Hadsell"
            ],
            "title": "Sim-to-real robot learning from pixels with progressive nets",
            "venue": "In Conference on robot learning,",
            "year": 2017
        },
        {
            "authors": [
                "Manolis Savva",
                "Abhishek Kadian",
                "Oleksandr Maksymets",
                "Yili Zhao",
                "Erik Wijmans",
                "Bhavana Jain",
                "Julian Straub",
                "Jia Liu",
                "Vladlen Koltun",
                "Jitendra Malik"
            ],
            "title": "Habitat: A platform for embodied ai research",
            "venue": "In Proceedings of the IEEE/CVF international conference on computer vision,",
            "year": 2019
        },
        {
            "authors": [
                "Christoph Schuhmann",
                "Richard Vencu",
                "Romain Beaumont",
                "Robert Kaczmarczyk",
                "Clayton Mullis",
                "Aarush Katta",
                "Theo Coombes",
                "Jenia Jitsev",
                "Aran Komatsuzaki"
            ],
            "title": "Laion-400m: Open dataset of clip-filtered 400 million image-text pairs",
            "venue": "arXiv preprint arXiv:2111.02114,",
            "year": 2021
        },
        {
            "authors": [
                "Christoph Schuhmann",
                "Romain Beaumont",
                "Richard Vencu",
                "Cade Gordon",
                "Ross Wightman",
                "Mehdi Cherti",
                "Theo Coombes",
                "Aarush Katta",
                "Clayton Mullis",
                "Mitchell Wortsman"
            ],
            "title": "Laion-5b: An open large-scale dataset for training next generation image-text models",
            "venue": "Advances in Neural Information Processing Systems,",
            "year": 2022
        },
        {
            "authors": [
                "Younggyo Seo",
                "Kimin Lee",
                "Stephen L James",
                "Pieter Abbeel"
            ],
            "title": "Reinforcement learning with action-free pre-training from videos",
            "venue": "In International Conference on Machine Learning,",
            "year": 2022
        },
        {
            "authors": [
                "Aliaksandr Siarohin",
                "St\u00e9phane Lathuili\u00e8re",
                "Sergey Tulyakov",
                "Elisa Ricci",
                "Nicu Sebe"
            ],
            "title": "Animating arbitrary objects via deep motion transfer",
            "venue": "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,",
            "year": 2019
        },
        {
            "authors": [
                "Uriel Singer",
                "Adam Polyak",
                "Thomas Hayes",
                "Xi Yin",
                "Jie An",
                "Songyang Zhang",
                "Qiyuan Hu",
                "Harry Yang",
                "Oron Ashual",
                "Oran Gafni"
            ],
            "title": "Make-a-video: Text-to-video generation without text-video data",
            "venue": "arXiv preprint arXiv:2209.14792,",
            "year": 2022
        },
        {
            "authors": [
                "Richard D Smallwood",
                "Edward J Sondik"
            ],
            "title": "The optimal control of partially observable markov processes over a finite horizon",
            "venue": "Operations research,",
            "year": 1973
        },
        {
            "authors": [
                "Jascha Sohl-Dickstein",
                "Eric Weiss",
                "Niru Maheswaranathan",
                "Surya Ganguli"
            ],
            "title": "Deep unsupervised learning using nonequilibrium thermodynamics",
            "venue": "In International Conference on Machine Learning,",
            "year": 2015
        },
        {
            "authors": [
                "Edward Jay Sondik"
            ],
            "title": "The optimal control of partially observable Markov processes",
            "venue": "Stanford University,",
            "year": 1971
        },
        {
            "authors": [
                "Richard S. Sutton"
            ],
            "title": "Learning to predict by the methods of temporal differences",
            "venue": "Machine Learning,",
            "year": 1988
        },
        {
            "authors": [
                "Richard S Sutton",
                "Doina Precup",
                "Satinder Singh"
            ],
            "title": "Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning",
            "venue": "Artificial intelligence,",
            "year": 1999
        },
        {
            "authors": [
                "Yi Tay",
                "Mostafa Dehghani",
                "Vinh Q Tran",
                "Xavier Garcia",
                "Jason Wei",
                "Xuezhi Wang",
                "Hyung Won Chung",
                "Dara Bahri",
                "Tal Schuster",
                "Steven Zheng"
            ],
            "title": "Ul2: Unifying language learning paradigms",
            "venue": "In The Eleventh International Conference on Learning Representations,",
            "year": 2022
        },
        {
            "authors": [
                "Shengwei Wang",
                "Xinqiao Jin"
            ],
            "title": "Model-based optimal control of vav air-conditioning system using genetic algorithm",
            "venue": "Building and Environment,",
            "year": 2000
        },
        {
            "authors": [
                "Teng Wang",
                "Ruimao Zhang",
                "Zhichao Lu",
                "Feng Zheng",
                "Ran Cheng",
                "Ping Luo"
            ],
            "title": "End-to-end dense video captioning with parallel decoding",
            "venue": "In Proceedings of the IEEE/CVF International Conference on Computer Vision,",
            "year": 2021
        },
        {
            "authors": [
                "Xin Wang",
                "Jiawei Wu",
                "Junkun Chen",
                "Lei Li",
                "Yuan-Fang Wang",
                "William Yang Wang"
            ],
            "title": "Vatex: A large-scale, high-quality multilingual dataset for video-and-language research",
            "venue": "In Proceedings of the IEEE/CVF International Conference on Computer Vision,",
            "year": 2019
        },
        {
            "authors": [
                "Yaohui Wang",
                "Di Yang",
                "Francois Bremond",
                "Antitza Dantcheva"
            ],
            "title": "Latent image animator: Learning to animate images via latent space navigation",
            "venue": "arXiv preprint arXiv:2203.09043,",
            "year": 2022
        },
        {
            "authors": [
                "Chung-Yi Weng",
                "Brian Curless",
                "Ira Kemelmacher-Shlizerman"
            ],
            "title": "Photo wake-up: 3d character animation from a single photo",
            "venue": "In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,",
            "year": 2019
        },
        {
            "authors": [
                "Ronald J Williams"
            ],
            "title": "Simple statistical gradient-following algorithms for connectionist reinforcement learning",
            "venue": "Machine learning,",
            "year": 1992
        },
        {
            "authors": [
                "Chenfei Wu",
                "Lun Huang",
                "Qianxi Zhang",
                "Binyang Li",
                "Lei Ji",
                "Fan Yang",
                "Guillermo Sapiro",
                "Nan Duan"
            ],
            "title": "Godiva: Generating open-domain videos from natural descriptions",
            "venue": "arXiv preprint arXiv:2104.14806,",
            "year": 2021
        },
        {
            "authors": [
                "Ziyi Wu",
                "Nikita Dvornik",
                "Klaus Greff",
                "Thomas Kipf",
                "Animesh Garg"
            ],
            "title": "Slotformer: Unsupervised visual dynamics simulation with object-centric models",
            "venue": "arXiv preprint arXiv:2210.05861,",
            "year": 2022
        },
        {
            "authors": [
                "Jun Xu",
                "Tao Mei",
                "Ting Yao",
                "Yong Rui"
            ],
            "title": "Msr-vtt: A large video description dataset for bridging video and language",
            "venue": "In Proceedings of the IEEE conference on computer vision and pattern recognition,",
            "year": 2016
        },
        {
            "authors": [
                "Tianfan Xue",
                "Jiajun Wu",
                "Katherine L Bouman",
                "William T Freeman"
            ],
            "title": "Visual dynamics: Stochastic future generation via layered cross convolutional networks",
            "venue": "IEEE transactions on pattern analysis and machine intelligence,",
            "year": 2018
        },
        {
            "authors": [
                "Mengjiao Yang",
                "Dale Schuurmans",
                "Pieter Abbeel",
                "Ofir Nachum"
            ],
            "title": "Dichotomy of control: Separating what you can control from what you cannot",
            "venue": "arXiv preprint arXiv:2210.13435,",
            "year": 2022
        },
        {
            "authors": [
                "Mengjiao Yang",
                "Yilun Du",
                "Bo Dai",
                "Dale Schuurmans",
                "Joshua B Tenenbaum",
                "Pieter Abbeel"
            ],
            "title": "Probabilistic adaptation of text-to-video models",
            "venue": "arXiv preprint arXiv:2306.01872,",
            "year": 2023
        },
        {
            "authors": [
                "Sihyun Yu",
                "Kihyuk Sohn",
                "Subin Kim",
                "Jinwoo Shin"
            ],
            "title": "Video probabilistic diffusion models in projected latent space",
            "venue": "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,",
            "year": 2023
        },
        {
            "authors": [
                "Xiaohua Zhai",
                "Alexander Kolesnikov",
                "Neil Houlsby",
                "Lucas Beyer"
            ],
            "title": "Scaling vision transformers",
            "venue": "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,",
            "year": 2022
        },
        {
            "authors": [
                "Chenshuang Zhang",
                "Chaoning Zhang",
                "Mengchun Zhang",
                "In So Kweon"
            ],
            "title": "Text-to-image diffusion model in generative ai: A survey",
            "venue": "arXiv preprint arXiv:2303.07909,",
            "year": 2023
        },
        {
            "authors": [
                "Daquan Zhou",
                "Weimin Wang",
                "Hanshu Yan",
                "Weiwei Lv",
                "Yizhe Zhu",
                "Jiashi Feng"
            ],
            "title": "Magicvideo: Efficient video generation with latent diffusion models",
            "venue": "arXiv preprint arXiv:2211.11018,",
            "year": 2022
        }
    ],
    "sections": [
        {
            "heading": "1 INTRODUCTION",
            "text": "Generative models trained on internet data can now produce highly realistic text (OpenAI, 2023), speech (Borsos et al., 2023), image (Ramesh et al., 2022), and video (Ho et al., 2022a). Perhaps the ultimate goal of generative models is to be able to simulate every visual aspect of the world, from how cars are driven on a street to how furniture and meals are prepared. With a comprehensive real-world simulator, humans can \u201cinteract\u201d with diverse scenes and objects, robots can learn from simulated experience without risking physical damage, and a vast amount of \u201creal-world\u201d like data can be simulated to train other types of machine intelligence. One roadblock to building such a real-world simulator lies in the datasets that are available. While there are billions of texts, images, and video snippets available on the internet, different datasets cover different information that have to be brought together to simulate realistic experience of the world. For instance, paired text-image data contains rich scenes and objects but little movement (Lin et al., 2014; Schuhmann et al., 2022; Zhai et al., 2022), video captioning and question answering data contain rich high-level activity descriptions but little low-level movement detail (Xu et al., 2016; Krishna et al., 2017), human activity data contains rich human action but little mechanical motion (Miech et al., 2019; Grauman et al., 2022), and robotics data contains rich robot action but are limited in quantity (Dasari et al., 2019; Mandlekar et al., 2018). Since different datasets are curated by different industrial or research communities for different tasks, divergence in information is natural and hard to overcome, posing difficulties to building a real-world simulator that seeks to capture realistic experience of the world we live in. In this work, we take the first steps towards building a universal simulator (UniSim) of real-world interaction through generative modeling. Specifically, we propose to combine a wealth of data\u2014 ranging from internet text-image pairs, to motion and action rich data from navigation, manipulation, human activities, robotics, and data from simulations and renderings\u2014in a conditional video generation framework. With careful orchestration of diverse data, we show that UniSim can success-"
        },
        {
            "heading": "UniSim",
            "text": "fully merge different information and generalize beyond the data, enabling rich interaction through fine-grained motion control of otherwise static scenes and objects. To support long-horizon repeated interactions, we use UniSim as an observation prediction model to approximate sampling in a partially observable Markov decision processes (POMDPs) (Sondik, 1971; Smallwood & Sondik, 1973), and show that UniSim can simulate long-horizon interactions consistently across video generation boundaries. While the potential applications of UniSim are broad, we demonstrate three specific use cases. We first show how UniSim enables an embodied vision-language planner to perform long-horizon goalconditioned tasks through hindsight relabeling of simulated experience (Andrychowicz et al., 2017). In addition to high-level planning, we illustrate how UniSim can enable learning low-level control policies by leveraging model-based reinforcement learning (RL) (Sutton, 1988). Both the highlevel vision-language planner and the low-level control policy, while trained purely in simulation, can generalize to real robot settings. This is enabled by using simulators that are nearly visually indistinguishable from the real world, achieving one step towards bridging the sim-to-real gap in embodied learning (Rusu et al., 2017). Lastly, we note that UniSim may be used to simulate rare events where data collection is expensive or dangerous (e.g., crashes in self-driving cars). Such simulated videos can then be used to improve other machine intelligence such as rare event detectors, suggesting broad applications of UniSim beyond embodied learning. The main contributions can be summarized as follows: \u2022 We take the first step toward building a universal simulator (UniSim) of real-world interaction\nby combining diverse datasets rich in different axes\u2014such as objects, scenes, actions, motions, language, and motor controls\u2014in a unified video generation framework. \u2022 We formulate UniSim as an observation prediction model which approximates sampling in a POMDP, and utilize multi-frame history conditioning to obtain consistent and long-horizon video rollouts. \u2022 We illustrate how UniSim can be used to train embodied planners, low-level control policies, and video captioning models, enabling other machine intelligence to generalize to the real world when trained purely in simulation, thereby bridging the sim-to-real gap."
        },
        {
            "heading": "2 LEARNING INTERACTIVE REAL-WORLD SIMULATORS",
            "text": "We define a simulator of the real world as a model that, given some state of the world (e.g., an image frame), can take in some action as input, and produce some visual consequence of the action (in the form of a video) as output. The input action can be anything that causes some visual consequence, such as language instructions, robot controls, and camera movements. The simulator should support (1) rich interactions (diverse actions) from the initial image (e.g., \u201cmove left\u201d, \u201clook up\u201d) and (2) repeated interactions (e.g., \u201clook up\u201d is simulated after \u201cmove left\u201d is simulated)."
        },
        {
            "heading": "2.1 ORCHESTRATING DIVERSE DATASETS",
            "text": "Learning the simulator described above is nontrivial. Fortunately, there exist billions video clips visualizing consequences of billions of texts on the internet, as well as various robotic, 3D, and nav-\nigation datasets scattered across institutions. However, these datasets often contain actions in different formats (e.g., text description in different languages, low-level robot controls, camera angles). If we can express action from each dataset in a unified format, we can effectively fuse information across datasets through a universal interface relating actions to visual consequence of the actions (i.e., videos). We can then training a simulator that operates through this universal interface. Below, we highlight diverse information in these datasets and how to extract and fuse information into a common format (see all datasets used to train UniSim in Appendix B). \u2022 Simulated execution and renderings. While annotating actions for real-world videos is expen-\nsive, simulation engines such as Habitat (Savva et al., 2019) can render a wide variety of actions. We use datasets previously collected from these simulators, i.e., Habitat object navigation with HM3D (Ramakrishnan et al., 2021) and Language Table Data from Lynch et al. (2023) to train UniSim. We extract text descriptions as actions when available. For simulated continuous control actions, we encode them via language embeddings and concatenate the text embeddings with discretized control values. \u2022 Real robot data. An increasing amount of video data of real-robot executions paired with task descriptions such as the Bridge Data (Ebert et al., 2021) and data that enabled RT-1 and RT-2 (Brohan et al., 2022) are becoming increasingly available. Despite low-level control actions often being different across robots, the task descriptions can serve as high-level actions in UniSim. We further include discretize continuous controls actions when available similar to simulated robotics data. \u2022 Human activity videos. Rich human activity data such as Ego4D (Grauman et al., 2022), EPICKITCHENS (Damen et al., 2018), and Something-Something (Goyal et al., 2017) have been curated. Different from low-level robot controls, these activities are high-level actions that humans take to interact with the world. But these actions are often provided as labels for video classification or activity recognition tasks (Goyal et al., 2017). In this case, we convert the video labels into text actions. In addition, we subsample the videos to construct chunks of observations at a frame rate that captures meaningful actions. \u2022 Panorama scans. There exists a wealth of 3D scans such as Matterport3D (Chang et al., 2017). These static scans do not contain actions. We construct actions (e.g., turn left) by truncating panorama scans and utilize the information of camera poses between two images. \u2022 Internet text-image data. Paired text-image datasets such as LAION (Schuhmann et al., 2021) contain rich static objects without actions. However, the captions often contain motion information such as \u201ca person walking\u201d. To use image data in UniSim, we treat individual images as singleframe videos and image captions as actions.\nFor each of these datasets, we process text tokens into continuous representations using T5 language model embeddings (Raffel et al., 2020) to better fuse with continuous actions such as robot controls. Fuse Information into UniSim. Given the observation and action data extracted from the broad datasets above, we train a diffusion model to predict observations conditioned on actions and previous observations as shown in Figure 2.Since the observations from different environments have all been converted to videos, while actions of different modalities (e.g., text descriptions, motor con-\ntrols, camera angles) have all been converted to continuous embeddings, UniSim can learn a single world model across all datasets."
        },
        {
            "heading": "2.2 SIMULATING LONG-HORIZON INTERACTIONS THROUGH LEARNED POMDPS",
            "text": "While combining diverse data might enable rich interaction, the primary value of a simulator lies in simulating experiences to optimize decisions through search (Silver et al.), planning (Montemerlo & Thrun, 2007), or RL (Mnih et al., 2013) algorithms. In this section, we show how UniSim can be seen as learning a partially observable Markov decision process (POMDP) of real-world dynamics visually. This connection allows UniSim to be used for learning policies with exisiting decision making algorithms. The Real World as a POMDP. A POMDP can be defined as a tuple M := \u27e8S,A,O,R, T , E\u27e9 consisting of state, action, and observation spaces as well as reward, transition, and observation emission functions. A POMDP can characterize interactions with the real world, where st \u2208 S is the true state of the world, ot \u2208 O contains video frames, and at \u2208 A contains actions carried out by humans or agents, all at interactive step t. A policy \u03c0 can learn to choose actions that lead to high rewards through interacting with M. In this real-world POMDP, the frequency and duration of interactions vary (e.g., a human opens a door in 2 seconds, a motor executing a robot control in 0.2 second). As a result, actions can be some temporally extended high-level text descriptions (e.g., \u201copen the door\u201d) or some low-level motor controls (e.g., \u2206x,\u2206\u03c9). Therefore, we want a simulator that can support temporally extended actions, which have been found beneficial in learning hierarchical policies (Kaelbling & Lozano-Pe\u0301rez, 2011; Nachum et al., 2018), skills, and options (Sutton et al., 1999; Konidaris & Barto, 2009). UniSim for Approximating POMDP. Directly learning the POMDP of the real world requires learning distributions over the truth state of the world st, which is difficult. Instead, we can approximate this POMDP by learning an observation prediction model that enables drawing observation samples conditioned on history, i.e., ot \u223c p(ot|ht\u22121, at\u22121), where ht\u22121 denotes the history up to time t\u22121, e.g., (o0, a0, . . . , at\u22122, ot\u22121). In classical POMDP terms, ht\u22121 corresponds to some belief state that could be computed using a Bayesian filter over history (o0, a0, . . . , at\u22122, ot\u22121) Kaelbling et al. (1998). Since this observation prediction model cannot assign probabilities to particular state transitions or observations, it is not suited for POMDP algorithms that rely on this knowledge. Nevertheless, we will illustrate that samples from the observation prediction model can enable effective downstream planning (Section 4.1) and RL (Section 4.2). To simulate long interactions, we can sample from the observation prediction model p(ot|ht\u22121, at\u22121) autoregressively conditioned on the previously sampled observations. Note that while the observation prediction model only predict videos, reward signals R can be extracted from the generated videos for optimizing \u03c0, as we will illustrate in Section 4.2. Parametrizing and Training UniSim. We parametrize p(ot|ht\u22121, at\u22121) using diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) as an instantiation UniSim outlined in Figure 2. Specifically, the reverse process learns a denoising model \u03f5\u03b8(o (k) t , k|ht\u22121, at\u22121) that denoises the next observation using conditioned on the history using K denoising steps. In practice, we only use previous video frames and omit previous actions as history, and concatenate previous video frames with initial noise samples o(K)t \u223c N (0, I) channelwise to serve as conditional inputs to the denoising model. To condition on an action at\u22121, we leverage classifier-free guidance (Ho & Salimans, 2022). The final T (ot|ht\u22121, at\u22121) is parametrized by the variance schedule:\n\u03f5\u03b8(o (k) t , k|ht\u22121, at\u22121) = (1 + \u03b7)\u03f5\u03b8(o (k) t , k|ht\u22121, at\u22121)\u2212 \u03b7\u03f5\u03b8(ot, k|ot\u22121), (1)\nwhere \u03b7 controls action conditioning strength. With this parametrization, we train \u03f5\u03b8 by minimizing LMSE = \u2225\u2225\u2225\u03f5\u2212 \u03f5\u03b8(\u221a1\u2212 \u03b2(k)ot +\u221a\u03b2(k)\u03f5, k\u2223\u2223\u2223ht\u22121, at\u22121)\u2225\u2225\u22252 ,\nwhere \u03f5 \u223c N (0, I), and \u03b2(k) \u2208 R are a set of K different noise levels for each k \u2208 [1,K]. Given the learned \u03f5\u03b8, an observation ot can be generated by sampling from the initial distribution o (K) t \u223c N (0, I) and iteratively denoising according to the following process for k from K to 0\no (k\u22121) t = \u03b1 (k)(o (k) t \u2212 \u03b3(k)\u03f5\u03b8(o (k) t , k|ht\u22121, at\u22121)) + \u03be, \u03be \u223c N ( 0, \u03c32kI ) , (2)\nwhere \u03b3(k) is the denoising step size, \u03b1(k) is a linear decay on the current denoised sample, and \u03c3k is a time varying noise level that depends on \u03b1(k) and \u03b2(k)."
        },
        {
            "heading": "Pick up bowl",
            "text": "Architecture and Training. We use the video U-Net architecture (Ho et al., 2022b) to implement UniSim by employing interleaved temporal and spatial attention and convolution layers in both the downsampling and upsampling passes. For history conditioning, we replicate the conditioning frames at all future frame indices, and concatenate the conditioning frames with the noise sample for each of the future frame to serve as input to the U-Net. The UniSim model has 5.6B parameters and requires 512 TPU-v3 and 20 days to train on all data. See more details in Appendix C."
        },
        {
            "heading": "3 SIMULATING REAL-WORLD INTERACTIONS",
            "text": "We now demonstrate UniSim in emulating real-world manipulation and navigation environments by simulating both action-rich and long-horizon interactions for both humans and robots."
        },
        {
            "heading": "3.1 ACTION-RICH, LONG-HORIZON, AND DIVERSE INTERACTIONS",
            "text": "Action-Rich Simulation. We first demonstrate action-rich interactions with UniSim through natural language actions. Figure 3 shows simulation of human manipulation and navigation starting from the same initial observation (left-most column). We can instruct a person in the initial frame to perform various kitchen tasks (top left), press different switches (top right), or navigate scenes (bottom). The model only trained on generic internet data, without action-rich manipulation data such as EPICKITCHENS (Damen et al., 2018), fails to simulate action-rich manipulations (Appendix F). Long-Horizon Simulation. Next, we illustrate 8 sequential interactions with UniSim in Figure 4. We condition the simulation of each interaction on previous observations and new language action as described in Section 2.2. UniSim successfully preserves objects manipulated by previous instruc-"
        },
        {
            "heading": "Toothpaste Spider PlateUncover Pen Bottle Pickup 1. Put cup 2. Pen 3. Apple",
            "text": "tions (e.g., the orange and can are preserved in the drawers in Columns 4, 5, 7, 8 after being put in the drawers). See additional long-horizon interactions in Appendix A.1. Diversity and Stochasticity in UniSim UniSim can also support highly diverse and stochastic environment transitions, e.g., diverse objects being revealed after removing the towel on top (Figure 5 left), diverse object colors and locations (cups and pens in Figure 5 right), and real-world variabilities such as change in camera angles. Flexibility in diffusion models promotes simulation of highly stochastic environments that cannot be controlled by actions, so that a policy can learn to only control the controllable part (Yang et al., 2022)."
        },
        {
            "heading": "3.2 ABLATION AND ANALYSIS",
            "text": "Frame Conditioning Ablations. We ablate over choices of past frames to condition on using a validation split of the Ego4D dataset (Grauman et al., 2022), which contains egocentric movement requiring proper handling of observation history. We compare UniSim conditioned on different numbers of past frames in Table 1. Conditioning on 4 frames is better than conditioning on a single frame, but conditioning on history that is too far in the past (4 frames with exponentially increasing distances) can hurt performance. Increasing the number of conditioning frames beyond 4 did not further improve performance on Ego4D, but it could be helpful for applications that require memory from distant past (e.g., navigation for retrieval). Simulating Low-Data Domains. During joint training of UniSim on diverse data, we found that na\u0131\u0308vely combining datasets of highly varying size can result in low generation quality in low-data domains. While we can increase the weight of these domains in the data mixture during training, we found that attaching a domain identifier such as the name of the dataset to the actions being conditioned on improves generation quality in low-data domains, as shown in Figure 6."
        },
        {
            "heading": "4 APPLICATIONS OF UNISIM",
            "text": "We now demonstrate how UniSim can be used to train other types of machine intelligence such as embodied planners, RL agents, and vision-language models through simulating highly realistic experiences."
        },
        {
            "heading": "4.1 TRAINING LONG-HORIZON EMBODIED PLANNER THROUGH HINDSIGHT LABELING.",
            "text": "Language models and vision language models (VLM) have recently been used as policies or planners that can operate in image or text based observation and action spaces (Du et al., 2023b; Driess et al., 2023; Brohan et al., 2023). One major challenge in learning such agents lies in the need for large amounts of language action labels. The labor intensity in data collection only increases as tasks"
        },
        {
            "heading": "Start Goal",
            "text": "increase in horizon and complexity. UniSim can generate large amounts of training data for VLM policies through hindsight relabeling. Setup and Baseline. We use data from the Language Table environment (Lynch & Sermanet, 2020) for learning geometric rearrangements of blocks on a table. We train an image-goal conditioned VLM policy to predict language instructions and the motor controls from the start and goal images using the PALM-E architecture (Driess et al., 2023) (See data and model details in Appendix D.1). For the baseline, the goal is set to the last frame of the original short-horizon trajectories. During each evaluation run, we set the long-horizon goal by modifying the location of 3-4 blocks, and measure the blocks\u2019 distance to their goal states after executing 5 instructions using the VLM policy. We define the reduction in distance to goal (RDG) metric as\nRDG = \u2225s0 \u2212 sgoal\u22252 \u2212 \u2225sT \u2212 sgoal\u22252\n\u2225s0 \u2212 sgoal\u22252 , (3)\nwhere sT represents the underlying block locations after executing the policy, s0 and sgoal represents the initial and goal block locations. Generating Hindsight Data with UniSim. To use UniSim for long-horizon tasks, we draw inspiration from hindsight relabeling (Rauber et al., 2019). Specifically, we create a total of 10k long-horizon trajectories from UniSim by doing rollouts in UniSim 3-5 times per trajectory, where each rollout corresponds to one scripted language instruction. We then use the final frame from each long-horizon rollout as a goal input and the scripted language instructions as supervision for training the VLM policy. Results on Zero-shot Real-World Transfer. Figure 7 shows that the language plans produced by the VLM, the generated videos from UniSim according to the language plans, and the executions on the real robot. The policy purely trained in UniSim can directly perform long-horizon tasks in the real world. See additional sim-to-real results with zero-shot real-world transfer in Appendix A.2. Results on Simulated Evaluation. In addition to testing real-world transfer, we also conduct simulator based evaluation to compare the reduction in distance to goal (RDG) of the VLM policy using UniSim\u2019s generated long-horizon data to using the original short-horizon data in Table 2. The VLM trained using long-horizon generated data performs 3-4 times better than using the original data in completing long-horizon goal-conditioned tasks."
        },
        {
            "heading": "4.2 REAL-WORLD SIMULATOR FOR REINFORCEMENT LEARNING",
            "text": "Reinforcement learning (RL) has achieved superhuman performance on difficult tasks such as playing Go and Atari games (Silver et al.; Mnih et al., 2015), but has limited real world applications due to the lack of a realistic environment simulator (Dulac-Arnold et al., 2019). We investigate whether UniSim can enable effective training of RL agents by providing the agent with a realistic simulator that can be accessed in parallel. Setup. We finetune the PaLI 3B vision-language model (Chen et al., 2022b) to predict low-level control actions (joint movements in \u2206x,\u2206y) from an image observation and a task description (e.g., \u201cmove the blue cube to the right\u201d) using behavioral cloning (BC) to serve as the low-level control policy and the baseline, which we call the vision-language-action (VLA) policy similar to Brohan et al. (2023). Because UniSim can take low-level control actions as input, we can directly conduct model-based rollouts in UniSim using control actions outputted by VLA policy. To acquire reward information, we use the number of steps-to-completion from the training data as a proxy reward to train a model that maps the current observation to learned reward. We then use the REINFORCE algorithm (Williams, 1992) to optimize the VLA policy, treating the rollouts from UniSim as the on-policy rollouts from the real environment and use the learned reward model to predict rewards from simulated rollouts. See details of RL training in Appendix D.2. Results. We first do a sanity check of UniSim in simulating real-robot executions by applying low-level control actions (e.g., \u2206x = 0.05, \u03b4y = 0.05) repeatedly for 20-30 environment steps to move the endpoint left, right, down, up, and diagonally in Figure 8 (top two rows). We see that the simulated rollouts capture both the endpoint movements and the physics of collision. To compare the RL policy trained in UniSim to the BC policy, we qualitatively assessed the simulated rollouts in UniSim. Table 3 shows that RL training significantly improves the performance of the VLA policy across a wide set of tasks, especially in tasks such as \u201cpoint to blue block\u201d. We then directly deploy the RL policy trained in UniSim onto the real robot in zero-shot, and observe successful task executions as shown in Figure 8 (bottom row). Additional results on zero-shot transfer to real robot can be found in Appendix A.3."
        },
        {
            "heading": "4.3 REALISTIC SIMULATOR FOR BROADER VISION-LANGUAGE TASKS.",
            "text": "UniSim can generate training data for other machine intelligence (e.g., event detector). This is especially useful when natural data is rare or difficult to collect (e.g., footage of crimes or accidents). We provide such a proof-of-concept by training visionlanguage models on purely generated data from UniSim, and observe significant performance benefits in video captioning tasks. Setup. We finetune PaLI-X (Chen et al., 2023), a VLM with 55B parameters pretrained on a broad set of image, video, and language tasks, to caption a set of videos generated by UniSim using texts from the\ntraining split of ActivityNet Captions (Krishna et al., 2017). We measure the CIDEr score of the finetuned model on the test split of ActivityNet Captions as well as other captioning tasks following the same setup as Chen et al. (2023). See finetuning details of PaLI-X in Appendix D.3.\nResults. We compare PaLI-X finetuned on purely generated videos to pretrained PaLI-X without finetuning and PaLI-X finetuned on original ActivityNet Captions in Table 4. Purely finetuning on generated data drastically improves the captioning performance from no finetuning at all on ActivityNet (15.2 to 46.23), while achieving 84% performance of finetuning on true data. Furthermore, PaLI-X finetuned on generated data transfers better to other captioning tasks such as MSR-VTT (Xu et al., 2016), VATEX (Wang et al., 2019), and SMIT (Monfort et al., 2021) than PaLI-X finetuned on true data, which tends to overfit to ActivityNet. These results suggest that UniSim can serve as an effective data generator for improving broader vision-language models."
        },
        {
            "heading": "5 RELATED WORK",
            "text": "Internet-Scale Generative Models. Language models trained on internet text succeed at text-based tasks (OpenAI, 2023; Anil et al., 2023) but not physical tasks, which requires perception and control. Internet-scale generative models can synthesize realistic images and videos (Wu et al., 2021; Ho et al., 2022a; Singer et al., 2022; Yang et al., 2023; Blattmann et al., 2023), but have mostly been applied to generative media (Zhang et al., 2023) as opposed to empowering sophisticated agents capable of multi-turn interactions. Du et al. (2023a) shows video generation can serve as policies, but the major bottleneck for policy learning often lies in limited access to real-world environments (Dulac-Arnold et al., 2019). We focus on this exact bottleneck by learning universal simulators of the real world, enabling realistic and unlimited \u201cenvironment\u201d access for training sophisticated agents interactively. Learning World Models. Learning an accurate world model in reaction to control inputs has been a long-standing challenge in model-based planning, optimization, and reinforcement learning (Sutton, 1988; Kaelbling et al., 1998; Wang & Jin, 2000). Most systems choose to learn dynamics models in lower dimensional state spaces as opposed to in the pixel space (Ferns et al., 2004; Achille & Soatto, 2018; Lesort et al., 2018; Castro, 2020), which limits knowledge sharing across systems. With large transformer architectures, learning image-based world models became plausible (Hafner et al., 2020; Chen et al., 2022a; Seo et al., 2022; Micheli et al., 2022; Wu et al., 2022; Hafner et al., 2023), but mostly in games or simulated domains with visually simplistic and abundant data. In video generation, previous works have leveraged text prompts (Yu et al., 2023; Zhou et al., 2022), driving videos (Siarohin et al., 2019; Wang et al., 2022), 3D geometries (Weng et al., 2019; Xue et al., 2018), physical simulations (Chuang et al., 2005), frequency information (Li et al., 2023), and user annotations (Hao et al., 2018) to introduce movements into videos. However, they focus on generating domain specific videos as opposed to building a universal simulator that can further improve other agents as in UniSim."
        },
        {
            "heading": "6 LIMITATIONS AND CONCLUSION",
            "text": "We have shown it is possible to learn a simulator of the real world in response to various action inputs ranging from texts to robot controls. UniSim can simulate visually realistic experiences for interacting with humans and training autonomous agents. We hope UniSim will instigate broad interest in learning and applying real-world simulators to improve machine intelligence. UniSim has a few limitations that call for future work: \u2022 Hallucination. When an action is unrealistic given the scene (e.g., \u201cwash hands\u201d is given to a\ntabletop robot), we observe hallucinations (e.g., the table turns into a sink or the view turning away from the tabletop robot and a sink shows up). Ideally, we want UniSim to detect actions that are not possible to simulate as opposed to Hallucinate unrealistic outcomes. \u2022 Limited memory. UniSim conditioned on a few frames of the recent history cannot capture longterm memory (e.g., an apple in a drawer could disappear when the drawer is opened if putting the apple in the drawer is not a part of the history for conditioning). How much history to condition on depends on the application of UniSim (e.g., whether UniSim will be used for policy learning in a near-Markov setting or question answering that requires long-term memory). \u2022 Limited out-of-domain generalization. This is especially true for domains that are not represented in the training data. For instance, UniSim is mostly trained on 4 robot morphologies, and its ability to generalize to an unseen robot is limited. Further scaling up training data could help, as UniSim\u2019s training data is nowhere near all the video data available on the internet. \u2022 Visual simulation only. UniSim is not suitable for environments where actions do not cause visual observation change (e.g., different forces in grasping a static cup). A true universal simulator should capture all aspects of the world beyond visual experience (e.g., sound, sensory, etc)."
        },
        {
            "heading": "A ADDITIONAL RESULTS",
            "text": ""
        },
        {
            "heading": "A.1 ADDITIONAL LONG-HORIZON INTERACTION",
            "text": "1. Close bottom drawer\n4. Close top drawer 3. Put bottle in drawer 2. Open top drawer 1. Move sponge close to chips\n3. Knock can over 2. Move can close to chips"
        },
        {
            "heading": "Start Goal",
            "text": ""
        },
        {
            "heading": "Start Goal",
            "text": ""
        },
        {
            "heading": "A.2 ADDITIONAL REAL-ROBOT RESULTS FOR LONG-HORIZON PLANNING",
            "text": ""
        },
        {
            "heading": "A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY IN UNISIM",
            "text": ""
        },
        {
            "heading": "Slide the red circle",
            "text": ""
        },
        {
            "heading": "B DATASETS",
            "text": "We provide the datasets used to train UniSim below, including dataset name, number of training examples (approximate), and weight in the data mixture. Miscellaneous data are collections of datasets that have not been published. Some of these datasets have been processed into train and validation split, hence the number of training examples may differ from the original data size. When text are available in the original dataset, we use T5 language model embeddings (Raffel et al., 2020) to preprocess the text into continuous representations. When low-level controls are available in the original dataset, we encode them both as text and normalize then discretize them into 4096 bins contatenated with language embeddings (if present). The choice of mixture weights are either 0.1 or 0.05 without careful tuning. How data mixture weights affect simulation performance is an interesting line of future work."
        },
        {
            "heading": "C ARCHITECTURE AND TRAINING",
            "text": "We the 3D U-Net architecture (C\u0327ic\u0327ek et al., 2016; Ho et al., 2022b) to parametrize the UniSim video model. We apply the spatial downsampling pass followed by the spatial upsampling pass with skip connections to the downsampling pass activations with interleaved 3D convolution and attention layers as in the standard 3D U-Net. The video models in UniSim consist of one history conditioned video prediction model as the base and two additional spatial super-resolution models similar to Ho et al. (2022a). The history conditioned base model operates at temporal and spatial resolution [16, 24, 40], and the two spatial super-resolution models operate at spatial resolution [24, 40] \u2192 [48, 80] and [48, 80] \u2192 [192, 320], respectively. To condition the base video model on the history, we take 4 frames from the previous video segment and concatenate them channelwise to the noise samples inputted to the U-Net. We employ temporal attention for the forward model to allow maximum modeling flexibility but temporal convolution to the super-resolution models for efficiency reasons similar to Ho et al. (2022a). The model and training hyperparamters of UniSim are summarized in Table 6."
        },
        {
            "heading": "D DETAILS OF EXPERIMENTAL SETUPS",
            "text": ""
        },
        {
            "heading": "D.1 DETAILS OF LONG-HORIZON PLANNING",
            "text": "Language Table Dataset and environment. The Language Table (Lynch & Sermanet, 2020) dataset consists of 160k simulated trajectories and 440k real trajectories where each trajectory contains a language instruction (e.g., \u201cmove blue cube to the right\u201d), a sequence of visuomotor controls, and a sequence of image frames corresponding to the execution of the task. The original trajectories have short horizons (e.g., only moving one block). PALM-E VLM Policy. We modify the original PALM-E 12B model (Driess et al., 2023) to condition on a goal image as additional input before decoding the text actions. The VLM is finetuned on either the original short horizon data or the long horizon simulated data using 64 TPUv3 chips for 1 day. The supervision for short-horizon baseline is the single step language instruction in the original data, whereas the supervision for long-horizon UniSim data is the scripted long-horizon language instructions chained together that generated the video data. Other model architecture and training details follow Driess et al. (2023). Simulated evaluation. In setting up goal in the simulated environments, a subset of 3-4 blocks (randomly selected) are moved by 0.05, 0.1, or 0.2 along the x,y axes (randomly selected). The original observation space has x \u2208 [0.15, 0.6] and y \u2208 [\u22120.3048, 0.3048]. So the modification of goal location corresponds to meaningful block movements. For executing the long-horizon VLM policy trained on UniSim data, we first sample one language instruction from the VLM, predict a video of 16 frames, and use a separately trained inverse dynamics model similar to Du et al. (2023a) to recover the low-level control actions, which we found to slightly outperform directly regressing on control actions from language outputs of the VLM. We execute 5 instructions in total, and measure the final distance to goal according to the ground truth simulator state. We 5 evaluations each with a different random seed for sampling the initial state and resetting the goal, and report the mean and standard error in Table 2."
        },
        {
            "heading": "D.2 DETAILS OF RL POLICY TRAINING",
            "text": "Stage 1 (Supervised Learning) Model Architecture The PaLI 3B model trained on LanguageTable uses a Vision Transformer architecture G/14 (Zhai et al., 2022) to process images, and the encoder-decoder architecture of UL2 language model (Tay et al., 2022) for encoding task descriptions and decoding tokens which can represent language, control actions, or other values of interest (described below). Objectives In the first stage of training, using a dataset of demonstrations, we finetune the pretrained PaLI 3B vision language model checkpoint (Chen et al., 2022b) with the following tasks:\n\u2022 Behavioral Cloning: Given observations and task instruction, predict the demonstration action. The continuous actions of the Language-Table domain are discretized into the form \u201c+1 -5\u201d, and represented using extra tokens from the PaLI model\u2019s token vocabulary. As an example, \u201c+1 -5\u201d is represented by the token sequence (<extra id 65>, <extra id 1>, <extra id 66>, <extra id 5>). \u2022 Timestep to Success Prediction: Given observations and task instruction, predict how many timesteps are left until the end of episode (i.e. success). Similar to actions, the number of steps remaining is represented via extra tokens from the PaLI model\u2019s token vocabulary. \u2022 Instruction Prediction: Given the first and last frame of an episode, predict the task instruction associated with that episode.\nWe use learning rate 0.001, dropout rate 0.1, and batch size 128 to finetune the PaLI 3B model for 300k gradient steps with 1k warmup steps on both the simulated and real Language Table dataset similar to RT-2 Brohan et al. (2023). Stage 2 (RL Training) Reward Definition As mentioned above, during Stage 1, given an observation and goal, the PaLI model is finetuned to predict how many timesteps are left until the demonstration episode reaches a success state. Let us denote this function by d(o, g). The reward we use during RL training is defined as r(ot, at, ot+1, g) = \u2212[d(ot+1, g) \u2212 d(ot, g)] \u00b7 C, where C > 0 is a small constant used to stabilize training (C = 5e \u2212 2 in this work). Intuitively, this reward tracks if from timestep t to t + 1 the policy arrived closer to accomplishing the desired goal. Before starting Stage 2, we make a copy of the Stage 1 model checkpoint and keep it frozen to use as the reward model for RL training. Environment Definition To implement video generation as environment transitions, we expose the inference interface of the video generation model\nthrough remote procedure call, and use the DeepMind RL Environment API (also known as DM Env API) (Tassa et al., 2018) to wrap the remote procedure call in the step function of the environment. When the environment is reset to start a new episode, a goal instruction is randomly sampled from the ones available in the dataset of demonstrations used in Stage 1. RL Method We initialize the RL trained policy using the Stage 1 checkpoint, which as mentioned was also trained with a Behavioral Cloning objective. A collection of actor processes perform policy rollouts in the video generation environment, and add rewards to the trajectories using the reward model defined above. The policy is updated using the REINFORCE (Williams, 1992) objective, i.e. \u2207\u03c0L(ot, at, g) = \u2207\u03c0 log \u03c0(at|ot, g) \u00b7 [\u2211T i=t \u03b3 i\u2212t \u00b7 r(oi, ai, oi+1, g) ] , where L(ot, at, g) repre-\nsents the loss associated with the observation-action pair (ot, at) in an episode with the goal g. The actors are rate limited to prevent generated trajectories from being very off-policy. We report the hyperparameters associated with RL training in Table 7."
        },
        {
            "heading": "D.3 DETAILS OF VIDEO CAPTIONING",
            "text": "Note that even though UniSim is a video based simulator trained to condition on past history, we can achieve text-only conditioning by inputting placeholder frames such as white images while increasing the classifier-free guidance strength on text. We found this to work well in generating videos purely from captions of ActivityNet Captions. For generating data to train VLMs, we take the training split of ActivityNet Captions which consists of 30,740 text-video examples after the 50/25/25% train/val1/val2 split as in Chen et al. (2023). For each of the 30,740 text, we generate 4 videos from UniSim, and use the text labels as supervision in finetuning PaLI-X. As a result, we have 4X amount of the original training data (in terms the number of videos). In addition, we found the generated videos to generally align better semantically than the original ActivityNet Captions videos, which could contain noise and ambiguous videos that could be labeled differently. We use ground truth temporal proposals at evaluation following Chen et al. (2023) and Krishna et al. (2017). Following Chen et al. (2023) and Wang et al. (2021), we use the val1 split for validation and val2 split for testing."
        },
        {
            "heading": "E ADDITIONAL ABLATIONS",
            "text": ""
        },
        {
            "heading": "E.1 ABLATIONS OF DATASETS",
            "text": "We conduct ablations on dataset used in UniSim by computing the FVD and CLIP scores over 1024 samples from the test split. We observe that including internet data and various activity and robot data performs the best. Removing the internet data led to significantly worse FVD, highlighting the importance of using internet data in UniSim."
        },
        {
            "heading": "E.2 ABLATIONS OF MODEL SIZE",
            "text": "We conduct ablations on model size by computing the FVD and CLIP scores over 1024 samples from the test split. We found that while increasing the model size improves the video modeling performance, the amount of improvement measured by FVD plateaus as the model gets bigger, which is slightly disappointing from a scaling point of view."
        },
        {
            "heading": "F FAILED SIMULATIONS WITHOUT JOINT TRAINING",
            "text": ""
        }
    ],
    "year": 2023
}