Many of our research projects are in one of the following general themes. Note that this page is still being updated to include all publications.

Deformable Object Manipulation

Deformable objects are challenging from both a perceptual and dynamic perspective: a crumpled cloth has many self-occlusions and its configuration is hard to infer from observations; further, the dynamics of a cloth are complex to model and incorporate into planning algorithms. We develop algorithms to handle deformable object manipulation tasks, such as cloth, liquids, dough, and articulated objects.

Relevant Publications
One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments
Yufei Wang, Zhanyi Sun, Zackory Erickson*, David Held*
Robotics: Science and Systems (RSS), 2023
Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking
Zixuan Huang, Xingyu Lin, David Held
International Conference on Robotics and Automation (ICRA), 2023
ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
Daniel Seita, Yufei Wang†, Sarthak J Shetty†, Edward Yao Li†, Zackory Erickson, David Held
Conference on Robot Learning (CoRL), 2022
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin*, Carl Qi*, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, David Held
Conference on Robot Learning (CoRL), 2022
Learning to Singulate Layers of Cloth based on Tactile Feedback
Sashank Tirumala*, Thomas Weng*, Daniel Seita*, Oliver Kroemer, Zeynep Temel, David Held
International Conference on Intelligent Robots and Systems (IROS), 2022 - Best Paper at ROMADO-SI
Learning Closed-loop Dough Manipulation using a Differentiable Reset Module
Carl Qi, Xingyu Lin, David Held
Robotics and Automation Letters (RAL) with presentation at the International Conference on Intelligent Robots and Systems (IROS), 2022
Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions
Yufei Wang, David Held, Zackory Erickson
Robotics and Automation Letters (RAL) with presentation at the International Conference on Intelligent Robots and Systems (IROS), 2022
Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation
Zixuan Huang, Xingyu Lin, David Held
Robotics: Science and Systems (RSS), 2022
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B. Tenenbaum, David Held, Chuang Gan
International Conference on Learning Representations (ICLR), 2022
Self-supervised Transparent Liquid Segmentation for Robotic Pouring
Gautham Narayan Narasimhan, Kai Zhang, Ben Eisner, Xingyu Lin, David Held
International Conference of Robotics and Automation (ICRA), 2022
Learning Visible Connectivity Dynamics for Cloth Smoothing
Xingyu Lin*, Yufei Wang*, Zixuan Huang, David Held
Conference on Robot Learning (CoRL), 2021
FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy
Thomas Weng, Sujay Bajracharya, Yufei Wang, David Held
Conference on Robot Learning (CoRL), 2021
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
Xingyu Lin, Yufei Wang, Jake Olkin, David Held
Conference on Robot Learning (CoRL), 2020
PLAS: Latent Action Space for Offline Reinforcement Learning
Wenxuan Zhou, Sujay Bajracharya, David Held
Conference on Robot Learning (CoRL), 2020 - Plenary talk (Selection rate 4.1%)
Cloth Region Segmentation for Robust Grasp Selection
Jianing Qian*, Thomas Weng*, Luxin Zhang, Brian Okorn, David Held
International Conference on Intelligent Robots and Systems (IROS), 2020

3D Affordance Reasoning for Object Manipulation

In order for a robot to interact with an object, the robot must infer its “affordances”: how the object moves as the robot interacts with it and how the object can interact with other objects in the environment. We develop robot perception algorithms that learn to estimate these affordances and then use such inferences to learn to manipulate objects to achieve a task.

Relevant Publications
Learning Distributional Demonstration Spaces for Task-Specific Cross-Pose Estimation
Jenny Wang*, Octavian Donca*, David Held
International Conference on Robotics and Automation (ICRA), 2024
One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments
Yufei Wang, Zhanyi Sun, Zackory Erickson*, David Held*
Robotics: Science and Systems (RSS), 2023
Neural Grasp Distance Fields for Robot Manipulation
Thomas Weng, David Held, Franziska Meier, Mustafa Mukadam
International Conference on Robotics and Automation (ICRA), 2023
TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation
Chuer Pan*, Brian Okorn*, Harry Zhang*, Ben Eisner*, David Held
Conference on Robot Learning (CoRL), 2022
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin*, Carl Qi*, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, David Held
Conference on Robot Learning (CoRL), 2022
Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions
Yufei Wang, David Held, Zackory Erickson
Robotics and Automation Letters (RAL) with presentation at the International Conference on Intelligent Robots and Systems (IROS), 2022
FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects
Ben Eisner*, Harry Zhang*, David Held
Robotics: Science and Systems (RSS), 2022 - Best Paper Finalist (Selection Rate 1.5%)
Learning Visible Connectivity Dynamics for Cloth Smoothing
Xingyu Lin*, Yufei Wang*, Zixuan Huang, David Held
Conference on Robot Learning (CoRL), 2021

Multimodal Learning

Robots should use all of the sensors available to them, such as depth, RGB, and tactile data. We have developed methods to intelligently integrate these sensor modalities.

Relevant Publications
Learning to Singulate Layers of Cloth based on Tactile Feedback
Sashank Tirumala*, Thomas Weng*, Daniel Seita*, Oliver Kroemer, Zeynep Temel, David Held
International Conference on Intelligent Robots and Systems (IROS), 2022 - Best Paper at ROMADO-SI
PLAS: Latent Action Space for Offline Reinforcement Learning
Wenxuan Zhou, Sujay Bajracharya, David Held
Conference on Robot Learning (CoRL), 2020 - Plenary talk (Selection rate 4.1%)
Cloth Region Segmentation for Robust Grasp Selection
Jianing Qian*, Thomas Weng*, Luxin Zhang, Brian Okorn, David Held
International Conference on Intelligent Robots and Systems (IROS), 2020
Multi-Modal Transfer Learning for Grasping Transparent and Specular Objects
Thomas Weng, Amith Pallankize, Yimin Tang, Oliver Kroemer, David Held
Robotics and Automation Letters (RAL) with presentation at the International Conference of Robotics and Automation (ICRA), 2020

Reinforcement Learning Algorithms

Robots can use data, either from the real world or from a simulator, to learn how to perform a task. This is especially important for tasks which are difficult for robots to achieve via traditional techniques such as motion planning, such as deformable object manipulation. We have developed novel reinforcement learning algorithms to more effectively learn from data.

Relevant Publications
Learning Off-policy for Online Planning
Harshit Sikchi, Wenxuan Zhou, David Held
Conference on Robot Learning (CoRL), 2021 - Oral presentation (Selection rate 6.5%); Best Paper Finalist
PLAS: Latent Action Space for Offline Reinforcement Learning
Wenxuan Zhou, Sujay Bajracharya, David Held
Conference on Robot Learning (CoRL), 2020 - Plenary talk (Selection rate 4.1%)
Adaptive Auxiliary Task Weighting for Reinforcement Learning
Xingyu Lin*, Harjatin Baweja*, George Kantor, David Held
Neural Information Processing Systems (NeurIPS), 2019
Automatic Goal Generation for Reinforcement Learning Agents
Carlos Florensa*, David Held*, Xinyang Geng*, Pieter Abbeel
International Conference on Machine Learning (ICML), 2018
Reverse Curriculum Generation for Reinforcement Learning
Carlos Florensa, David Held, Markus Wulfmeier, Pieter Abbeel
Conference on Robot Learning (CoRL), 2017
Constrained Policy Optimization
Joshua Achiam, David Held, Aviv Tamar, Pieter Abbeel
International Conference on Machine Learning (ICML), 2017

Autonomous Driving

In the domain of autonomous driving, we have developed novel methods for every part of the perception pipeline: segmentation, object detection, tracking, and velocity estimation.

Relevant Publications
Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting
Tarasha Khurana, Peiyun Hu, David Held, Deva Ramanan
Conference on Computer Vision and Pattern Recognition (CVPR), 2023
Differentiable Raycasting for Self-supervised Occupancy Forecasting
Tarasha Khurana*, Peiyun Hu*, Achal Dave, Jason Ziglar, David Held, Deva Ramanan
European Conference on Computer Vision (ECCV), 2022
Semi-supervised 3D Object Detection via Temporal Graph Neural Networks
Jianren Wang, Haiming Gang, Siddharth Ancha, Yi-ting Chen, and David Held
International Conference on 3D Vision (3DV), 2021
Active Safety Envelopes using Light Curtains with Probabilistic Guarantees
Siddharth Ancha, Gaurav Pathak, Srinivasa Narasimhan, David Held
Robotics: Science and Systems (RSS), 2021
Safe Local Motion Planning with Self-Supervised Freespace Forecasting
Peiyun Hu, Aaron Huang, John Dolan, David Held, Deva Ramanan
Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Active Perception using Light Curtains for Autonomous Driving
Siddharth Ancha, Yaadhav Raaj, Peiyun Hu, Srinivasa Narasimhan, David Held
European Conference on Computer Vision (ECCV), 2020 - Spotlight presentation (Selection rate 5.3%)
Uncertainty-aware Self-supervised 3D Data Association
Jianren Wang, Siddharth Ancha, Yi-Ting Chen, David Held
International Conference on Intelligent Robots and Systems (IROS), 2020
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics
Xinshuo Weng, Jianren Wang, David Held, Kris Kitani
International Conference on Intelligent Robots and Systems (IROS), 2020
Just Go with the Flow: Self-Supervised Scene Flow Estimation
Himangi Mittal, Brian Okorn, David Held
Conference on Computer Vision and Pattern Recognition (CVPR), 2020 - Oral presentation (Selection rate 5.7%)
What You See is What You Get: Exploiting Visibility for 3D Object Detection
Peiyun Hu, Jason Ziglar, David Held, Deva Ramanan
Conference on Computer Vision and Pattern Recognition (CVPR), 2020 - Oral presentation (Selection rate 5.7%)
Learning to Optimally Segment Point Clouds
Peiyun Hu, David Held*, Deva Ramanan*
Robotics and Automation Letters (RAL) with presentation at the International Conference of Robotics and Automation (ICRA), 2020
PCN: Point Completion Network - Best Paper Honorable Mention
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, Martial Hebert
International Conference on 3D Vision (3DV), 2018
A Probabilistic Framework for Real-time 3D Segmentation using Spatial, Temporal, and Semantic Cues
David Held, Devin Guillory, Brice Rebsamen, Sebastian Thrun, Silvio Savarese
Robotics: Science and Systems (RSS), 2016

Active Perception

Rather than statically observing a scene, robots can take actions to enable them to better perceive a scene, known as “active perception.”

Relevant Publications
Active Velocity Estimation using Light Curtains via Self-Supervised Multi-Armed Bandits
Siddharth Ancha, Gaurav Pathak, Ji Zhang, Srinivasa Narasimhan, David Held
Robotics: Science and Systems (RSS), 2023
Active Safety Envelopes using Light Curtains with Probabilistic Guarantees
Siddharth Ancha, Gaurav Pathak, Srinivasa Narasimhan, David Held
Robotics: Science and Systems (RSS), 2021
Exploiting & Refining Depth Distributions with Triangulation Light Curtains
Yaadhav Raaj, Siddharth Ancha, Robert Tamburo, David Held, Srinivasa Narasimhan
Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Active Perception using Light Curtains for Autonomous Driving
Siddharth Ancha, Yaadhav Raaj, Peiyun Hu, Srinivasa Narasimhan, David Held
European Conference on Computer Vision (ECCV), 2020 - Spotlight presentation (Selection rate 5.3%)

Self-Supervised Learning for Robotics

Rather than relying on hand-annotated data, self-supervised learning can enable robots to learn from large unlabeled datasets.

Relevant Publications
Active Velocity Estimation using Light Curtains via Self-Supervised Multi-Armed Bandits
Siddharth Ancha, Gaurav Pathak, Ji Zhang, Srinivasa Narasimhan, David Held
Robotics: Science and Systems (RSS), 2023
Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting
Tarasha Khurana, Peiyun Hu, David Held, Deva Ramanan
Conference on Computer Vision and Pattern Recognition (CVPR), 2023
Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking
Zixuan Huang, Xingyu Lin, David Held
International Conference on Robotics and Automation (ICRA), 2023
Differentiable Raycasting for Self-supervised Occupancy Forecasting
Tarasha Khurana*, Peiyun Hu*, Achal Dave, Jason Ziglar, David Held, Deva Ramanan
European Conference on Computer Vision (ECCV), 2022
Self-supervised Transparent Liquid Segmentation for Robotic Pouring
Gautham Narayan Narasimhan, Kai Zhang, Ben Eisner, Xingyu Lin, David Held
International Conference of Robotics and Automation (ICRA), 2022
OSSID: Online Self-Supervised Instance Detection by (and for) Pose Estimation
Qiao Gu, Brian Okorn, David Held
Robotics and Automation Letters (RAL) with presentation at the International Conference of Robotics and Automation (ICRA), 2022
Self-Supervised Point Cloud Completion via Inpainting
Himangi Mittal, Brian Okorn, Arpit Jangid, David Held
British Machine Vision Conference (BMVC), 2021 - Oral presentation (Selection rate 3.3%)
Safe Local Motion Planning with Self-Supervised Freespace Forecasting
Peiyun Hu, Aaron Huang, John Dolan, David Held, Deva Ramanan
Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Visual Self-Supervised Reinforcement Learning with Object Reasoning
Yufei Wang*, Gautham Narayan Narasimhan*, Xingyu Lin, Brian Okorn, David Held
Conference on Robot Learning (CoRL), 2020
Uncertainty-aware Self-supervised 3D Data Association
Jianren Wang, Siddharth Ancha, Yi-Ting Chen, David Held
International Conference on Intelligent Robots and Systems (IROS), 2020
Just Go with the Flow: Self-Supervised Scene Flow Estimation
Himangi Mittal, Brian Okorn, David Held
Conference on Computer Vision and Pattern Recognition (CVPR), 2020 - Oral presentation (Selection rate 5.7%)

Previous Directions

Object tracking

Tracking involves consistently locating an object as it moves across a scene, or consistently locating a point on an object as it moves. In order to understand how robots should interact with objects, the robot must be able to track them as they change in position, viewpoint, lighting, occlusions, and other factors. Improvements in this area should enable autonomous vehicles to interact more safely around dynamic objects (e.g. pedestrians, bicyclists, and other vehicles).

Relevant Publications
Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking
Zixuan Huang, Xingyu Lin, David Held
International Conference on Robotics and Automation (ICRA), 2023
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics
Xinshuo Weng, Jianren Wang, David Held, Kris Kitani
International Conference on Intelligent Robots and Systems (IROS), 2020
Just Go with the Flow: Self-Supervised Scene Flow Estimation
Himangi Mittal, Brian Okorn, David Held
Conference on Computer Vision and Pattern Recognition (CVPR), 2020 - Oral presentation (Selection rate 5.7%)
A Probabilistic Framework for Real-time 3D Segmentation using Spatial, Temporal, and Semantic Cues
David Held, Devin Guillory, Brice Rebsamen, Sebastian Thrun, Silvio Savarese
Robotics: Science and Systems (RSS), 2016