Robust depth estimation for robots with stereo cameras using bespoke DNNs


Santa Clara Valley Robotics and Automation Society Chapter

Autonomous robots depend on their perception systems to understand the world around them. These machines often leverage a host of sensors including cameras, lidars, radars, and ultrasonic sensors to create this environmental understanding. Stereo cameras play a big role in providing depth perception to robotic systems. This depth information can be estimated using classical computer vision techniques, like semi-global matching (SGM) or leverage deep neural networks (DNNs). Each individual algorithm may struggle in a particular set of operating conditions. But when multiple depth estimation algorithms are leveraged simultaneously, It is possible that more robust depth information can be calculated.

In this talk, we'll cover work at NVIDIA to train the ESS DNN model for determining stereo disparity using both synthetic and real-world data to perform well where SGM may not. We'll also introduce the Bi3D model which is trained on the simplified question of "is X closer than M meters?" rather than "how far away is X?", yielding improvements in both accuracy and speed. As every approach has deficiencies on its own, we'll touch upon how ensembling the responses of ESS and Bi3D, DNNs developed specifically for robotic perception with SGM could lead to robust obstacle detection. Finally, we'll discuss how we've tuned the performance of these models to run on embedded compute for the responsive stopping behavior required in autonomous mobile robots (AMRs).

  Date and Time




  • Date: 07 Dec 2022
  • Time: 06:30 PM to 08:00 PM
  • All times are (UTC-08:00) Pacific Time (US & Canada)
  • Add_To_Calendar_icon Add Event to Calendar

Tom Coughlin is inviting you to a scheduled Zoom meeting.

Join Zoom Meeting

Meeting ID: 879 6968 7824
Passcode: 528158
One tap mobile
+16694449171,,87969687824#,,,,*528158# US
+16699006833,,87969687824#,,,,*528158# US (San Jose)

Dial by your location
        +1 669 444 9171 US
        +1 669 900 6833 US (San Jose)
        +1 253 205 0468 US
        +1 253 215 8782 US (Tacoma)
        +1 346 248 7799 US (Houston)
        +1 719 359 4580 US
        +1 360 209 5623 US
        +1 386 347 5053 US
        +1 507 473 4847 US
        +1 564 217 2000 US
        +1 646 931 3860 US
        +1 689 278 1000 US
        +1 929 205 6099 US (New York)
        +1 301 715 8592 US (Washington DC)
        +1 305 224 1968 US
        +1 309 205 3325 US
        +1 312 626 6799 US (Chicago)
Meeting ID: 879 6968 7824
Passcode: 528158
Find your local number:

  • Starts 17 November 2022 09:16 AM
  • Ends 05 December 2022 10:00 PM
  • All times are (UTC-08:00) Pacific Time (US & Canada)
  • No Admission Charge


Hemal Shah of NVIDIA


Hemal Shah, Director of Robotics Solutions Engineering at NVIDIA:  Hemal leads Isaac ROS at NVIDIA, delivering high-performance computing to the ROS2 ecosystem to power intelligent robotics. Prior to joining NVIDIA, Hemal has led engineering teams at Google, X, and Boston Dynamics, working  on logistics automation, fleet management, perception, and machine learning at scale. He holds a BS in computer science from Georgia Institute of Technology and a MS in computer science from Stanford University focusing on ML and robotics.

Gerard Andrews of NVIDIA


Gerard Andrews, Sr Product Manager - Robotics at NVIDIA:  Gerard Andrews is focused on revolutionizing the way intelligent robots are developed, trained, tested and deployed using the NVIDIA Isaac robotics platform. Prior to joining NVIDIA, Gerard was at Cadence where he was Product Marketing Director, responsible for product planning, marketing, and business development for licensable processor IP. He holds a BS in electrical engineering from Southern Methodist University and an MS in electrical engineering from Georgia Institute of Technology. 


6:30 PM Introduction (Tom Coughlin)

6:45 PM Talk

7:30 PM Q&A

8:00 PM End