Creating a 3D World from 2D Images

#computer-vision #image-processing #digital #PDH #CEU #LICN #software #vision #maps #deep-learning #image-registration #3d

Have you ever tried Google’s Live View feature, where you just point your phone at a building and it instantly figures out exactly where you are? Or maybe you’ve wondered how action movies pull off those cool 3D scenes of the places they’re about to rob? Ever stopped to think about how virtual tours and street views come to life?

In this talk, we will explore the various algorithms that are used to create vivid and comprehensive 3D scenes from just a handful of images collected from the internet. The talk will be divided in 3 key sections:

Artifact Mitigation
Images collected from the internet are not always perfect. Some may have memes or text overlaid on them, while others may have been compressed multiple times for efficient storage, resulting in pixelation. Some images might be blurry, have too much sun exposure, or be taken at night. There can also be moving objects like people and cars, which aren’t needed and could obstruct the 3D reconstruction of a scene.

In this section, we’ll learn about the deep learning algorithms used to remove these kinds of artifacts and transient objects from images.

Image Registration and Geo-localization
Once the images have been pre-processed, the next step is to determine their relative pose with respect to each other. Imagine trying to figure out whether a photo was taken from the left side of the Eiffel Tower or the right, or whether the person with the camera was 100 meters away or just 50 meters. Sometimes, the images might even come from a drone! So how do we place all these different frames into a common reference frame? The more viewpoints we have of an area, the more complete our 3D models will be.

In this section, we’ll learn how Structure-from-Motion (SfM) is used to assign poses to these images. We’ll explore techniques for using background details to determine pose, especially when the object of interest looks the same from every angle. And finally, we’ll briefly discuss how these images can be geo-localized; meaning their latitude and longitude can be estimated, even when no GPS information is available.

3D Reconstruction
Now that our images are cleaned up and we know their poses, we can dive into the techniques used to transform these images to build a 3D scene. We’ll discuss, at a high level, some traditional 3D reconstruction methods along with more recent AI-based approaches such as neural rendering and gaussian splatting.

The applications of these computer vision and deep learning techniques we’ve talked about are widespread, and I hope you’ll start noticing them being used all around you, whether it’s in mobile phones, self-driving technology, biometric scanners, delivery robots, security cameras, and many other places.

7:00 PM   Networking and Announcements

7:10 PM   Presentation