Point clouds: turning depth into a usable 3D map
Once a system has depth values, whether from stereo matching, LiDAR, structured light, or a monocular model, the next step is usually to convert that depth data into a point cloud: a collection of 3D points in space, each with an x, y, and z coordinate. A dense depth image combined with the camera's known field of view lets every pixel be projected out into 3D space as one point, and stitching together depth readings from multiple angles or moments in time builds up a fuller 3D picture of the surrounding environment.
Point clouds are the working representation robots and self-driving cars actually navigate with, rather than raw depth images. A robot planning a path around obstacles reasons about which regions of 3D space are occupied versus free, which is a much more direct question to answer from a point cloud than from a 2D depth map. A self-driving car builds up and continuously updates a 3D model of everything around it, other vehicles, curbs, pedestrians, road surface, as a point cloud that its planning system can reason about in real, physical units.