Portable recording system mounted on a backpack

Left: A base with four cameras. Right: The same base on the roof of our van in Chapel Hill.

For data collection at ground-level, we have constructed two recording systems. The image on the top left shows a low-cost, man-portable camera capture system, designed for flexibility and mobility. The setup consists of a Point Grey Research Ladybug2 omnidirectional camera, a Garmin consumer-grade GPS receiver with Wide Area Augmentation System (WAAS) capability, and a Microstrain 3DM-G inertial sensor. The Ladybug is a multi-camera system consisting of six cameras, of which five form a ring and the sixth camera points upwards. Together, these provide video coverage of most of the upper hemisphere about the camera unit, except for the area directly below the camera. The GPS unit is accurate to approximately five meters under optimal conditions meaning many visible satellites, no or low multipath error etc. In practice this is highly unusual in urban environments and errors on the scale of 10 or more meters are more typical due to the urban canyon effect where very little of the sky is visible to the GPS receiver because of surrounding buildings. Finally the 3DM-G provides an absolute orientation measurement accurate to +/-5 degrees. The image on the top right shows the system we use for large-scale data collection, constructed using multiple synchronized cameras on a base and mounted on a car. Three of the four cameras are horizontal, pointing forward, backward and to the side, while the fourth one is tilted upward to capture the upper parts of building facades. While additional sensors such as GPS and inertia sensors can be integrated to give accurate position measurements, our system can function in the absence of this additional information as well.

Below is a composite video captured by the four camera recording system in Chapel Hill

Composite video captured with the four camera system

 

We have also obtained 3D reconstructions from aerial video with a helicopter, using a nose-mounted, gyro-stabilized HDTV camera system. Using both ground and aerial video allows us to obtain increased coverage of the facades and rooftops in urban scenes. By aligning the ground and aerial reconstructions, we are thus able produce more complete 3D models. Below is a photo of the aerial recording system, along with a sample video.

 

---

 

Processing

The input to our processing pipeline consists of the video sequence, which may be optionally augmented by the trajectory of the vehicle, in the form of GPS/INS measurements . Since the cameras do not overlap, reconstruction is performed on the frames of a single camera as it moves through the scene. The main steps of the processing pipeline are the following:

• 2D feature tracking: Our highly optimized GPU implementation of the KLT tracker is used to track features in the image, and is able to achieve processing rates of more than 200 frames per second on on state-of-the-art consumer GPUs for PAL (720 × 576) resolution data. The features are used in the structure-from-motion computation and in the sparse scene analysis step. Our implementation is also capable of estimating a global gain ratio between successive frames in order to compensate for changes in the camera exposure. The sources for the KLT tracker can be downloaded from here.
• Pose estimation and refinement: Using the tracked 2D features, we compute camera poses using various structure-from motion techniques. The computation is done in real-time using robust estimation techniques. In the event where additional trajectory information is available, this may be fused with visual measurements via an Extended Kalman Filter to refine the pose of each camera at each frame.
• Sparse scene analysis: The tracked features can be reconstructed in 3D, given the camera poses, to provide valuable information about the scene surfaces and their orientation.
• Multi-way plane-sweeping stereo: We use the plane-sweeping algorithm for stereo reconstruction. Planes are swept in multiple directions to account for slanted surfaces and a prior probability estimated from the sparse data is used to disambiguate textureless surfaces. Our stereo algorithm is also run on the GPU to take advantage of its efficiency in rendering operations, which are the most costly computations in plane-sweeping stereo.
• Depth map fusion: Stereo depth maps are computed for each frame in the previous step in real time. There are large overlaps between adjacent depth maps that should be removed to produce a more economical representation of the scene. The depth map fusion stage combines multiple depth estimates for each pixel of a reference view, enforces visibility constraints and thus improves the accuracy of the reconstruction. The result is one fused depth map that replaces several raw depth maps that cover the same part of the scene.
• Model generation: A multi-resolution mesh is generated for each fused depth map and video frames are used for texture-mapping. The camera gain is adjusted within and across video streams so that transitions in appearance are smoother. In addition, partial reconstructions are merged and holes in the model are filled in, if possible.
• Model matching: We have also developed a technique that may be used to efficiently perform 3D scene alignment. By leveraging local shape information, an invariant feature descriptor is extracted which is then used in a hierarchical matching scheme to perform efficient matching and alignment of 3D scenes. This allows us to align multiple 3D models; such as those obtained from ground and aerial video, and also to perform loop completion.

   

---

 

Reconstructions