Abstract

Neural implicit fields have been a de facto standard in novel view synthesis. Recently, there exist some methods exploring fusing multiple modalities within a single field, aiming to share implicit features from different modalities to enhance reconstruction performance. However, these modalities often exhibit misaligned behaviors: optimizing for one modality, such as LiDAR, can adversely affect another, like camera performance, and vice versa. In this work, we conduct comprehensive analyses on the multimodal implicit field of LiDAR-camera joint synthesis, revealing the underlying issue lies in the misalignment of different sensors. Furthermore, we introduce AlignMiF, a geometrically aligned multimodal implicit field with two proposed modules: Geometry-Aware Alignment (GAA) and Shared Geometry Initialization (SGI). These modules effectively align the coarse geometry across different modalities, significantly enhancing the fusion process between LiDAR and camera data. Through extensive experiments across various datasets and scenes, we demonstrate the effectiveness of our approach in facilitating better interaction between LiDAR and camera modalities within a unified neural field. Specifically, our proposed AlignMiF, achieves remarkable improvement over recent implicit fusion methods (+2.01 and +3.11 image PSNR on the KITTI-360 and Waymo datasets) and consistently surpasses single modality performance (13.8% and 14.2% reduction in LiDAR Chamfer Distance on the respective datasets).

Framework

Experiment

Conclusion

In this paper, we thoroughly investigated and validated the misalignment issue in multimodal NeRF through various analyses, such as the examination and visualization of raw sensor inputs, hash features, and density values, as well as experiments on various network architectures. Futhermore, we propose AlignMiF, with two simple yet effective mod- ules, Geometry-Aware Alignment (GAA) and Shared Geometry Initialization (SGI), to address the misalignment issue by aligning the consistent coarse geometry of different modalities while preserving their unique details. We conduct extensive experiments on multiple datasets and scenes and demonstrate the effectiveness of our proposed method in improving multimodal fusion and alignment within a unified NeRF framework. We hope that our work can inspire future research in the field of multimodal NeRF.

Acknowledgement

This work was supported in part by the National Key R&D Program of China under Grant No. 2020AAA0109700, Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061), Mobility Grant Award under Grant No. M-0461, Shenzhen Science and Technology Program (Grant No. RCYX20200714114642083), Shenzhen Science and Technology Program (Grant No. GJHZ20220913142600001), Nansha Key RD Program under Grant No.2022ZD014.