Building Semantic 3D Maps for Language-guided Robot Navigation
Bertolo Stahl, Gustavo Henrique
Bertolo Stahl, Gustavo Henrique
Citations
Altmetric:
Supervisor
Department
Computer Vision
Embargo End Date
Type
Thesis
Date
2026
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Robotic navigation in real-world environments requires not only accurate geometric representations of space but also the ability to interpret high-level semantic concepts expressed through natural language. Classical approaches such as Simultaneous Localization and Mapping (SLAM) provide precise spatial reconstructions but lack flexibility for open-vocabulary text-based localization, while vision–language models offer strong semantic understanding but are typically limited to image-level representations. This thesis addresses the problem of enabling robots to navigate toward locations specified through natural language, including targets outside the robot’s immediate field of view. To this end, a pipeline is proposed that integrates three-dimensional reconstruction and vision–language encoding. The method is divided into an offline mapping stage, where a dense 3D reconstruction is built and enriched with semantic embeddings, and an online navigation stage, where the robot is localized through visual matching and a language query is used to retrieve the target region. These components are combined to compute a navigation path in the environment. The approach is evaluated on a custom dataset of annotated 3D scenes, demonstrating its ability to perform zero-shot 3D localization and associate textual queries with spatial regions. Overall, this work demonstrates strong potential and develops a building block for semantic robot navigation.
Citation
Bertolo Stahl, Gustavo Henrique, "Building Semantic 3D Maps for Language-guided Robot Navigation," M.S. Thesis, Computer Vision, MBZUAI, 2026.
Source
Conference
Keywords
Vision-Language Models, Vision-Language Maps, 3D Reconstruction, Navigation, Language-driven Navigation