SegNav

SegNav: Fine-Grained Semantic Perception for LLM-Based Vision-and-Language Navigation.

SegNav is an LLM-based navigation framework that addresses visual referential ambiguity through fine-grained semantic perception. Built on top of MapGPT, SegNav explicitly aligns linguistic instructions with visual observations using semantic segmentation masks, enabling precise grounding and decision-making in complex indoor environments.

Method Overview

SegNav enhances LLM-based navigation through three core mechanisms:

1. Fine-Grained Visual-Semantic Alignment At each step, the agent receives both a raw RGB observation $O_t$ and a corresponding semantic segmentation mask $S_t$ generated by an open-vocabulary foundation model (ODISE/SEEM). Each candidate viewpoint is represented as an RGB–Mask image pair. A vocabulary-based filtering strategy (datasets/label_mapping_lite.json) retains only indoor-relevant semantic categories, providing the LLM with a constrained legend that forces grounding on navigable landmarks while suppressing environmental noise.

2. Structured Chain-of-Thought with Geometry-Aware Stop Check The agent is required to produce structured JSON output at each step:

• Spatial Analysis: interprets segmentation masks to identify instruction-relevant landmarks.
• Thought: reasons about the next action, penalizing revisits to already-explored nodes.
• Stop Check: enforces a strict termination condition — the agent must confirm physical arrival at the target, not merely visual sighting.
• Action: the chosen movement option.

3. Perception-Driven Self-Correction via Topological Memory The agent maintains a semantically-aligned topological map $M_t$ that records RGB–Mask pairs for all explored and observed nodes. When a persistent mismatch between instruction requirements and accumulated visual evidence is detected (semantic dead-end), the LLM initiates backtracking. Upon revisiting a node, fresh perceptual queries are issued to update the stored visual evidence, enabling robust error recovery.

Setup

Install Matterport3D simulators: follow instructions here. We use the latest version instead of v0.1.

Install requirements:

conda create -n segnav python=3.9
conda activate segnav
pip install -r requirements.txt

Prepare data:

• You can follow DUET and set the annotations for testing the val-unseen set.
• We sample a subset containing 72 scenes and 216 cases for quick and cost-effective testing. Download MapGPT_72_scenes_processed.json and place it in the datasets/R2R/annotations directory.
• RGB observations (datasets/RGB_OBS/): perspective images collected from the Matterport3D simulator. You can use the RGB_Observations.zip we have processed and place it at datasets/RGB_OBS/.
• Segmentation masks (datasets/ODISE_OBS/): pre-generated semantic masks produced by ODISE or SEEM. The mask filenames mirror the RGB structure with .png extension. Place the processed masks at datasets/ODISE_OBS/. If you wish to reproduce our results without installing third-party segmentation models (e.g., ODISE or SEEM), you can directly download our pre-computed segmentation masks for all scenes from https://huggingface.co/datasets/chaoyue1997/SegNav.
• The indoor-relevant semantic legend is already provided at datasets/label_mapping_lite.json.

GPT key: please set your API key in GPT/api.py.

Inference

SegNav supports GPT-4o and GPT-5 as the backbone LLM. Both use --response_format json to enable structured Chain-of-Thought output.

Run with GPT-4o:

bash scripts/gpt4o.sh

Run with GPT-5:

bash scripts/gpt5.sh

You can modify the script arguments to configure your run:

--root_dir ${DATA_ROOT}          # root directory for R2R data
--img_root datasets/RGB_OBS      # path to RGB observation images
--split MapGPT_72_scenes_processed
--end 10                         # number of cases to test
--output_dir ${outdir}
--max_action_len 15
--save_pred
--stop_after 3                   # prevent stopping within the first 3 steps
--llm gpt-4o-2024-05-13          # or gpt-5
--response_format json
--max_tokens 4000

Results

Results on 72 various scenes of the R2R dataset. "Exp" refers to the number of GPT experts.

Methods	LLMs	Exp	NE ↓	OSR ↑	SR ↑	SPL ↑
NavGPT	GPT-3.5	3	8.02	26.4	16.7	13.0
DiscussNav	GPT-4	5	6.30	51.0	37.5	33.3
MapGPT	GPT-4	1	5.80	61.6	41.2	25.4
MapGPT	GPT-4V	1	5.62	57.9	47.7	38.1
MapGPT	GPT-4o	1	5.11	56.9	46.3	37.8
SegNav (Ours)	GPT-4o	1	3.70	76.6	53.3	43.2