Add offline object inventory and localize perception pipeline DIM1343

[bogwi]

What features we implement in this PR

From the issue feature list, this branch covers A, D, E, and F.

• (A) Look at an image and identify all objects: inventory() discovers every in-scope support on the table without a text prompt, then names each instance.
• (D) Text prompt to 2D masks: localize() runs SigLIP retrieval, OmDet boxes, and EdgeTAM segment() so a query string becomes masks on ranked frames.
• (E) Prompt to point cloud / latest 3D pose: localize() lifts those masks through depth, clusters them, verifies across views, and returns the newest verified cloud and position. Offline over a recording, no live robot.
• (F) Object deduplication: inventory() merges same-frame fragments and associates tracks with geometry only (Hungarian assignment plus fragment merge). Labels are applied after identity is fixed.

Not in this branch: (B) exact dimensions, (C) 360-degree clouds, (G) re-ID.

What is the proposed API

Localize

use localize() to query an object or a list of objects over a recorded window

e2e example

from dimos.memory2.store.sqlite import SqliteStore
from dimos.models.embedding.siglip import SigLIPModel
from dimos.models.segmentation.edge_tam import EdgeTAMImageSegmenter
from dimos.perception.detection.detectors.omdet import OmDetDetector
from dimos.perception.memory.localize import embed_index, localize
from dimos.utils.data import get_data

store = SqliteStore(
    path=get_data(
        "xarm6_worldbelief_realsense_d435i_stationery_calibrated/"
        "xarm6_worldbelief_20260729_203624_161992.db"
    )
)
lo, _ = store.streams.color_image.get_time_range()

# The models and the frame index belong to you: load once, query many times.
siglip = SigLIPModel()
detector = OmDetDetector()
segmenter = EdgeTAMImageSegmenter()
index = embed_index(store, siglip, lo + 53, lo + 129)

# Single query: one Localization or None.
hit = localize(store, "book", index=index, siglip=siglip, detector=detector, segmenter=segmenter)
if hit is not None:
    print(f"book: {hit.position_world_xyz} score={hit.semantic_score:.2f}")

# List query: one shared detection pass, one result per label, in order.
queries = ["pen", "red marker"]
hits = localize(store, queries, index=index, siglip=siglip, detector=detector, segmenter=segmenter)
for query, hit in zip(queries, hits):
    if hit is None:
        print(f"{query}: not found")
    else:
        print(f"{query}: {hit.position_world_xyz} score={hit.semantic_score:.2f}")

siglip.stop()
detector.stop()
del segmenter

Its output:

book: (0.6025082879068396, -0.170543153625002, 0.12976971814597688) score=0.70
pen: (0.6144240271930878, 0.03811344227187729, 0.10974094282609587) score=0.76
red marker: (0.7413823721338351, 0.22396659566268498, 0.18055518829473222) score=0.76

INITIAL PR1 LIMITATIONS

1. The cloud is the latest single-view depth lift - cloud_mode="latest_visible" - and not a fused 360 cloud. This is a feature (C), not implemented today. Grasp candidates on the visible face work - grasps that need the back side - do not. Same for (B) exact dimensions => extent is the latest-view AABB.

2. Manipulation team should plan from the points not the quaternion.

If the manipulation team feeds the points into their grasp planner and lets it work out grasp poses from the geometry, they are fine. If they read orientation_world_xyzw and treat it as the object's pose, they will get grasps that are rotated wrong in a way that depends on where the camera happened to be.

3. Fully offline-by-design (E): replay recordings, no live robot. It is meant for grasp-proposal experiments against replays like running graspgenx on the returned clouds.

localize CLI example with a dedicated tool - `tool_localize()

uv run python -m dimos.perception.memory.tool_localize \
  "book" \
  --from 53 \
  --duration 76

hit 'book': position=(0.603, -0.171, 0.130) frame=world ts_offset=128.8s points=24715 views=3 score=0.70 margin=1.00

localize performance

cpu: AMD Ryzen AI 7 350 w/ Radeon 860M (8c/16t)
ram: 30.1 GiB
gpu: NVIDIA GeForce RTX 5070 Laptop GPU, 7.5 GiB, driver 595.84, cuda 12.8, torch 2.7.1+cu128
recording: 605s total, windows end at +129s

cold, 76s window, query 'book':
index build 4.20s (includes SigLIP weight load)
first query 15.42s (includes OmDet + EdgeTAM weight load)
total 19.62s
warm query, same window: 9.67s 9.79s 9.93s (mean 9.80s)

ram: 3.0 GiB resident, 3.1 GiB peak
vram: 1.35 GiB weights resident, 1.68 GiB peak, 1.96 GiB reserved, 2.1 GiB process total (nvidia-smi)

window scaling, warm weights, index rebuilt per window:
last 30s: index 0.39s (30 frames, 13 ms/frame), query 6.04s
last 60s: index 0.73s (60 frames, 12 ms/frame), query 8.45s
last 120s: index 1.75s (119 frames, 15 ms/frame), query 14.98s

Inventory

inventory() takes a vocabulary of labels as an argument to label all that is detected. With empty vocabulary you still got N instances as unknown-*.

inventory CLI example with a dedicated tool - tool_inventory

uv run python -m dimos.perception.memory.tool_inventory \
  --from 253 \
  --duration 76 \
  --labels \
    "marker|red marker" \
    "sticky notes|post-it notes" \
    "pen|black pen" \
    "book" \
    "roll of tape|black tape"

result

0  id=obj-00  name=book  xyz=(0.604,-0.169,0.129)  ts_offset=328.4  members=23  extent=(0.262,0.239,0.047)  sigma=(0.002,0.004,0.001)  coverage=0.12
1  id=obj-01  name=unknown-0  xyz=(0.459,-0.388,0.130)  ts_offset=285.9  members=4  extent=(0.136,0.138,0.076)  sigma=(0.001,0.000,0.000)  coverage=0.12
2  id=obj-02  name=unknown-1  xyz=(0.667,-0.682,0.158)  ts_offset=285.9  members=6  extent=(0.168,0.023,0.106)  sigma=(0.002,0.003,0.008)  coverage=0.12
3  id=obj-03  name=sticky notes  xyz=(0.795,0.038,0.111)  ts_offset=328.4  members=22  extent=(0.068,0.095,0.010)  sigma=(0.004,0.010,0.002)  coverage=0.12
4  id=obj-04  name=marker  xyz=(0.776,-0.081,0.179)  ts_offset=328.4  members=24  extent=(0.009,0.015,0.110)  sigma=(0.001,0.001,0.006)  coverage=0.25
5  id=obj-05  name=unknown-2  xyz=(0.516,-0.451,0.121)  ts_offset=285.9  members=4  extent=(0.038,0.036,0.014)  sigma=(0.000,0.000,0.000)  coverage=0.12
6  id=obj-06  name=unknown-3  xyz=(1.032,-0.987,0.114)  ts_offset=284.1  members=2  extent=(0.168,0.198,0.021)  sigma=(0.003,0.008,0.000)  coverage=0.12
7  id=obj-07  name=pen  xyz=(0.614,0.039,0.110)  ts_offset=328.4  members=24  extent=(0.028,0.121,0.014)  sigma=(0.002,0.010,0.001)  coverage=0.25
8  id=obj-08  name=unknown-4  xyz=(0.586,-0.404,0.112)  ts_offset=285.9  members=4  extent=(0.032,0.051,0.015)  sigma=(0.006,0.004,0.002)  coverage=0.12
9  id=obj-09  name=unknown-5  xyz=(0.602,-0.498,0.116)  ts_offset=285.9  members=2  extent=(0.079,0.087,0.027)  sigma=(0.001,0.001,0.001)  coverage=0.12
10  id=obj-10  name=unknown-6  xyz=(1.056,-1.102,0.127)  ts_offset=284.1  members=3  extent=(0.100,0.080,0.036)  sigma=(0.008,0.007,0.020)  coverage=0.12
11  id=obj-11  name=unknown-7  xyz=(0.919,0.530,0.121)  ts_offset=270.9  members=2  extent=(0.041,0.036,0.005)  sigma=(0.001,0.001,0.000)  coverage=0.12
12  id=obj-12  name=unknown-8  xyz=(1.019,0.475,0.122)  ts_offset=275.9  members=2  extent=(0.157,0.186,0.005)  sigma=(0.002,0.009,0.000)  coverage=0.12
13  id=obj-13  name=unknown-9  xyz=(0.997,0.311,0.119)  ts_offset=275.9  members=2  extent=(0.071,0.051,0.005)  sigma=(0.011,0.001,0.000)  coverage=0.12

Prerequisites

recording

xarm6_worldbelief_realsense_d435i_stationery_calibrated/xarm6_worldbelief_20260729_203624_161992.db

models

1. EdgeTAM.
2. SigLIP and OmDet (Hugging Face transformers).

[codecov]

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
4260	3	4257	69

View the top 2 failed test(s) by shortest run time

dimos.e2e_tests.test_dimsim_path_replaning::test_path_replanning

Stack Traces | 115s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f54a417d1c0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f5487e6ec00>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f54a46c5ac0>
direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a453e780>
spawn_wall_on_pose = <function spawn_wall_on_pose.<locals>.spawn at 0x7f5487e6f600>

    @pytest.mark.self_hosted_large
    def test_path_replanning(
        lcm_spy, start_blueprint, dim_sim, direct_cmd_vel_explorer, spawn_wall_on_pose
    ) -> None:
        start_blueprint(
            "--dimsim-scene=empty",
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        # robot spawns at (3, 2)
    
        # side wall
        dim_sim.add_wall(2, -2.5, 12, -2.5)
        # other side wall
        dim_sim.add_wall(2, 3.5, 12, 3.5)
        # back wall (behind robot)
        dim_sim.add_wall(2, -2.5, 2, 3.5)
        # forward wall (far end)
        dim_sim.add_wall(12, -2.5, 12, 3.5)
        # dividing wall at x=7 with doors at y=[-1.5,-0.5] and y=[1.5,2.5]
        dim_sim.add_wall(7, -2.5, 7, -1.5)
        dim_sim.add_wall(7, -0.5, 7, 1.5)
        dim_sim.add_wall(7, 2.5, 7, 3.5)
    
        direct_cmd_vel_explorer.linear_speed = 0.8
>       direct_cmd_vel_explorer.follow_points([(10, 2), (2.5, 2), (3, 2)])

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f54a46c5ac0>
direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a453e780>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f54a417d1c0>
spawn_wall_on_pose = <function spawn_wall_on_pose.<locals>.spawn at 0x7f5487e6f600>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f5487e6ec00>

dimos/e2e_tests/test_dimsim_path_replaning.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../simulation/mujoco/direct_cmd_vel_explorer.py:105: in follow_points
    self._wait_for_pose()
        self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a453e780>
        waypoints  = [(10, 2), (2.5, 2), (3, 2)]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a453e780>

    def _wait_for_pose(self) -> PoseStamped:
        self._new_pose.clear()
        self._new_pose.wait(timeout=5.0)
>       assert self._pose is not None, "No odom received"
E       AssertionError: No odom received

self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a453e780>

.../simulation/mujoco/direct_cmd_vel_explorer.py:65: AssertionError

dimos.e2e_tests.test_dimsim_walk_forward::test_walk_forward

Stack Traces | 201s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f54a5974950>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f5487ebc400>
human_input = <function human_input.<locals>.send_human_input at 0x7f5487ebc4a0>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f55a86b40b0>

    @pytest.mark.self_hosted_large
    def test_walk_forward(lcm_spy, start_blueprint, human_input, dim_sim) -> None:
        start_blueprint(
            "run",
            "--disable",
            "spatial-memory",
            "--disable",
            "security-module",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        origin_x, origin_y = 1, 2
        dim_sim.set_agent_position(origin_x, origin_y)
    
        human_input("move forward 3 meter")
    
>       lcm_spy.wait_until_odom_position(origin_x + 3, origin_y, threshold=0.4, timeout=120)

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f55a86b40b0>
human_input = <function human_input.<locals>.send_human_input at 0x7f5487ebc4a0>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f54a5974950>
origin_x   = 1
origin_y   = 2
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f5487ebc400>

dimos/e2e_tests/test_dimsim_walk_forward.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/lcm_spy.py:167: in wait_until_odom_position
    self.wait_for_message_result(
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7f5487ebcd60>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f54a5974950>
        threshold  = 0.4
        timeout    = 120
        x          = 4
        y          = 2
dimos/e2e_tests/lcm_spy.py:153: in wait_for_message_result
    wait_until(
        event      = <threading.Event at 0x7f54a41b3e90: unset>
        fail_message = 'Failed to get to position x=4, y=2'
        listener   = <function LcmSpy.wait_for_message_result.<locals>.listener at 0x7f5487ebd120>
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7f5487ebcd60>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f54a5974950>
        timeout    = 120
        topic      = '/odom#geometry_msgs.PoseStamped'
        type       = <class 'dimos.msgs.geometry_msgs.PoseStamped.PoseStamped'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <bound method Event.is_set of <threading.Event at 0x7f54a41b3e90: unset>>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Failed to get to position x=4, y=2

deadline   = 3347350.818442812
interval   = 0.1
message    = 'Failed to get to position x=4, y=2'
predicate  = <bound method Event.is_set of <threading.Event at 0x7f54a41b3e90: unset>>
timeout    = 120

.../utils/testing/waiting.py:35: TimeoutError

View the full list of 1 ❄️ flaky test(s)

dimos.e2e_tests.test_dimsim_spatial_memory::test_go_to_the_bed

Flake rate in main: 16.67% (Passed 125 times, Failed 25 times)

Stack Traces | 107s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f55a89fc560>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f5487e6ff60>
human_input = <function human_input.<locals>.send_human_input at 0x7f5487ebc180>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f54a599ea20>
explore_house = <function explore_house.<locals>.explore at 0x7f5487ebc860>

    @pytest.mark.self_hosted_large
    def test_go_to_the_bed(lcm_spy, start_blueprint, human_input, dim_sim, explore_house) -> None:
        start_blueprint(
            "run",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
>       explore_house()

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7f54a599ea20>
explore_house = <function explore_house.<locals>.explore at 0x7f5487ebc860>
human_input = <function human_input.<locals>.send_human_input at 0x7f5487ebc180>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7f55a89fc560>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7f5487e6ff60>

dimos/e2e_tests/test_dimsim_spatial_memory.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/conftest.py:249: in explore
    direct_cmd_vel_explorer.follow_points(points)
        direct_cmd_vel_explorer = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a5975f40>
        points     = [(3.881, 4.803), (4.16, 1.615), (1.596, 1.505), (1.649, 0.137), (-3.644, -0.064), (-3.759, -2.661), ...]
.../simulation/mujoco/direct_cmd_vel_explorer.py:105: in follow_points
    self._wait_for_pose()
        self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a5975f40>
        waypoints  = [(3.881, 4.803), (4.16, 1.615), (1.596, 1.505), (1.649, 0.137), (-3.644, -0.064), (-3.759, -2.661), ...]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a5975f40>

    def _wait_for_pose(self) -> PoseStamped:
        self._new_pose.clear()
        self._new_pose.wait(timeout=5.0)
>       assert self._pose is not None, "No odom received"
E       AssertionError: No odom received

self       = <dimos.simulation.mujoco.direct_cmd_vel_explorer.DirectCmdVelExplorer object at 0x7f54a5975f40>

.../simulation/mujoco/direct_cmd_vel_explorer.py:65: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

rewriting user facing API in the second pass, will comment with the linear spec now