feat(evals): pointcloud comprehension suite over a go2 office replay by spomichter · Pull Request #4075 · dimensionalOS/dimos

spomichter · 2026-09-11T06:26:47Z

Problem

hidden eval for @hvent90 milestone tomorrow

Solution

new suite dimos.evals.suites.pointcloud_office — 5 hand-labelled cases against the go2_office_pc recording. the agent sees the final global_map frame (via PointCloud2.agent_encode) plus the odom path.
brings over PointCloud2.agent_encode (+ tests) so the map is legible to the model on main.
the full recording (all streams, untruncated) ships in LFS as go2_office_pc.db.

questions + ground truth:

how many rooms did you walk into → 2
how many rooms did you walk past incl. into → 3
how many open doorways → 4
does the biggest room have objects over 50% of its 2d area → yes
was the first room bigger or smaller than the second → bigger

Breaking Changes

None

How to Test

one line (checkout pulls the LFS dataset, then run):

uv run dimos evals run dimos.evals.suites.pointcloud_office --agent dimos.evals.agents.question_answer

baseline so far: question_answer scores 0.20 (1/5) with gpt-5.6-luna and does not beat the blind ablation (0.25) — the geometric encoding isnt giving the model usable signal for these semantic room questions yet. that gap is the point.

run it blind to confirm guessability:

uv run dimos evals run dimos.evals.suites.pointcloud_office --agent dimos.evals.agents.blind

Contributor License Agreement

I have read and approved the CLA

five hand-labelled questions (room counts, doorways, occupancy, size compare) against the final global_map frame + odom path. brings over PointCloud2.agent_encode so the map is legible to the model on main. dataset (3mb slice) in lfs as go2_office_pc.db.

…uncated)

greptile-apps · 2026-09-11T06:35:58Z

Confidence Score: 4/5

Not safe to merge until point-cloud encoding can safely handle invalid sensor coordinates. The evaluation-scoring issue is non-blocking but should be corrected so reported results reflect the documented policy.

Findings

Filter Non-Finite Points ▶
Restore Off-By-One Credit ▶

Summary

Point-cloud encoding should discard non-finite coordinates before calculating bounds, raster cells, and derived geometry.
The office count evaluation does not provide the documented partial credit for answers one integer away.

T-Rex validation blocked

The focused non-finite point-cloud check could not import PointCloud2 because structlog is unavailable, so it did not reach agent_encode().

_{Reviews (1) · Last reviewed commit: "chore(evals): use the full go2_office_pc..."}

greptile-apps · 2026-09-11T06:36:01Z

+        n = int(pts.shape[0])
+        out: dict[str, object] = {
+            "frame_id": self.frame_id,


Filter Non-Finite Points

A point cloud containing a non-finite XYZ return can break agent_encode(). PointCloud2 construction and the ROS/WebRTC decoding paths preserve NaN and infinity values, but the raster calculation passes them through min/max, floor, integer conversion, and array indexing. Invalid lidar returns can therefore crash the encoding instead of producing an agent observation. Filter non-finite rows before calculating the encoding.

greptile-apps · 2026-09-11T06:36:01Z

+        grade=lambda o: within(1.0)(2.0, first_number(o.trajectory.final_answer)),
+        tags=frozenset({"pointcloud", "count"}),
+    ),
+    EvalCase(
+        id="pc_rooms_passed",
+        inputs=(
+            "How many distinct rooms did you walk past or into in total? "
+            "Answer with just the number."
+        ),
+        environment=_env(),
+        grade=lambda o: within(1.0)(3.0, first_number(o.trajectory.final_answer)),
+        tags=frozenset({"pointcloud", "count"}),
+    ),
+    EvalCase(
+        id="pc_open_doorways",
+        inputs=(
+            "How many open doorways did you see? The building has only standard "
+            "residential-width doors. Answer with just the number."
+        ),
+        environment=_env(),
+        grade=lambda o: within(1.0)(4.0, first_number(o.trajectory.final_answer)),


Restore Off-By-One Credit

The three count graders use within(1.0), so an integer answer exactly one away from the expected count receives zero credit. This conflicts with the documented off-by-one partial-credit policy and makes the count cases effectively exact-match for integer responses, distorting the suite's reported mean score. This is non-blocking for runtime behavior, but use a tolerance greater than one or otherwise implement the stated policy.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Evidence from the check

Authored and executed Python harness that invokes each supplied count grader with integer answers one below and one above its expected count, showing the tested grading scope.

Command output from the check

Successful execution capture of all six one-away integer grading cases, each scoring 0.0, showing no off-by-one partial credit.

Command output from the check

Second successful execution capture at the identical scope, again showing all six one-away integer grading cases score 0.0.

_{Ran code and verified through T-Rex}

codecov · 2026-09-11T06:38:36Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
5633	1	5632	96

View the top 1 failed test(s) by shortest run time

dimos.msgs.sensor_msgs.test_PointCloud2::test_agent_encode_fused_map_fits_the_cap

Stack Traces | 0.005s run time

@pytest.mark.self_hosted
    def test_agent_encode_fused_map_fits_the_cap() -> None:
        """A whole recording fused into one map is the largest cloud the agent
        reads; it must still come back in one readout, at a coarser cell."""
>       from dimos.evals.suites.lib.generate import _dataset
E       ImportError: cannot import name '_dataset' from 'dimos.evals.suites.lib.generate' (.../suites/lib/generate.py)


.../msgs/sensor_msgs/test_PointCloud2.py:426: ImportError

View the full list of 7 ❄️ flaky test(s)

dimos.experimental.memory.test_rust_recorder_e2e::test_cli_recording_uses_existing_binary_for_both_formats[mcap]