Skip to content

feat: add vision-relative front-flap pregrasp and teleop data path - #51

Merged
Alpenj merged 1 commit into
mainfrom
pro/vision-relative-manipulation-02
Sep 7, 2026
Merged

Alpenj merged 1 commit into
mainfrom
pro/vision-relative-manipulation-02

Conversation

@Alpenj

@Alpenj Alpenj commented Sep 4, 2026 •

Copy link
Copy Markdown
Owner

Scope

  • derive the measured shoe-box pose from simulated H201 metric depth without MuJoCo body pose or segmentation IDs
  • target the robot-facing front tuck flap right-hand grasp region
  • solve guarded right-arm IK segments and add bounded right-wrist RGB correction
  • keep the existing MuJoCo teleop, camera payload, episode quality gate, and DAPIER-native ACT dataset
  • add a separate RoboTwin 2.0 Dual SO-101 embodiment and one deeply verified bimanual handover task

RoboTwin SIM result (2026-09-05)

  • generated Dual SO-101 URDF from the official arm URDF and loaded 12 active joints in SAPIEN
  • handled RTX 5050 / CuRobo 5-DoF IK without the failing fused LBFGS path
  • split concave gripper collision meshes and added body, mast, opposite-arm collision geometry
  • completed left grasp and lift, right approach, physical transfer, gradual left release, and left retreat
  • final gate: both right jaws in contact, opposing normals -0.96178, inter-arm impulse 0, object z 0.899101 m
  • no equality constraint or fake object attachment
  • generated H.264 video with 1,256 frames and HDF5 with 1,255 state/action steps
  • converted the real episode to 12D state/action, left/right 320x240 RGB, and H201 640x460 uint16-mm depth
  • replaced broken OIDN ray-traced output with a DAPIER-only SAPIEN raster path
  • record measured SAPIEN qpos instead of drive targets; maximum 15 Hz arm step is 0.033236 rad

Verification

  • DAPIER RoboTwin 12-axis and mount contract: PASS
  • RoboTwin HDF5 to canonical episode fixture: PASS, including unsafe action rejection
  • official runtime installer and SAPIEN load: PASS, 12 joints / 25 links
  • successful physical-contact SIM seed: 3; failed seeds 0-2 were not counted
  • actual HDF5 canonical conversion gate: PASS
  • Python compilation, patch reverse-check, and git diff check: PASS

Not yet complete

  • multiple-seed robustness and measured-range domain randomization
  • ACT training and held-out RoboTwin evaluation with these episodes
  • physical wrist/H201 intrinsics and extrinsics
  • STS3215 dynamics identification, follower calibration mapping, shadow inference, and low-speed rollout
  • physical lid opening and shoe extraction

This PR remains Draft. SIM and mock evidence does not authorize physical execution.

Document phase-specific IL/IK/rule ownership, Python/C++/C responsibilities, optional upstream runtimes and sim-to-real evidence boundaries. Link AGENTS and clarify legacy reference guides.

Checks: git diff --check; local Markdown links/private-source scan PASS; installed Hermes load_soul_md and _load_agents_md reference checks PASS. Documentation only; no runtime tests or hardware access. Private profile/source index remains local.
@Alpenj
Alpenj changed the base branch from pro/mujoco-shoe-followup-01 to main September 7, 2026 01:54
@Alpenj
Alpenj merged commit 137678d into main Sep 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant