Original experiment footage is shared with the Japanese edition. Some on-screen labels are in Japanese; methods, results and video descriptions are provided in English below.
When a person tracked behind a shelf by a fixed camera appears in G1’s own camera, can the robot continue tracking them as the same person? We connected the fixed camera and the camera on a G1 walking with Visual SLAM to a shared tracking pipeline.
Carry the fixed-camera ID and history into G1’s observations
The fixed-camera detector records appearance features and an estimated floor position. When the person appears in G1’s camera, the tracker compares position in the common map and image features, then adds the observation to an existing ID. The shared history retains the source camera and observation timestamp.
In this 18-second run, ID 1 entered the designated area at 10.33 seconds and triggered a ROS 2 stop request. At 10.53 seconds, an observation from G1’s camera was associated with the same ID 1. It joined a history containing 106 observations immediately before handoff.
The scoring procedure matched 125 fixed-camera observations and 11 G1 observations to the person, all with the same ID. The 11 matched pairs across cameras also agreed. This is a short, single-person scene. Scoring associates estimated floor locations with simulator person positions by distance; it is not a general person re-identification accuracy result.
What the shared ID is for
A shared ID is not required simply to stop when a person enters an area. Receiving information from a fixed camera can itself enable an earlier stop request. The ID links the target that caused the stop to the target G1 subsequently sees. ROS 2 records both the stop request and the target ID, while simultaneous observations are consolidated into one target entry.
Our first release tested ID improvements between two fixed cameras separately from stop requests to G1. The stop comparison alone did not demonstrate a handoff from a fixed camera to G1. We implemented that missing connection and ran the moving robot, image recognition and ROS 2 together.
Stop release and route changes based on tracking history are not implemented. This experiment does not show that adding a shared ID improved stopping performance.
Visual SLAM, timestamps and coordinate frames
G1 walks in Isaac Sim physics. Head-mounted stereo images feed cuVSLAM, whose estimated pose drives walking commands along a short route. Initial placement on the map is assumed known. Simulator ground-truth position is not used as the robot’s localization input for route following.
Fixed cameras use their installation poses; G1’s camera uses its Visual SLAM pose. Person positions are transformed into the common map. We assume the bottom center of each detection box is the person’s foot position and that the floor is planar. Upper-body-only detections or crouching violate this assumption. Matching calibration and tf2 calculations does not establish that the estimated person position is correct.
Timing matters too. For a person walking at 1.55 m/s, a one-second-old position is about 1.55 m behind. Treating it as current can cause incorrect associations or missed area entries. The tracker separates observation and receipt timestamps, rejecting stale, out-of-order and duplicate inputs.
The new run produced 450 Visual SLAM estimates. After the stop request, G1 took about 2.04 seconds and traveled about 40 cm before horizontal speed stayed below 0.08 m/s for 0.5 seconds. We measure the command and actual deceleration separately.
An earlier comparison with and without fixed-camera information produced a stop request 0.50 seconds earlier and a stopping position about 27 cm earlier. Those numbers belong to the earlier experiment, not to the current ID integration.
Verify the handoff by replaying the same logs
We recorded recognition inputs, camera transforms, clocks, stop requests and stop-reason IDs in rosbag2. Recomputing tracking from the new run’s 180 recognition inputs reproduced IDs, handoff events, stop-reason IDs and region decisions. Coordinates were compared with a 1e-9 m tolerance to separate small numerical differences. Actual ROS 2 replay also preserved the contents and order of 180 recognition inputs and 900 stop requests.
We also tested saved inputs from another run. Cutting fixed-camera input after 10.5 seconds still allowed G1 observations to join the ID with its previous 105 observations. Reversing camera processing order preserved person association, and duplicate inputs were rejected. Making G1 inputs 400 ms older prevented handoff because the observations were rejected as stale.
Input cut-off and delay tests operate on saved logs; the physical simulation was not rerun under those conditions. Reproducing decisions from recorded inputs is also different from proving that each physical simulation run is identical.
What carries over from five-person tracking, and what remains
The shared tracker uses an earlier improvement tested on five people. It retains initial appearance features and freezes feature updates while person boxes overlap, reducing contamination from another person’s clothing or the background.
Across 300 images of the same five actions, cross-camera ID agreement improved from 648/724 pairs (89.5%) to 720/724 (99.4%). In another sequence where everyone walked, it rose from 396/553 (71.6%) to 526/553 (95.1%). Misidentifications remain, and both sequences were used during development. The five people are color variants of the purchased 142_Kei model, not five distinct faces and body types.
The fixed-camera/G1 test contains one person. Multiple people crossing in front of G1, identical clothing, re-identification after long occlusion and real footage remain untested. The detector also sometimes labels G1 itself as a person; that issue remains visible.
Person paths, appearance times and decision regions were set in advance, but neither those schedules nor ground-truth IDs were passed to the tracker. The simulation clock pauses during image inference, so this is not a real-time performance result. It is not a guarantee of a physical robot’s safety or stopping performance. It connects image recognition, ID handoff, stop reasons and replay evaluation in one experiment.
Experiment date: September 10, 2026. Isaac Sim 5.1.0.0, Isaac Lab 2.3.0, ROS 2 Humble, cuVSLAM 17 and NVIDIA L4. No external MCT product is connected.
LET’S BUILD TOGETHER
What are you working on?
Tell us about the software you need to implement or test. We can discuss the work, the development setup and how we could contribute.
Discuss your development needs