Making a convincing tree does not make a robot able to harvest it. Appearance and physical interaction require separate checks. Team Z is building a simulation environment for G1 mandarin harvesting, starting with one nearby tree and one dynamic fruit.
So far, we have checked the tree's appearance, rendered G1 alongside the orchard, and tested one fruit's attachment, release, fall and ground contact. We have not achieved harvesting with G1's hand or autonomous harvesting with a learned policy.
## Refining one tree first
Using photographs as references, we adjusted fruit proportions, branch curvature and the spacing and orientation of leaves. Repeating identical leaves can look artificial even when there are many of them. We used Blender's GUI to adjust edge and surface variation and directly refine foreground leaves, then inspected close and wider renders.
A mandarin close-up rendered in Blender. This is a CG work in progress, not a photograph or an Isaac Sim render.
The tree contains 5,305 leaves and 168 fruit. Those counts are not a measure of realism: leaf thickness and veins, branch structure and calyx details remain simplified. This is not a surveyed reconstruction of a real orchard.
## Separating scenery from the fruit being touched
We exported evaluated Blender geometry to USD and added 14 background trees as references to the same geometry. Rather than simulating every leaf and fruit, we made one foreground fruit dynamic.
The orchard and G1 rendered in Isaac Sim for a static import check. The flat ground is for testing; this is not a harvesting demonstration or a finished photorealistic orchard.
An independent USD read checked units, target position, background references and texture dependencies. Successful geometry export does not establish that Blender materials look identical in Isaac Sim. The two images also use different rendering and lighting setups.
## The fruit did not fall after release
The target centre starts approximately 1.04 m high and the fruit is about 7.7 cm wide. We assigned a provisional mass of 90 g and represented its attachment with a fixed joint. The test holds it for two seconds, disables the joint and simulates four seconds of falling and ground contact at 120 Hz. Collision uses a convex hull of the fruit mesh.
Initially, disabling the joint left the fruit suspended. We recorded the physics engine's internal position as well as the displayed USD position; neither moved. The body remained asleep. After processing the joint change and explicitly waking the body, it fell under gravity and settled near the ground.
The single-condition test passed three checks: less than 5 mm displacement while held; a drop from approximately 1.04 m to a centre height of 0.010 m; and less than 5 mm vertical variation in the final samples. The ground is at −0.02 m. The settled centre height reflects this collision model, not a validated model of a real mandarin. One condition does not establish a general success rate.
A frame from a separate recorded release test. Only the fruit falling in front of the trunk is dynamic. Ground settling was checked using physics-engine position records.
G1's hand was not part of this test. Disabling a joint does not simulate cutting with scissors. Peel deformation, damage and flexible branches remain untested. Background trees were disabled during the numerical test, and branches and leaves do not yet have contact physics. The runtime is Isaac Sim 6.0.0-rc.59.
## Reaching exposes a gap between calculation and motion
With G1's base fixed, we placed the palm target 11 cm in front of the fruit centre. The first inverse-kinematics joint commands left a final palm error of approximately 5.2 cm, failing the criterion of less than 3 cm.
We added feedback from the physics engine's measured palm position to correct the joint commands. The final error decreased to approximately 4.3 mm and passed the same criterion. This compares two controllers in one fixed-base configuration. It uses known simulator positions, not image recognition or a VLA. Grasping and avoiding branches and leaves were not part of this test.
## Collecting data without teleoperation
Our next step is a scripted teacher that uses known simulator positions. Before collecting successful demonstrations, we need to verify reach, contact, grip retention after release and transport. Making the fruit follow the hand directly would not count as a physical grasp.
The teacher may use the exact fruit position, while the learned policy will receive camera images, joint state and task instructions. Feeding ground truth to the policy would not test whether it can act from images. We plan to synchronize images, state and the next command and separate training and evaluation by scene configuration.
ACT is the planned imitation baseline, followed by fine-tuning an existing SmolVLA model. A vision-language-action model generates actions from visual and language inputs. We use the official LeRobot SmolVLA documentation as the reference, and plan to record data in LeRobotDataset format. This is task-specific fine-tuning, not training a foundation model from scratch.
Orchard demonstration collection, ACT training and SmolVLA fine-tuning have not yet been performed. Imitating motion with ACT is also different from selecting a target through language. That requires tests with the same visual scene and different instructions, such as choosing the left or right fruit.
## Next: closing the gap between hand and fruit
The approximately 7.7 cm fruit differs in size and shape from the cylinder used in an earlier, separate grasping project. That project's results cannot establish harvesting success. After reaching in front of the fruit, the next steps are contact, grasping, release and transport. Hardware harvesting remains untested.
The aim is to connect visual modelling, physics checks, teacher demonstrations and learned-policy evaluation while preserving evidence of where each stage fails.
