Experimental protocol
How ForceBench evaluates force integration
The observations, data, training setup, and execution rules used in the manuscript.
1. Simulation and observation interface
ForceBench is built on Isaac Lab / Isaac Sim and uses a Franka arm with a parallel gripper. Parameterized assets vary mass, center of mass, resistance, and initial poses across task instances.
| Component | Submitted-paper specification |
|---|---|
| Vision | External and wrist RGB images, each 224 × 224 pixels. |
| Robot state | Seven values: TCP position, roll–pitch–yaw, and gripper state. |
| Language | A task instruction. |
| Force observation | Six-axis wrist wrench: Fx, Fy, Fz, Tx, Ty, Tz. |
| Wrench convention | Parent-on-child reaction at the wrist–hand joint, expressed in the joint frame without gravity compensation. |
| Action | Absolute TCP pose and gripper targets. |
The no-force baseline receives images, language, and proprioception without the wrench observation. Privileged simulator state is available to demonstration experts, but excluded from policy inputs.
2. Demonstrations and training
Task-specific scripted experts collect 1,600 successful demonstrations: 200 per task. Only trajectories satisfying the evaluation success criteria are retained. Physics runs at 60 Hz; control and synchronized observation–action recording run at 15 Hz.
- Training instances: Seeds 0–199. A separate policy is trained for each task and method, including the external references.
- Shared-backbone comparison: Seven π0.5 configurations use the same task demonstrations and train for 10,000 steps with batch size 32.
- Additional references: π0, ForceVLA, and ForceFlow. The paper specifies 10,000 training steps for π0 and 100,000 for ForceFlow.
3. Simulation execution
| Setting | Protocol |
|---|---|
| Physics rate | 60 Hz. |
| Control rate | 15 Hz. |
| Episode budget | 1,500 control steps, equivalent to 100 seconds of simulated time. |
| Controller | Damped-least-squares inverse kinematics with synchronized quintic joint interpolation. |
| π0.5 configurations, π0, ForceVLA | Predict 50-step action chunks and execute all 50 steps before querying with updated observations, unless the episode ends earlier. |
| ForceFlow | Predict 64 steps and execute the first 32 before the next query. |
15 Hz is the control rate, not the policy-query rate. The system obtains a new action chunk after executing the specified number of control steps. The external references retain the difference in execution horizon described above.
4. Evaluation and success
All ten systems use the same evaluation seeds, 300–499, for each task. Each task–method pair has 200 evaluation episodes, for 16,000 simulation episodes across ten systems and eight tasks. Overall and category-level means weight their constituent tasks equally.
Full success uses task-specific goal, release, clearance, stability, or latched-completion requirements. Per-episode destination choices, mechanism progress, and assembly geometry provide additional diagnostics. Reaching an intermediate physical state does not automatically count as complete success.
Faucet Direction: the shared criterion
For direction sign d ∈ {−1, +1}, final angle θf, and resistance-peak magnitude C, all ten systems use dθf ≥ C, with no upper-angle bound. The gripper must be at least 65% open, the TCP at least 9 cm above the closed-angle grasp height, and the absolute valve angular speed no greater than 0.01 rad/s.
See the eight task descriptions for the other decision and completion requirements, and the results page for task-level success rates.
5. Interpreting the method comparison
The seven π0.5 configurations comprise a no-force baseline and six force integration designs:
| Design | Force pathway |
|---|---|
| Token Fusion | An MLP maps the current wrench to six continuous tokens before action tokens. |
| Prompt Fusion | Force is appended to robot state, normalized, and discretized into numerical prompt tokens. |
| Cross-Attention | Force tokens condition the action-expert output before action projection. |
| Token Fusion + Pred | Token Fusion plus training-only prediction of ten future force samples, with auxiliary weight 3 × 10−4. The prediction head does not select or modify deployed actions. |
| Temporal Prompt | The current sample and per-axis mean and standard deviation over the last 16 samples enter the numerical prompt. |
| Dual-Path Fusion | The Temporal Prompt pathway plus four continuous action-expert tokens derived from 25 masked window statistics. This second pathway uses 30% window dropout during training; the window is always provided at deployment. |
Input designs jointly vary representation, normalization, capacity, and insertion location. Temporal systems are separate references, not a controlled history-length ablation. The predictive variant retains the Token Fusion input pathway while adding supervision.
6. Real-world evaluation
The physical study evaluates eight systems on five tasks using a Franka Research 3, a six-axis KWR75B force/torque sensor, a Robotiq gripper, and external and wrist cameras.
| Category | Tasks |
|---|---|
| Physical-property inference | Weight Sort. |
| Resistance-dependent interaction | Plate Retrieve, Stamp Press, Plug Insert. |
| Contact-rich assembly | Block Insert. |
- Seven π0.5 configurations and ForceVLA.
- 200 demonstrations per task for each system.
- 10,000 training steps with batch size 64.
- The same 20 initial conditions per task across methods.
- 20 trials per task–method pair, totaling 800 trials.
- The overall mean weights all five tasks equally.
This complementary study examines whether the main qualitative, task-dependent trends also appear in physical manipulation.