Every motion metric is blind to something
EMMIBS: execution-free scoring of commanded trajectories for manipulation policies. A score, Φ, that judges the motion a policy commands before the robot executes it.
Example: Seven trajectories sampled for one action chunk. Six the arm can follow; one it cannot. Φ tells them apart before anything moves.
Abstract
Evaluating a learned manipulation policy currently means running it on the robot. Robot time is the scarcest resource in a training pipeline, and a rollout returns a single bit, success or failure, that does not say whether the plan, the grasp or the motion itself was at fault.
Φ judges the motion a policy commands before the robot executes it. It is training-free: a robot model and two constants per joint measured on the bench are all it needs; nothing is simulated, learned or tuned by hand. Each action chunk becomes five terms, scored by how unusual the vector is against human demonstrations of the same task, with the responsible term named. Under injected spikes, jitter and chunk-boundary seams, every single metric we test misses at least one class; Φ misses none, in 1.55 ms per chunk.
We built Φ: a scorer that ranks a policy's commanded trajectory before the robot executes it.
THIS PAGE IS A PLAIN-LANGUAGE WALK THROUGH THE CORE OF THE METHOD. EVERY DETAIL IS IN THE PAPER.
It reads the commands, not the outcome. Each action chunk becomes five numbers referenced to what the arm can actually do, and one distance from how humans drive the same task. The rest of this page shows how, what it catches that other metrics miss, and where it stops.
A learned policy emits its actions in chunks. Φ sits between the policy and the servos and scores each chunk before it moves anything.
One inference step, left to right: the policy turns its observations into a block of joint targets; Φ reduces that block to five terms referenced to what the arm can do, asks how unusual the vector is against human demonstrations of the same task, and returns a score with its cause. Pass, and the chunk executes. Flag, and it can be dropped, resampled or logged with the term that failed.
Where Φ sits: on the command path, between the policy and the servos. Drawn for a VLA, the case it was built and tested on; any imitation-learned policy that commands joint targets goes in the same slot. What happens inside Φ is explained below.
A chunk becomes five numbers. Φ is how far those five sit from the way humans drive the same arm.
The distance splits exactly into five signed shares, so the score always comes with the term that caused it.
Four terms describe the shape of the motion, one the join to the previous chunk. Each is referenced to the arm's measured top speed and rate constant, so the same score means the same thing on any joint.
Humans set the reference. A replayed demonstration shows what a good score looks like.
Before deployment, the terms are computed for every chunk of the teleoperated demonstrations of the task, on the same arm. Their mean μ and covariance Σ are the reference, and Φ is the distance from it, counted in σ: how much humans themselves vary on each term.
Φ combines the terms into one distance and splits it back into per-term shares, so a high score always comes with the term and the σ that caused it.
Distance is not a verdict. A practice we found useful, and recommend: before reading any policy, score a trajectory you trust. Replay a recorded demonstration on the robot and score it like a rollout; we call it the oracle. Its Φ per chunk and its three summaries are the yardstick a policy's rollout is read against. Φ says the arm was never driven like this by a person, and names the reason; whether that matters for the task is your call.
Every rollout, paired with its command log. Φ is read on each chunk as the policy emits it.
Each player shows one recording: the clip on the left, the joint angles the policy commanded on the right, drawn as the clip plays, with the score of each chunk appearing the moment that chunk is produced. The tiles underneath switch the player to another recording. First the oracle, a human demonstration replayed on the arm, the numbers that good looks like here; then the policy rollouts; then the same human episode replayed with defects written into it.
Three summaries of the same per-chunk scores, because a trajectory can fail in three ways: one catastrophic chunk, uniformly mediocre chunks, or some bad ones. The large Φ in each player is different: it is the score of the single chunk being executed right now.
{{ st.lead }}
{{ st.body }}
TRAJECTORY
Strip: Φ per chunk, the executing chunk in red. Next to the live Φ: the term that carries it and its distance in human standard deviations. Reference values are placeholders until the fitted (μ, Σ) are in; the numbers are provisional, the reading is not.
Every single metric we tested misses at least one defect class. Φ misses none, on both arms.
Three defects were injected into held-out human episodes at magnitudes from 0.25° to 32°, doubling each step: an isolated command spike, high-frequency jitter, and a seam, a jump at the boundary between two chunks. ε50 is the smallest magnitude at which a detector flags half of the episodes. Each of the five terms alone, and the smoothness scalars SPARC and LDLJ, leaves at least one class undetected on at least one arm. Φ reads all five at once and detects all three on both arms, with no per-arm tuning beyond the two bench constants.
Tables III and IV. Median ε50 over {{ episodeCount }} held-out episodes on the {{ armLabel }}; SPARC and LDLJ per chunk. never: no detection even at 32°. The pose term responded to none of the three classes on either arm and is left out. Per-episode ranges are in the paper.
The same human episode, clean and disturbed.
The clean replay scores lowest of every recording on the RoArm. Each row below is that episode again with one defect written into the commands. All strips share one scale, so the height of a bar is directly comparable across rows.
Computed from the replay logs with the same scorer as the demo. × clean is Φbad of the row divided by Φbad of the clean replay. The reference is provisional until the fitted values are in, so the numbers will move; the ordering is what to read.
Two bench measurements per joint and one pass over your demonstrations. Nothing is learned and nothing is tuned by hand.
Everything Φ needs at inference is fixed before deployment, once per arm and task. The steps below are what the paper did for the SO-101 and the RoArm-M3 Pro; the same recipe applies to any joint-controlled arm with a teleoperated dataset.
Two numbers per joint, read off the hardware. A ramp test gives the top speed q̇peak: command a long move at full rate and take the steady velocity. A step test gives the rate constant: command a small step, fit the response as dead time plus a first-order lag, and keep the time constant τ; the bandwidth fbw = 1 / (2πτ) follows from it. Repeat once per joint.
Take the teleoperated episodes you fine-tune on, on the same arm. Cut each into chunks the way your policy emits them, compute the terms for every chunk with the constants from step 1, and keep the mean μ and covariance Σ over all chunks. That pair is the reference. The paper used 200 episodes per platform; the two datasets are public.
Replay one held-out demonstration on the robot and score it like a rollout. Its Φmax, Φbad and Φmean are what good looks like on your hardware; every policy is read against them, as in the demo above. Φ per chunk costs 1.55 ms, so the same code runs offline over a dataset and online in the control loop.
A distance from teleoperation, not a certificate of admissibility: a motion humans never make can still be safe, and a human-like one can still fail the task. It reads kinematics, not dynamics, so it knows nothing of payload or contact forces. And it needs no task labels, which also means it cannot tell you whether the task was done.
Bench constants and reference for both arms ship as phi/ref_so101.json and phi/ref_roarm.json, together with the scorer that runs on this page; the recipe for reproducing them from the datasets is in the repository.
[ example code, coming soon ]
[ example code, coming soon ]
Hand the setup file to a coding agent: it installs the environment, fetches the reference and a recorded episode, and leaves one command to run. No robot needed.
Krilyshyn, V. and Polrolniczak, P. (2026). EMMIBS: Every Motion Metric Is Blind to Something. Execution-Free Scoring of Commanded Trajectories for Manipulation Policies. Submitted to IEEE ICRA 2027.
@article{krilyshyn2026emmibs,
title = {EMMIBS: Every Motion Metric Is Blind to Something. Execution-Free Scoring of Commanded Trajectories for Manipulation Policies},
author = {Krilyshyn, Viacheslav and Polrolniczak, Pawel},
year = {2026},
note = {Submitted to IEEE ICRA 2027},
url = {https://ghostpai.github.io/emmibs}
}