Most Robot Papers Test on 10 to 30 Trials. Can We Trust the Results?

A real-robot result of "70% success" usually rests on 10 to 20 trials per condition, and at that size the 95% Wilson interval for 7 successes in 10 runs spans roughly 40% to 89% (Paper Huddle calculation, below). A 2026 survey inside the PhAIL paper says the modal per-condition sample in 13 recent real-robot VLA papers is 10 to 20, and that none of the 13 report confidence intervals or paired tests. Single numbers from such runs are anecdotes with decimals attached, but the field is building better tools.

How many trials do robot papers actually run?

Typically 10 to 30 per task. The PhAIL author, Sergey Arkhangelskiy, says real-world VLA evaluation "still rests on binary success rate" with N ≤ 25 rollouts per condition, almost always without confidence intervals or paired comparison. That survey is one author's reading of 13 papers (2023 to 2025) and has not been independently replicated.

Individual papers show the same pattern. OpenVLA evaluated each method in 170 rollouts on BridgeData V2 (17 tasks with 10 trials each) and 60 rollouts on Google robot tasks (12 tasks with 5 trials each). Its appendix lists successes out of 10 trials per task. A follow-up on fine-tuning VLAs evaluated its real-world ALOHA tasks, "fold shorts" and "fold shirt", with 10 trials each.

The authors of ArmnetBench (arXiv 2607.24481, 27 July 2026) put the cause plainly: physical cost limits many studies to a few dozen trials on one robot in one lab, which makes improvements hard to distinguish from noise.

What does 7 successes in 10 trials actually tell you?

It tells you the true success rate is probably somewhere between about 40% and 89%. The Wilson score interval is a standard choice here because it can be used safely with small samples. The table below is a Paper Huddle calculation (95% two-sided, Python/scipy), not a figure from any paper.

Going from 10 to 50 trials roughly halves the interval width. Even at 50 trials, a 70% result is compatible with anything from the mid-50s to the low 80s.

The exact Clopper-Pearson interval is wider still: for 7 of 10 it runs from about 35% to 93%.

Can 10 to 30 trials tell two policies apart?

Rarely, unless the gap is huge. Paper Huddle ran Fisher exact tests on four plausible head-to-head results, and none cleared p < 0.05. A uniform-prior Bayesian reading is more forgiving, but it still leaves meaningful doubt.

Even a 70% versus 20% split at 10 trials each gives p = 0.07 in this calculation. A 20-point gap between 80% and 60% is far harder to confirm.

Power is the other side of the problem. For a true 60% versus 80% gap, a normal-approximation sample-size calculation needs about 80 trials per policy (alpha 0.05, 80% power). A Paper Huddle simulation of Fisher exact tests (20,000 runs, fixed seed) found the following detection rates for that same gap.

Detecting a true 70% versus 80% gap needs about 290 trials per policy under the same approach.

Why do labs run so few trials?

Each real rollout needs hardware, an operator to reset the scene, and a person to score it. The STEP paper (Snyder et al., arXiv 2503.10966) says policy comparison is constrained by a small feasible sample size, "e.g., 10 or 50", because of human effort and limited inference throughput.

STEP's authors also point to a statistical trap. Adding trials after seeing a disappointing result risks inadvertent p-hacking. Their answer is a sequential test that lets researchers decide whether to continue based on intermediate results. The authors report it cuts evaluation trials by up to 32% against state-of-the-art baselines while preserving statistical power (self-reported; simulation plus real-robot experiments).

What do better robot evaluations look like?

They report more than a percentage and, where possible, run many more trials or use a smarter design. The table compares five proposals, all authors' own claims unless noted.

The first paper's abstract lists four practices: explicitly report experimental conditions, add metrics beyond success rate, run statistical analysis, and describe failure modes. It also complains that papers often give little or no information on the number of runs, initial conditions and success criteria.

The LBM work, published in Science Robotics, is named by PhAIL's survey as the single recent counter-example to standard practice. Its 50 real rollouts per task still give an interval of about 25 points at a 70% result (table above), so even the careful protocol is not precise.

On PhAIL's own results, the test separated GR00T from ACT and OpenPI from ACT, but the closest pair (OpenPI versus GR00T) stayed unresolved within its budget. The abstract also reports the best evaluated VLA as about 7 times slower per operation than the human reference.

How should you read a robot result?

Check five things before trusting a headline number. This is Paper Huddle's checklist, derived from the papers above.

1. Raw counts. "16/20" lets you compute an interval; "80%" does not.
2. Trials per task, not just total. OpenVLA's 170 rollouts are 17 cells of 10. Pooled numbers are tighter than per-task numbers, but pooling assumes tasks are comparable.
3. Initial conditions and success criteria. The Cornell and Toyota paper flags these as often missing.
4. Blinding and ordering. The LBM study used blind A/B testing and randomized policy ordering.
5. Interval or test. If the gap between two policies is smaller than the intervals above, treat it as unresolved.

What's still unclear?

Several points remain open.

• How representative is the PhAIL survey? It covers 13 papers from 2023 to 2025 and comes from a single-author preprint. The direction matches the OpenVLA and ArmnetBench evidence, but the exact "10 to 20" mode is that author's count.
• Is a p-value the right bar? The tables above show a gap between frequentist significance and a Bayesian "probably better". 8 of 10 versus 5 of 10 gives p = 0.35 but a 0.91 posterior probability that A is better under a uniform prior. Both are fair readings; a paper should say which it uses.
• Do pooled tasks rescue small cells? Pooling shrinks intervals, but only if success on one task says something about another. That assumption is rarely tested.
• Can large protocols scale? LBM, RoboArena and ArmnetBench (2,518 policy rollouts across 7 policies and 12 tasks, self-reported) show it is possible, but each needs substantial infrastructure. Most labs cannot afford it for every paper.
• Scope of sources. The PhAIL survey section and the OpenVLA evaluation text come from the full PDFs. The STEP, Cornell and Toyota, LBM, RoboArena and ArmnetBench claims above rest on abstracts and introductions. All interval, p-value and power figures are Paper Huddle's own arithmetic, not results from those papers.

The conclusion is narrow. A 10 to 30 trial result is not worthless, but it supports "this works sometimes" far better than "this beats that". Close comparisons need either more trials, a sequential or paired design, or an interval printed next to the number.