Putting Benchmarks in Docker and Cutting Cost to 1/50: Behind SenseTime's RoboRSI, How Far Is Embodied AI From 'Actually Getting Work Done'?

Putting Benchmarks in Docker and Cutting Cost to 1/50: Behind SenseTime’s RoboRSI, How Far Is Embodied AI From “Actually Getting Work Done”?

Introduction

A robotic arm fumbles while stacking bowls. It tries for a long time, finally succeeds — but it averages about 131 minutes and roughly $19 in estimated inference cost. Inserting a pen into a pen holder is even more extreme: 263 minutes on average, around $55. The task did get “done” — but at that speed, it’s light-years from “everyday usable.”

That’s a scene SenseTime encountered while systematically testing OpenAI’s GPT-6-Astra. It exposes a question the whole industry has struggled to quantify: between high scores in virtual benchmarks and stable execution in the real world, what exactly is the engineering gap?

On October 20, SenseTime released a systematic research report — “Has GPT-6-Astra Achieved Embodied Intelligence?” — and open-sourced two things at once: the SenseNova-RoboRSI framework (packaging various embodied benchmarks into Docker images and decoupling benchmark environment from model via a unified HTTP API), and a 27B embodied model, SenseNova-RoboRSI-27B-Preview (internalizing object understanding, action decisions, and feedback correction into its parameters, cutting embodied-task cost to 1/50 of GPT-6-Astra).

In this article, I follow SenseTime’s research and open-source move to unpack the “benchmark-efficiency-deployment” triangle of embodied AI: starting with why evaluation is the choke point, then how RoboRSI uses Docker to “decouple the benchmark environment” and lower reproduction barriers, then how the 27B model reaches a 1/50 cost, and finally landing on the consensus that “the hard part of embodied AI is turning ‘knowing how’ into ‘reliably doing it.’”

Part One High-Scoring Models Fail Once They Hit Real Robots

Let me first lay out the core findings of SenseTime’s research.

SenseTime tested GPT-6-Astra with 12 simulation benchmarks and 8 real robots. The results showed a signature duality:

• On some traditional embodied benchmarks, GPT-6-Astra scores extremely high: 100% on LIBERO-Long++, 99.3% on LIBERO-Pro, 98.8% on SIMPLE, 81.9% on RoboDojo;
• But once in real-robot environments, performance on complex tasks drops visibly. Plug insertion, soft-material manipulation, long-horizon navigation — all expose discontinuities among vision, motion control, time constraints, and physical feedback.

More telling is that “five straight failures” episode. In the RoboDojo simulation, SenseTime found 9 tasks that were originally all-fail across five tries. They didn’t retrain the model — they did just one thing: allowed it to keep interacting, raising the step cap to 10× the original. Then those 9 tasks became 5-of-5 successes.

That finding is deeply counterintuitive, and deeply expensive. Look at the costs SenseTime disclosed: stacking bowls averaged ~131 minutes and ~$19 in estimated inference; inserting a pen into a pen holder averaged ~263 minutes and ~$55. In other words, some tasks “aren’t that the model can’t do them — it’s that we pressed stop too early.” But the cost is that task time and inference cost soar together. Being able to do it within a reasonable budget is the real skill.

There was also a telling real-robot clip: during a flower-arrangement experiment, GPT-6-Astra couldn’t see the bottle mouth clearly, so it proactively moved the idle arm, using the camera on its other hand to “see for itself,” then completed the insertion. It turned the second hand into an “eye” to finish the task. This shows general models do have strong visual understanding and spatial reasoning — “knowing what to do” it achieves; “reliably doing it” still falls short. And that shortfall sits exactly in the crack between evaluation and real use.

Part Two Why Evaluation Is the Hidden Choke Point of Embodied AI

SenseTime’s critique of the gap between existing benchmarks and real use is sharp, and it sketches three criteria for what future embodied leaderboards should look like:

First, tasks should stem from real human needs. Evaluation should be designed around what humans actually want robots to do, while accounting for side effects of operations and those “responsibilities that aren’t explicitly stated.”

Second, experience should be gained through autonomous interaction. When demonstrations for a specific task or scene are lacking, the robot must explore the environment on its own, learn methods, and adjust strategy based on feedback — rather than parroting demonstrations.

Third, the world won’t wait for the model to think. While the model is thinking, the environment keeps changing. So evaluation should count both “thinking” and “acting” in world time, examining whether the task is completed, whether it’s timely, and the consequences during execution.

This is a structural problem the embodied-AI industry has long overlooked. Traditional benchmarks tend to ask only “did the action succeed,” while ignoring: Does the task come from real need? Did the robot learn step by step, or copy a demonstration? In a real scene where both environment and time keep flowing, can it really finish on time with acceptable side effects?

SenseTime even built a physical-interaction “showroom” to demonstrate this: the robot operates furniture, objects, and appliances inside, and even while the model is thinking, water flows and heating continues. In that showroom they observed several representative problems with GPT-6-Astra — tidying a desk, it moves the book in hand but knocks over other items and tilts the monitor; stowing food, it moves off to carry fruit while leaving the fridge door open for minutes. These are the discontinuities between “high-scoring virtual models” and “stable real-world execution,” and evidence that the whole loop of navigation, perception, planning, control, and feedback hasn’t truly clicked together.

Part Three RoboRSI: Put Benchmarks in Docker and Decouple Environment from Model via HTTP API

Once you understand evaluation is the choke point, SenseTime’s open-source move becomes clear in direction.

The core idea of SenseNova-RoboRSI is to package each embodied benchmark and its runtime dependencies into independent Docker images, and expose environment interaction plus native scoring through a unified HTTP API, thereby decoupling the “benchmark environment” from the “Agent model / policy.” The direct benefit is obvious: researchers no longer need to manually deploy complex simulation environments; they only prepare a config file and can run end-to-end evaluation and reproduce results with one click. This lowers the reproduction barrier and makes it easy for the community to validate results under a unified environment and compare different models and Harness designs.

Here’s a small schematic of how this “benchmark-environment decoupling” works — isolating the benchmark in a container while the model talks to it via HTTP API:

import subprocess, requests, time, json

class BenchmarkGateway:
    """Package the eval env as an isolated benchmark process/container that
       exposes a unified HTTP API to the model."""
    def __init__(self, image_tag, env_cfg, api_base="http://127.0.0.1:8600"):
        self.api_base = api_base
        self.image_tag = image_tag
        self.env_cfg = env_cfg

    def launch(self):
        cmd = ["docker", "run", "-d", "--rm", "-p", "8600:8600", self.image_tag]
        self.container_id = subprocess.check_output(cmd, text=True).strip()
        for _ in range(60):
            try:
                r = requests.get(self.api_base + "/health", timeout=1)
                if r.status_code == 200:
                    print(f"[gateway] container ready: {self.container_id}")
                    return
            except requests.RequestException:
                time.sleep(1)
        raise RuntimeError("benchmark container failed to become ready")

    def reset_task(self, task_id):
        return requests.post(self.api_base + "/reset",
                             json={"task": task_id, "cfg": self.env_cfg}).json()

    def step(self, action):
        return requests.post(self.api_base + "/step", json={"action": action}).json()

    def submit_result(self, task_id, trace):
        return requests.post(self.api_base + "/score",
                             json={"task": task_id, "trace": trace}).json()

# Usage: wrap a "stacking bowls" benchmark into an image; the model only
# talks to the environment through the HTTP API.
gw = BenchmarkGateway(image_tag="robobench/stacking:0.6",
                      env_cfg={"max_steps": 50_000, "world_time": True})
gw.launch()
obs = gw.reset_task("stack_bowls")
# ... policy(obs) -> action -> gw.step(action) ...
print("submitted:", gw.submit_result("stack_bowls", trace={}))

What this code conveys is not a real implementation, but the value of turning “evaluation” into reusable, reproducible, comparable infrastructure in itself: when every benchmark can be spun up with one click and settled through the same HTTP API, “leaderboard scores” actually become an engineering discipline that can be repeatedly examined — not a number game where each lab talks to itself.

Part Four How the 27B Model Reaches a 1/50 Cost

Beyond the evaluation framework, SenseTime’s other move is “distilling the experience validated in experiments back into the model itself.” SenseNova-RoboRSI-27B-Preview goes through multimodal supervised iteration, internalizing object understanding, action decisions, and feedback correction into its parameters. The official results across several benchmarks are striking:

• RoboDojo: 26.13 points, 20.83% task success rate;
• LIBERO-Pro: whether using slow-fast collaboration (System-2 + System-1) or letting System-2 directly control, performance breaks 90 points — second only to GPT-6-Astra — showing generalization across system-architecture designs;
• RoboCasa365: task success up from 26.0% to 46.5%, while task-token overhead drops 73%.

The key is that it trades “small model + capability internalization” for extreme cost efficiency. Compared to GPT-6-Astra’s costly “drive a big model and retry repeatedly” approach — stacking bowls 131min/$19, pen 263min/$55 — the 27B model completes embodied tasks at 1/50 the cost. It cuts per-inference cost through a smaller footprint, and lowers “unit task cost” by internalizing embodied skills and reducing redundant inference and trial-and-error.

Here’s a small code sketch comparing the cost of “internalization reduces trial-and-error” vs “big model brainstorming retries”:

def task_cost_estimate(model_tag, base_inference_per_step, success_steps_mean,
                       budget_multiplier=1.0):
    expected_steps = success_steps_mean * budget_multiplier
    expected_cost = base_inference_per_step * expected_steps
    return {"model": model_tag, "expected_steps": expected_steps,
            "expected_cost_usd": round(expected_cost, 2)}

# Assume: GPT-6-Astra has high per-step inference cost and tends to succeed
#         only through lots of retries (budget ×10);
#         27B is cheap per step and, after internalization, retries little (budget≈1)
gpt6  = task_cost_estimate("GPT-6-Astra", base_inference_per_step=0.15,
                           success_steps_mean=3600, budget_multiplier=10)
roborsi = task_cost_estimate("RoboRSI-27B", base_inference_per_step=0.002,
                             success_steps_mean=720, budget_multiplier=1)
print(gpt6)
print(roborsi)
print(f"cost ratio (27B/GPT-6): {roborsi['expected_cost_usd']/gpt6['expected_cost_usd']:.2f}x")

(Numbers are illustrative; official figures prevail.)

The real engineering logic underneath: embodied AI ultimately competes not only on foundation-model capability, but on how the model calls tools, understands feedback, allocates inference budgets, and coordinates across different robot bodies. “Unit task cost” and “effective completion rate” are replacing raw benchmark rankings as the hard metrics of whether embodied AI is truly commercializable.

Part Five Re-visiting the “Do We Give It Another Chance” Question

Before unpacking the implications for SenseTime and the industry, I want to magnify the most uncomfortable question in the study — “which failures deserve one more chance?” It sounds simple, yet it’s a watershed for embodied-AI engineering deployment.

We all saw the result: those 9 “all-fail-in-five-tries” RoboDojo tasks went from total failure to total success merely by enlarging the step cap to 10×. That finding excites and frightens at once. Excitement: many tasks aren’t “learned-not-to-do”; we just gave too little patience. Fear: 10× steps buys 131 minutes, 263 minutes — astronomical time and cost. In other words, “giving more chances” and “controlling cost” are naturally opposed, and you must find a sustainable balance between the two.

In my view, judging “is it worth one more chance” should look at at least three dimensions:

• Task reversibility: if a failed operation causes irreversible damage to equipment, environment, or people, you shouldn’t let the model retry endlessly; switch to a more conservative strategy after N failures;
• Immediate gain of success vs. cost of failure: for tasks like stacking bowls or arranging flowers where “failure is no big deal,” more chances are worthwhile; for precision assembly or hazardous-environment work where a failure costs dearly, you must have a hard step/budget red line;
• Problem structure: if the bottleneck is “under-observation” (e.g., can’t see the bottle mouth), giving more interaction so the model learns to adjust its viewpoint is far more valuable than blind repetition; if the bottleneck is “action precision” itself, retrying more won’t break physical limits anyway.

In other words, “give it one more chance” shouldn’t be a romantic slogan but a quantifiable budget-allocation strategy. Here’s a code sketch showing how to use “risk weight + success probability” to decide whether to extend the interaction budget:

def should_retry(fail_cost, retry_cost, p_success_per_retry, max_retries,
                 irreversibility=0.0):
    """
    Decide whether more chances are worth it given failure cost, per-retry
    cost, per-retry success probability, and task irreversibility.
    """
    for attempt in range(1, max_retries + 1):
        expected_gain = p_success_per_retry * fail_cost
        expected_loss = (1 - p_success_per_retry) * retry_cost
        if irreversibility > 0.7:
            return False, attempt - 1            # high-risk: stop early
        if expected_gain > expected_loss:
            return True, attempt
    return False, max_retries

# Case A: stacking bowls (failure harmless, cheap, success rises with interaction)
print("stacking:", should_retry(fail_cost=0.5, retry_cost=0.05,
                                p_success_per_retry=0.8, max_retries=10))
# Case B: precision assembly (irreversible, high risk)
print("precision:", should_retry(fail_cost=100, retry_cost=0.2,
                                 p_success_per_retry=0.2, max_retries=10,
                                 irreversibility=0.9))

What this code conveys is the engineering rationality behind “give it more chances”: not every task should be retried endlessly, and not every task should be abandoned in one go. Embodied-AI deployment is essentially a dynamic trade-off among success probability, failure cost, time budget, and cost budget. And work like RoboRSI — turning evaluation into reproducible infrastructure — is precisely what makes this trade-off quantifiable, comparable, and verifiable for the first time.

Part Six The Double Meaning for SenseTime and the Industry

The significance of SenseTime’s open-source move is underrated. It advances embodied-AI research from “model evaluation” to “system efficiency.”

First, the company and industry. In H1 2026, SenseTime’s revenue was RMB 2.91B, up 23.4% YoY, achieving its first IFRS-compliant profit (net profit RMB 620M), with generative-AI revenue of RMB 2.33B accounting for ~80%; its SenseCore GPU service hit 2.4 trillion daily tokens in July, up ~22× YoY, covering 20+ countries/regions and serving 4,500+ customers. SenseTime’s “model system + Token factory + Agent Harness” framework essentially solves a commercialization problem: how to turn increasingly powerful model capabilities into stable task delivery. Embodied AI magnifies this exact problem — each robot task faces real-world uncertainty, and RoboRSI’s path of “make evaluation into infrastructure + compress execution cost” resonates with its pursuit of commercialization efficiency.

Second, the bigger picture. In recent years, big-model competition has revolved around parameters, inference, context, and multimodality, while robots mostly rely on VLA models trained for specific embodiments. GPT-6-Astra’s arrival put both technical routes on the same exam: general models can enter the arena directly with strong vision and reasoning — but “knowing what to do” and “reliably doing it” remain two different things. Simulation especially hides this gap — the digital world has clear rules and controllable state, and the model can observe and retry; the real world has mechanical collisions, overheating motors, constrained viewpoints, and irreversible actions. What embodied AI truly needs to beat is not just “spatial intelligence,” but the full closed loop of perception, planning, control, and feedback.

Further, as robot bodies get cheaper and hardware forms multiply, the scarce link is shifting toward the “embodied brain” and task systems — whoever can understand the environment, plan actions, and revise strategy from execution feedback moves closer to the general-robot software layer. That’s both a demonstration for the open-source community and a wake-up call against the industry’s misconception of treating “leaderboard scores” as “commercialization progress.”

Conclusion

From “stacking bowls takes 131 minutes, inserting a pen 263 minutes,” to those 9 “all-fail-in-five-tries” RoboDojo tasks flipping to success once the step cap was enlarged 10×, SenseTime used a set of honest, uncomfortable data to lay bare the gap between “high scores” and “can it actually work.” And its answer — put benchmarks in Docker, decouple environments via a unified API, use a 27B model to push cost to 1/50 — is less a tech release than a recalibration of embodied-AI research: don’t just chase leaderboard scores; the real battlefield is real-task completion capability, unit execution cost, and reliability in a changing environment.

Going forward, embodied-AI competition will increasingly revolve around “unit task cost” and “effective completion rate.” For everyone involved, the crux might be condensed into one plain sentence: the model knows what to do, and the robot has to actually, quickly, stably, and cheaply do it. That road is still long.

• ←Previous
• Next→