Embodied AI: RoboTwin evaluation (StarVLA codebase)

One StarVLA checkpoint must be evaluated across 50 RoboTwin manipulation tasks. At first, running them on several GPUs appears to require only a loop. In practice, balancing the work, tracking failures, and cleaning up every process requires a 548-line Bash launcher.

The application

StarVLA develops Vision-Language-Action policies that turn camera observations, robot state, and language instructions into actions. Its RoboTwin 2.0 integration evaluates a policy on 50 simulated tasks in clean or randomized settings.

Each case starts a StarVLA policy server on a GPU and a RoboTwin simulator in a separate Python environment. Cases are independent and may finish at different times, so a free GPU should take the next case immediately.

What the project has to maintain

StarVLA's start_eval.sh already performs that dynamic scheduling. The launcher alone is 548 lines, excluding the policy-server and evaluation scripts it invokes. It must discover GPUs, allocate ports, track jobs and PIDs, wait for servers, refill free slots, collect failures, organize logs, and recursively clean up processes.

Its scheduling shape, heavily abridged, looks like this:

# Abridged architecture, not the complete 548-line launcher.
TASKS=(...50 RoboTwin task names...)
SLOT_GPUS=()
SLOT_PORTS=()
ACTIVE_PIDS=()
ACTIVE_TASKS=()
FAILED_TASKS=()

trap cleanup_active_jobs EXIT INT TERM

launch_task_in_slot() {
    start_policy_server "$gpu" "$port" &
    wait_for_server "$port"
    run_robotwin_eval "$task" "$mode" "$port"
}

while (( completed_tasks < total_tasks )); do
    for slot in "${slots[@]}"; do
        collect_finished_process "$slot"
        if slot_is_free "$slot"; then
            launch_task_in_slot "$slot" "${TASKS[$next_task]}" &
            remember_pid_task_and_logs "$slot" "$!"
        fi
    done
    sleep 5
done

This is reasonable engineering. Dynamic parallelism requires the launcher to schedule jobs and manage processes. Progress is stored only in the launcher and its logs, so after an interruption the researcher must determine what finished before starting the next run.

What Labtasker handles

The useful Task boundary is one checkpoint × RoboTwin task × mode. Submitting all 50 tasks in both modes creates 100 explicit Tasks:

# Architectural excerpt, not a drop-in StarVLA integration.
from itertools import product

import labtasker

# TODO: Replace these placeholders with values from your StarVLA configuration.
ROBOTWIN_TASKS = [...]
CHECKPOINT = ...
SEED = ...
MODES = ["demo_clean", "demo_randomized"]

for task, mode in product(ROBOTWIN_TASKS, MODES):
    labtasker.submit_task(
        {
            "checkpoint": CHECKPOINT,
            "task": task,
            "mode": mode,
            "seed": SEED,
        },
        name=f"{task}-{mode}",
        routes=["robotwin"],
        max_attempts=3,
    )

# Architectural excerpt: run one Worker on each selected GPU.
import labtasker


@labtasker.loop(route="robotwin")
def evaluate(
    checkpoint: str = labtasker.TaskArg(),
    task: str = labtasker.TaskArg(),
    mode: str = labtasker.TaskArg(),
    seed: int = labtasker.TaskArg(),
) -> None:
    # TODO: Replace this call with your actual one-case evaluation code.
    result = run_one_robotwin_case(checkpoint, task, mode, seed)
    labtasker.finish(
        {
            "success_rate": result.success_rate,
            "log": str(result.log_path),
        }
    )


evaluate()

These excerpts show the design and are not a complete StarVLA integration. The Worker contains the StarVLA-specific code for one case. Labtasker distributes the cases and records progress, retries, recovery, and final status.

StarVLA and RoboTwin still own the policy, simulator, task definitions, metrics, environments, logs, and videos. The researcher still chooses which GPUs run Workers. Labtasker replaces only the project-specific coordination layer and exposes the same operations to humans and agents.