Technical specification

1. Scope and current release

AstroBench studies how a fixed-base arm captures and retains free-floating objects inside a zero-gravity cabin. An unsupported target moves when touched: a poorly timed approach can push it out of reach, and a closed gripper can still lose it. The evaluation separates acquiring a grasp from keeping it until the end of the episode.

The current release provides a portable Python evaluation package and a gallery of 40 selected grasping examples across 20 object types. The package generates episode plans, scores supplied traces and aggregates results. A separate matched cube pilot contains 40 completed episodes. These three resources serve different purposes: protocol implementation, visual examples and a development experiment.

The broader design covers 11 task families. A procedural object generator, frozen generalization splits and a portable full environment are still under development. The current robot base is fixed; free-floating base–arm reaction dynamics are outside this configuration.

2. Task definitions

A task family defines the goal and judge. An instance fixes the object or mechanism, scene and target. An episode binds that instance to initial conditions, seeds and a protocol version. Repeated poses of one asset remain episodes of the same object.

The gallery currently covers grasping. Other families are a task specification, not a completed benchmark suite. Fixed mechanisms follow their joints and rails. Insertion covers geometric mating and seating; electrical connection, sealing, flexible cables and release-to-lock behavior need additional task definitions.

3. Environment and reset

The engineering environment uses Isaac Sim / Isaac Lab with a Franka Panda arm and Robotiq 2F-85 gripper. The target is a dynamic rigid body under ideal zero gravity. Mass, inertia and friction still affect contact. No supporting surface or hidden attachment should hold the target in place.

At reset, load the declared assets and robot state, set the target pose and velocities once, then refresh sensors and read back the actual state. Verify no initial penetration, forbidden support contact, pre-existing grasp or satisfied goal. Cabin visuals do not define collision boundaries; mounting structures and workspace limits must be checked separately.

After reset, target motion follows dynamics and robot contact. An adapter must not keep writing the object pose to follow a desired trajectory. Record gravity, physics timestep, solver settings, collision geometry and sensor bindings. A seed alone cannot reconstruct an episode when assets or engine settings change.

4. Object identity and splits

The gallery uses fixed assets. The proposed object generator would jointly produce visual geometry, collision geometry, mass properties and metadata for containers, tools, handheld devices and maintenance parts. Selecting a fixed asset or changing its pose is not procedural shape generation. The initial scope is rigid, single-object manipulation.

Generated mass, center of mass and inertia must agree with geometry and material assumptions. Reject invalid collision meshes, initial intersections and out-of-workspace instances before policy evaluation. Keep rejection reasons and resampling indices. A policy failure is not a reason to remove a valid test instance.

The public catalog requires object_id, asset_sha256, geometry_group and split. Split values are train, validation or test. Asset hashes and related geometry groups cannot cross splits. Hashes identify bytes; manual or geometric checks are still needed to find near-duplicate shapes.

The proposed evaluation crosses seen/unseen geometry with seen/held-out motion. Development positions and regression assets remain development data. Formal split sizes and motion distributions have not been frozen. The public planner expands every catalog object over four draft tracks:

The seed chooses signed x-axis translation and z-axis rotation, shared across paired tracks. This is not full orientation or motion-direction coverage. The proposed faster-motion holdout must be excluded from training by the experiment owner; the planner cannot enforce training-data access.

5. Observations, actions and timing

Separate policy inputs from scoring measurements. RGB images and declared proprioception can be policy inputs; object pose and target contact are scoring evidence. A state-feedback reference may use privileged object state, but must be identified as such. Recording cameras are not automatically the cameras used by a policy.

For each policy, record camera pose and calibration, image size, crop and normalization, sensor rate, observation history, joint order and units. State whether actions are absolute joint targets, increments or task-space commands; also declare clipping, reference frame, gripper direction and invalid-action handling.

The engineering control ultimately drives joint-position targets. The matched cube pilot uses seven absolute arm-joint targets plus a binary gripper command at 15 Hz, with 120 Hz physics and eight physics steps per control step. Other engineering recordings use different schedules. The public trace judge expects measurements at nominally 50 Hz and does not prescribe a policy action rate.

Keep physics time, sensor time, control time, policy decision rate and action-chunk length separate. Report inference wall-clock time separately from the episode budget. Clear action queues and hold windows at every reset.

6. Capture and terminal retention

The public judge is versioned as astrobench.capture-retain/0.1-draft. A valid contact segment requires both fingers to contact the selected target, measured closure, no forbidden contact or escape, and bounded relative motion and pose excursion. Pose drift is measured from the start of the candidate segment in the gripper frame.

The first valid sample starts the clock; it does not contribute a sample period of hold time. Invalid contact or excessive motion breaks the segment. Capture is the first completed hold window. Primary success requires a completed window ending at the 12-second horizon. Early capture followed by loss is a failure. Gaps beyond the limit and malformed or truncated traces are rejected rather than silently scored.

These are draft parameters for the portable scorer, awaiting calibration with an environment adapter. They do not retroactively score the gallery or the 15 Hz cube pilot. The criterion only checks sampled evidence; it does not certify contact between samples. Buttons, insertion and release-based stowage require their own judges.

7. Trace and adapter contract

The trace envelope contains episode_id, spec_hash and samples. The identity must match the registered plan. Each sample has exactly the ten fields below (paired names denote separate fields). All numbers must be finite; flags must be JSON booleans.

At nominal 50 Hz, include 601 synchronized samples from t = 0 through t = 12 s. Convert units and frames before scoring. For target position r in the gripper frame, relative linear velocity must include the gripper angular-velocity contribution, rather than just subtracting world-frame velocities.

The adapter verifies asset bytes, scene and robot configuration, reset readbacks and physical sensor mapping. Save engine and dependency versions, solver settings, action contract, camera calibration, checkpoint hash, hardware and seeds alongside the trace. The CPU scorer validates the log contract; it does not perform those physical checks or certify asset provenance.

8. Install and run the public package

Python 3.11 or newer is required. The protocol package uses the standard library and needs no GPU. Run these commands from a fresh checkout; replace EPISODE_ID with the printed identifier after your adapter has exported its trace.

git clone https://github.com/AstroUpLab/AstroBench.git
cd AstroBench
python3 -m venv .venv
. .venv/bin/activate
python -m pip install .
python -m unittest discover -s tests -v
astrobench plan examples/catalog.json --seed 42 --output plan.json
python -c 'import json; print(json.load(open("plan.json"))["episodes"][0]["episode_id"])'

# After producing trace.json for that episode:
astrobench score plan.json EPISODE_ID trace.json --output episode-result.json
python -c 'import json; json.dump([json.load(open("episode-result.json"))], open("results.json", "w"))'
astrobench report plan.json results.json --output report.json

The example catalog is synthetic input for testing the interface; it contains no physical assets. Replace it with a verified asset catalog for an experiment. The repository does not provide a complete environment runner or pretrained policy, so these commands do not launch robot episodes.

results.json is an array of per-episode outputs. The example wraps one result and therefore produces an incomplete report for a multi-episode plan. For a complete report, include every planned result from one policy/checkpoint. CLI outputs refuse to overwrite existing files; use a new output path for a new run.

9. Metrics and complete reporting

Report successful and completed episode counts together with the success rate and a 95% Wilson binomial interval. Report planned count, runtime errors and missing episodes separately. A normal task timeout is a completed failure; a broken execution chain is a runtime error with a null success value. Missing evidence is not a zero score.

Include time to first capture, loss after capture and terminal retention as diagnostics. If policies acquire at different times, they face different post-capture observation durations under a fixed horizon; report that duration before interpreting retention as grasp quality.

The aggregator separates suites and task IDs. A full macro score is withheld when a plan is incomplete. Publish per-task results and keep task IDs fixed so that splitting one task does not change its weight. Wilson intervals describe episode-level binomial uncertainty, not variability across independently trained models.

Retain every planned episode, failure and initialization error. Record retry reasons and links to the original attempt; do not replace failures with successful reruns. Changes to scenes, thresholds or control settings require a new version and separate results.

10. Matched cube development pilot

A frozen ACT checkpoint and an adapted state-feedback reference were evaluated at ten development positions, each with static and moving initial conditions: 20 episodes per method, 40 in total, with no retraining. Object and robot initial states were checked across all 20 pairs. The moving condition starts at 0.006 m/s along x and 0.08 rad/s about z.

This experiment uses astrobench.cube-native15hz-fixed20s/v1: 15 Hz control, a 20-second horizon and 15 consecutive native-valid samples. Native checks include target-specific opposing contacts, measured closure and geometry, force above 0.2 N per finger, relative speeds below 0.05 m/s and 1 rad/s, and drift within 0.01 m and 0.2 rad. Sample counting is not the public scorer’s elapsed-time window.

ACT acquired a grasp in 14 episodes, but 11 of those did not retain it to the horizon. All ten moving reference episodes ended at the controller’s 0.10 m tracking-workspace guard. Those are controller-specific failures, not evidence that state feedback in general cannot handle moving targets.

The reference has privileged object state, unlike the visual ACT policy. Its median acquisition time in the static condition was 19.0 s, leaving about 1.0 s of observation; ACT acquired at medians of 8.47 s (static) and 8.20 s (moving). The unequal post-capture observation durations prevent a clean ranking of intrinsic grasp stability.

The cube and positions are development data, not unseen-object tests. These results are separate from the 20-object gallery and are not a generalization leaderboard. Equal-duration retention tests and a broader frozen test split remain to be completed.

11. How to read the recordings

The gallery contains 20 object types, 40 selected examples and 80 videos: one external and one cabin view per episode. Static examples start unsupported at rest; moving examples start with translation and rotation. Recordings play at their original 30 Hz and end with about one second of frozen footage.

Expand a card to inspect its recorded motion and historical hold label. That label belongs to its original engineering judge; the frozen ending is not extra evidence of retention. Selected successful examples do not provide a success-rate denominator and are not the matched cube pilot.

Watch the approach direction and target rotation, the change in motion at first contact, and object motion relative to the closed gripper. Both views help reveal occlusion, but a video alone does not measure target-specific contact force or certify a continuous hold.

Open the episode library →

12. Reproducibility and release scope

A reproducible report needs the protocol and plan hashes, asset identities, actual reset state, policy/checkpoint identity, declared observation permissions, action and sensor timing, termination reason and complete episode accounting. Keep raw measurements so the judge can be rerun independently.

Validate the adapter with empty closure, contact with the wrong object, brief touches, insufficient hold, slip, stale sensors and reset contamination. Passing synthetic scorer tests verifies logic, not sensor bindings or physical task success. Changes in asset bytes, collision geometry, controller or scoring thresholds must be versioned.

The public repository contains the evaluation protocol, synthetic catalog, tests and usage documentation. Environment and asset redistribution require their own dependency and license checks. Demonstration collection and training internals are outside the release. The paper is a working draft; there is no arXiv identifier yet.

GitHub repository · Repository protocol reference

Related work includes Space Robotics Bench. AstroBench’s current evaluation centers on contact and retention for unsupported objects with a fixed-base arm.