DexJoCo-X: Benchmarking Action Representations for Multi-Hand Dexterous Manipulation

Xiangwei Jiang; Yao Mu; Lixin Duan; Wen Li

MULTI-HAND DEXTEROUS MANIPULATION

DexJoCo-X

Benchmarking Action Representations
for Multi-Hand Dexterous Manipulation

Seven different hands. Shared manipulation tasks.
A common foundation for studying how dexterous skills transfer across morphologies.

Paper PDF arXiv soon Code soon Dataset soon Video

7dexterous hands

6manipulation tasks

2,100demonstrations

42hand–task pairs

DexJoCo-X overview: seven hand morphologies, a demonstration collection pipeline, and matched six-task policy evaluation. — A shared benchmark connecting diverse hands, balanced demonstrations, and consistent evaluation.

THE RESEARCH QUESTION

How do we learn across different hands?

Robot hands differ in their kinematics, actuation, and control. DexJoCo-X makes it possible to compare action representations using the same tasks, demonstrations, scenes, and success criteria.

Our study examines how action coordinates interact with pretraining and model architecture. We compare Native coordinates, function-aligned slots (FAAS), and learned cross-hand latents (DexLatent), alongside per-hand π_0.5 (Ego-Pi) and joint seven-hand Being-H0.5 policies.

A matched benchmark

Common scenes, success rules, and execution interfaces across all 42 hand–task pairs.

A balanced dataset

50 demonstrations per pair, with synchronized RGB, robot states, actions, and masks.

Reusable evaluation

Connect your own policy to the same simulator and compare representations under consistent conditions.

AllegroInspireLEAPLinkerHandSharpa WaveWujiXHand

SEE IT IN ACTION

One task, seven embodiments.

Watch the benchmark overview and recorded simulation demonstrations across all seven hands.

Recorded demonstrations in simulation · 2 min 40 sec

SIX COMPLEMENTARY TASKS

From tool use to bimanual coordination.

SINGLE ARM

Bucket lifting · Nail hammering · Tower of Hanoi

BIMANUAL

Microwave cooking · iPad unlocking · Photography

Six illustrated task sequences, including bucket lifting, hammering, Hanoi, microwave cooking, iPad unlocking, and photography, with task success conditions. — Task-specific success rules cover grasp stability, tool use, sequencing, and coordinated manipulation.

Consistent evaluation

Three independent runs of 50 rollouts per hand–task pair. Every pair receives equal weight in the full-grid macro-average.

DEMONSTRATION COLLECTION

From human motion to balanced data.

Seven glove-to-hand mappings support demonstration recording. Automated scene expansion and verification produce the balanced training dataset.

Cartoon diagram of glove and tracker capture, mappings to seven hands, and automated demonstration expansion and verification. — Reviewed source demonstrations are expanded, replayed, verified, and exported with synchronized observations and commands.

ACTION REPRESENTATIONS

Shared skills, morphology-specific control.

Native, FAAS, and DexLatent share arm control and execution conditions. Their hand-action representations are decoded into each embodiment's native commands.

Native coordinates, function-aligned slots and latent encoders, with Ego-Pi interleaved prediction and Being-H0.5 policy inference. — Representation, pretraining, and architecture jointly shape multi-hand learning.

BUILD ON DEXJOCO-X

Paper, data, and evaluation.

READ

The paper

The benchmark design, action representations, and multi-hand learning study.

Read PDF

TRAIN

The dataset

2,100 demonstrations in six task packages, with native tensors, RGB videos, and checksums.

Release in preparation

EVALUATE

The code

Simulation environments, evaluation interfaces, data conversion, and a π_0.5 inference example.