IROS 2026 | DexViTac: Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous ManipulationIROS 2026｜DexViTac：面向接触密集型灵巧操作的人类多模态示范采集与学习
Everyday dexterous manipulation depends not only on motion trajectories but also on tactile cues such as contact location and pressure changes. Professor Min Li’s team presents DexViTac, a framework that collects multimodal demonstrations from natural human manipulation and learns contact-rich robot policies from them.
The paper, DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation, is published at IROS 2026. Authors: Xitong Chen, Yifeng Pan, Min Li, and Xiaotian Ding (corresponding author: Min Li).
日常灵巧操作中，机器人不仅要看运动轨迹，更依赖接触位置、压力变化等触觉信息。李敏教授团队提出面向接触密集型灵巧操作的多模态示范采集与学习框架 DexViTac，以人类自然操作为数据来源，将便携式采集系统、多模态时空对齐、触觉表征学习与机器人策略训练相结合。
相关成果以《DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation》为题发表于 IROS 2026，作者为 Xitong Chen、Yifeng Pan、Min Li 和 Xiaotian Ding，通讯作者为 Min Li。
Tactile-centric wearable hardware
The mirrored human–robot interface follows three principles: tactile-centric, human-like dexterity, and embodiment-agnostic. It combines high-density HIT LongLin-96 fingertip sensors (8×16 taxels, 0–20 N, 0.01 N sensitivity, 1000 Hz) on a motion-capture glove, IMU-based Manus Quantum Metagloves for hand kinematics, a GoPro Hero 12 with a 177° fisheye lens, and a RealSense T265 for 6-DoF pose. The same tactile sensors are mounted on the robot hand, closing the human–robot sensing gap at the hardware level.
以触觉为中心的可穿戴硬件
人—机对称的示教接口遵循三条原则：以触觉为中心、类人灵巧性、具身无关性。系统集成高密度 HIT LongLin-96 指尖触觉传感器（8×16 阵列、0–20 N 量程、0.01 N 灵敏度、1000 Hz 采样）、带 IMU 的 Manus Quantum Metagloves 动捕手套、GoPro Hero 12 + 177° 鱼眼镜头，以及 RealSense T265 六自由度位姿跟踪。机器人末端安装同款触觉传感器，从硬件层面消除人机感知差距。
Tactile-anchored synchronization
Tactile and vision streams run at 60 Hz, kinematics at 120 Hz, and pose at 200 Hz. Built on ROS2, the system buffers in hardware and aligns in software, using tactile timestamps as the global anchor so that contact and release events keep their causal order.
触觉主导的多模态时间同步
触觉/视觉 60 Hz、运动学 120 Hz、位姿 200 Hz。系统基于 ROS2，采用“硬件缓冲 + 软件对齐”两阶段机制，以触觉时间戳为全局锚点，保证接触与释放等事件的因果完整性。
Kinematics-grounded two-stage learning
After hand and arm retargeting and tactile denoising, stage one pretrains a kinematics-grounded tactile encoder against a frozen ViT-B/16 visual encoder with an InfoNCE contrastive loss, aligning vision and touch and predicting tactile evolution. Stage two plugs the pretrained encoder into ACT to generate action chunks, reducing control latency and jitter.
基于运动学约束的两阶段学习
经过手部/机械臂重定向与触觉去噪后，阶段一以冻结的 ViT-B/16 视觉编码器为语义锚点，通过运动学条件编码器融合触觉与手部姿态，并用 InfoNCE 对比损失实现视觉—触觉语义对齐与触觉演化预测；阶段二将预训练编码器接入 ACT，一次性生成动作块，缓解控制延迟与抖动。
Results
Experiments use a RealMan RM-65 arm with an Inspire RH56DFTP five-finger hand. The dataset spans more than 10 real-world environments, 40+ tasks, and 2,400+ demonstrations.
- Policy success: across pipetting, board erasing, marker insertion, and fruit collection, the full model averages 85.8%, versus 17.5% for the vision-only baseline (only 6.7% on pipetting).
- Collection efficiency: 14.5 s per demonstration on average, over 248 demonstrations per hour — far above teleoperation (84/h) and close to bare human hands (275/h).
- Tracking robustness: compared with the vision-based RTMPose under hand–object occlusion, the glove-based capture gives smooth, physically consistent trajectories without jitter or penetration.
Next, the team plans bimanual and mobile versions toward whole-body dexterous manipulation in the wild, and will open-source the hardware design, dataset, and code.
实验结果
实验平台为 RealMan RM-65 机械臂 + Inspire RH56DFTP 五指灵巧手，数据集覆盖 10 余种真实环境、40 余项任务、2400 余条示教。
- 策略成功率： 在移液、白板擦除、记号笔插入、水果收集四项任务上，完整模型平均成功率 85.8%，纯视觉基线仅 17.5%（移液仅 6.7%）。
- 采集效率： 平均每条示教 14.5 秒，每小时超过 248 条，远高于传统遥操作的 84 条，接近自然人手的 275 条。
- 跟踪鲁棒性： 在手—物遮挡场景下，相比视觉跟踪算法 RTMPose，物理动捕手套输出平滑、连续、符合物理约束的轨迹。
未来团队将拓展双臂架构与移动平台，面向真实环境中的全身级灵巧操作，并开源完整硬件设计、大规模数据集与代码。
