V-HOP
We fuse visual and haptic sensing to achieve accurate real-time in-hand object tracking.

The visual modality, based on FoundationPose, uses a visual encoder to process RGB-D observations (real and rendered) into feature maps, which are concatenated and refined through a ResBlock to produce visual embeddings. The haptic modality encodes a unified hand-object point cloud, derived from 9D hand \(\mathcal{P}_h\) and object \(\mathcal{P}_o\) point clouds, into a haptic embedding that captures hand-object interactions. These visual and haptic embeddings are processed by Transformer encoders to estimate 3D translation and rotation.
<div class="row justify-content-sm-center">
<div class="col-sm-8 mt-3 mt-md-0">
{% include figure.html path="assets/img/6.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
</div>
<div class="col-sm-4 mt-3 mt-md-0">
{% include figure.html path="assets/img/11.jpg" title="example image" class="img-fluid rounded z-depth-1" %}
</div>
</div>
Related Works
2024
- IROSHyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signal Through Contrastive LearningIn IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024Unfortunately, we are unable to publish the code and the dataset per the company policy.
2023
- RA-L / ICRAViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape CompletionIEEE Robotics and Automation Letters, 2023Unfortunately, we are unable to publish the code and the dataset per the company policy.
Presented at ICRA 2024 in Yokohama, Japan.
Presented at NeurIPS 2023 Workshop on Touch Processing in New Orleans, LA.