> For the complete documentation index, see [llms.txt](https://udexreal.gitbook.io/udexreal-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://udexreal.gitbook.io/udexreal-docs/robotics/usage-for-udexreal-robotics-products.md).

# Usage for UDEXREAL Robotics Products

## 1. Overview

UDEXREAL products in the field of robotics primarily fall into two major categories.

1. **UDCap DataGlove Teleoperation:** A product centered around UDCap DataGlove (UDCap DataGlove, referred to as UDEXREAL Data Glove) for teleoperation control of DexHands.

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FKVN55JgMO4eXkwGN25Rg%2F98845dd217ae73cb06480648f9d175a5.png?alt=media&amp;token=c6d25871-96eb-4a3d-a545-45257186888a" alt="" width="563"><figcaption></figcaption></figure>

2. **UDEXREAL Robotics Teleoperation System (**[**see below for details**](https://udexreal.gitbook.io/udexreal-docs/robotics/pages/sTUd3VorYOJBxvaHhaf6#id-3.-udexreal-robotics-teleoperation-system)**)**
   1. UDEXREAL Robot Headset Teleoperation Solution: Teleoperate the robot via a headset (PICO/DPVR, etc.).
   2. UDEXREAL Robot Teleoperation Simulation System: This system can connect to UDEXREAL DataGloves, half‑body/full‑body motion capture devices, and headsets to teleoperate the robot.

## 2. How to Use DataGloves to Teleoperate DexHand

{% hint style="warning" %}
To perform teleoperation control of a robot dexterous hand, the software on the human side and the machine side must work together.

**Human side:** The HandDriver software bundled with the UDEXREAL gloves, or the [VRTeleopMR App](#id-3.1.1.-udexreal-pico-teleoperation-solution) running on the head-mounted display (HMD), etc. Users use these applications to connect to the gloves, perform motion calibration, and configure the outgoing transmission of motion data.

**Machine side:** The dexterous hand control program, which incorporates calls to the dexterous hand SDK provided by the hand manufacturer. This program receives the motion data from the human side and uses it to drive the movement of the dexterous hand.
{% endhint %}

### 2.1. Introduction to Using HandDriver

HandDriver is the companion software for UDEXREAL gloves, running in a PC environment.

#### 2.1.1. HandDriver Download and Supported Operating Systems

HandDriver supports both Windows and Linux Ubuntu.

[Click here to download HandDriver](/udexreal-docs/faqs/release-notes.md)

HandDriver is available in two versions. It is recommended to use the HandDriver Web version, which adopts a B/S architecture and supports accessing the graphical interface via a browser (the graphical interface can be disabled). For detailed usage instructions, please visit:

{% content-ref url="/pages/fKXFmoz7NkCQh9Bec4B4" %}
[HandDriver Web Deployment and Usage Guide](/udexreal-docs/software-instruction/handdriver-web-deployment-and-usage-guide.md)
{% endcontent-ref %}

(<mark style="color:red;">if you need to perform firmware updates, it is advised to download the Windows standalone software version. Please note: always upgrade the glove firmware first, then upgrade the receiver</mark>).

#### 2.1.2. HandDriver User Manual

First, you need to understand how to use the DataGlove. This is the main entry of the user manual, where all information can be accessed.

{% content-ref url="/pages/gZUdWQvZDFV9PflEiJN6" %}
[UDEXREAL UDCAP DataGlove  Product Introduction](/udexreal-docs/udexreal-udcap-dataglove-product-introduction.md)
{% endcontent-ref %}

#### 2.1.3. HandDriver Quick Guide

Using the HandDriver to Connect the DataGlove → Calibration → Configuring Data Transmission. These three steps are the core procedures for using the DataGlove. Please refer to the corresponding sections in the user manual.

Calibration is crucial for action accuracy. [Please refer to the reference video in the Calibration section](/udexreal-docs/software-instruction/software-instruction/calibration-gloves-and-controller.md).

If you want to get started quickly, refer to the corresponding sections in the user manual:

{% content-ref url="/pages/hKX4WVMs8zHun4zOCyZC" %}
[Quick Guide of DataGlove and HandDriver2](/udexreal-docs/quick-guide-of-dataglove-and-handdriver2.md)
{% endcontent-ref %}

#### 2.1.4. HandDriver Supports DexHand Adaptation

Because there are significant differences between human hands and robotic DexHands, the supporting software HandDriver for the gloves needs to be adapted for DexHands.&#x20;

<mark style="color:red;">Adaptation is a core and essential task for robot teleoperation, which UDEXREAL typically provides free of charge, but it requires the DexHands to be supplied to UDEXREAL.</mark>

After the adaptation is complete, you can achieve the effect for a specific brand and model of dexterous hand in HandDriver via the "DexHand String Matching" feature.

**2.1.4.1. DexHand String Matching**

HandDriver v2.2.1 Windows and Linux Web versions have added the "DexHand String Matching" feature. After entering the designated string for a specific DexHand in HandDriver, the software will output data compatible with that DexHand model. The codes for the DexHands will be provided by UDEXREAL once cooperation is confirmed.

{% stepper %}
{% step %}
**After entering the DexHand string, click the "√" button.**

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FHmGvqy60HFCGWB7hKVMy%2F339e65c46702b0de26ef7574a4b01842.png?alt=media&amp;token=4dd0c4e3-8392-4592-8510-d22385d8a816" alt="" width="389"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
After successfully matching the DexHand string, the string will remain in the dropdown list. The system will provide corresponding prompts to indicate whether the string matching succeeded or failed.

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FtyNG5RrSOEfXlEBk316m%2Faee8e45b216c871b22c3e768c38998f7.png?alt=media&amp;token=c7b884ed-cb66-4bbd-947b-00d27ebf1bea" alt="" width="399"><figcaption></figcaption></figure>

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FZrW3cTwEEzT02mZbtOen%2Fbe37ccf299202f0a15ee6ee709cd71d3.png?alt=media&amp;token=453ca966-71cf-4627-b698-db787a3a6e84" alt="" width="473"><figcaption></figcaption></figure>

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FKGhxCXxidDd2qzM16FCs%2F4a0f3343ff8d03cccdaa3fbb30145ea9.png?alt=media&amp;token=e75c6fa8-54c6-4b75-bccc-a3d566332ac3" alt="" width="492"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

**2.1.4.2. DexHand Data Transmission and Action Display**

The action display in HandDriver uses human hand data (the "Data" page also shows human hand data), while the data sent externally is determined by the "DexHand String Matching" feature.

### 2.2. How to Operate the DexHand

#### 2.2.1. The Working Principle of DexHand Motion Control

To make the DexHand operate, a control program for the DexHand needs to be implemented. This program primarily consists of three parts:

1. **Receiving and parsing data sent by HandDriver**

After calibration with the gloves on, HandDriver sends real-time (120Hz) angle data for each joint of the human hand via UDP through its data transmission function.&#x20;

The data transmission formats are divided into two types: JSON and Protobuf, encompassing data contents such as joint angles (Euler angles), quaternions, and pose. <mark style="color:red;">It is generally recommended to use joint angles (Euler angles) to actuate the DexHands.</mark>&#x20;

The following documents provide detailed explanations of the data format and how to receive and parse the data sent by HandDriver. The SDK for this is open-source.

<a href="/udexreal-docs/glove-data-c++-python-sdk-for-robotics/json-c++-python-sdk_angle-manual.md" class="button primary medium" data-icon="memo-pad">JSON C++/Python SDK\_Angle Manual</a>(used by most customers)

<a href="/udexreal-docs/glove-data-c++-python-sdk-for-robotics/json-c++-python-sdk_quaternion-manual.md" class="button primary medium" data-icon="memo-pad">JSON C++/Python SDK\_Quaternion Manual</a>(used by very few customers)

<a href="/udexreal-docs/glove-data-c++-python-sdk-for-robotics/protobuf-c++-python-sdk-manual.md" class="button primary medium" data-icon="memo-pad">Protobuf C++/Python SDK Manual</a>(supports Euler angles, Quaternions, or Pose; Protobuf offers higher transmission efficiency, while Pose is suitable for customers whose existing technical approach uses spatial poses)

2. **Retargeting HandDriver's joint angle data to the corresponding joints of the DexHand**

Select the required data based on the joints of the DexHand and process it into a format suitable for the DexHand, such as converting to radian mapping. An example of the mapping is shown below:

| Joints of DexHand | Joints of HandDriver |
| :---------------: | :------------------: |
|      Joint 0      |     output \[20]     |
|      Joint 1      |      output \[2]     |
|      Joint 2      |      output \[1]     |
|      Joint 3      |      output \[7]     |
|      Joint 4      |      output \[6]     |
|      Joint 5      |     output \[10]     |

3. **Invoke the DexHand SDK to control the movement of the DexHand**

Send the processed joint angle data to the DexHand SDK (by calling the SDK) to perform angle control.

#### 2.2.2. Sample Dexterous Hand Control Program

UDEXREAL provides an adapted sample dexterous hand control program. This sample is a client application; for details, please refer to&#x20;

{% content-ref url="/pages/M1dr1CEyLLEqxxxmOTb8" %}
[UDEXREAL DexHand Control Program Client Guide](/udexreal-docs/robotics/udexreal-dexhand-control-program-client-guide.md)
{% endcontent-ref %}

Using the sample lets you get the dexterous hand moving, and UDEXREAL will continue to expand its adaptation support for dexterous hands across various brands and models.

{% hint style="warning" %}
In principle, the dexterous hand control program is to be developed by the customer; UDEXREAL only provides the aforementioned mapping relationship.

You may also contact the UDEXREAL sales team to purchase the sample's source code, which facilitates secondary development integrated with a robotic arm within the robot's main control system.
{% endhint %}

#### **2.2.3. Explanation of Hand Motion Spatial Pose**

The UDEXREAL DataGlove is based on flexible sensor principles, and the data acquired are joint angle data for each joint. Unlike optical, inertial, and other types of sensors, they do not provide spatial pose information.

If you wish to adopt a spatial pose data path, the following approaches are available:

1. Obtain Pose (position and rotation) data via the Protobuf SDK mentioned above. This data is converted from Euler angles, and its accuracy depends on the Euler angles.
2. Compute it yourself by superimposing the joint angles onto the initial pose data of the UDEXREAL glove model. For the initial state of the UDEXREAL glove model, please refer to:

{% content-ref url="/pages/v2hEYABXAtdco96sBinp" %}
[Hand Model and Data Specification](/udexreal-docs/glove-data-c++-python-sdk-for-robotics/hand-model-and-data-specification.md)
{% endcontent-ref %}

The final section of that document provides the position and rotation of each joint point (including fingertips), with the origin located at the midpoint of the line connecting the roots of the two gloves.

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FUUpu11Py1WAscbRGtVKG%2Fimage.png?alt=media&amp;token=e161de2f-b576-43c2-8459-186621cd2a42" alt="" width="375"><figcaption></figcaption></figure>

## 3. UDEXREAL Robotics Teleoperation System

UDEXREAL Robot Teleoperation System may involve customized development. Service process: Preliminary technical consultation --> Solution design --> Project evaluation --> Equipment procurement --> Project implementation --> Post-project technical support.

### 3.1. UDEXREAL Robot Headset Teleoperation Solution

Teleoperate the robot via headsets such as PICO and DPVR. One of the purposes of using a headset is to obtain the spatial pose of the hands relative to the body through the headset's controllers or trackers, while the hand motions are provided by UDEXREAL gloves. The joystick and buttons on the UDEXREAL gloves enable customizable teleoperation interactions.

#### 3.1.1. UDEXREAL PICO Teleoperation Solution

The PICO teleoperation solution **requires plugging the glove's Type-C receiver into the headset. Please contact UDEXREAL Business to obtain this type of receiver.**

Alternatively, the gloves can be connected to a PC, and the hand poses can be acquired through the headset's pose‑tracking devices, then both are transmitted together to the robot's main control system to achieve teleoperation.

1. **Teleoperate using UDEXREAL Robot VRTeleopMR**

{% stepper %}
{% step %}
Install the UDEXREAL VRTeleopMR App on the PICO headset, plug the glove receiver into the headset, use the App to connect and calibrate the gloves, and transmit glove motion data, joystick and button data, headset pose, controller poses, and motion tracker pose data (up to 3) through the App.
{% endstep %}

{% step %}
Develop your own robot main control system to receive the above data, and implement inverse kinematics and motion control for the robotic arm on your own.
{% endstep %}

{% step %}
In your self-developed robot main control system, integrate the dexterous hand control program mentioned in this document to achieve motion control of the dexterous hand.
{% endstep %}

{% step %}
VRTeleopMR Application Manual:

{% content-ref url="/pages/7ZFWp36Lefzjyn5NKxX2" %}
[Robot Real-time Collaborative VRTeleopMR Teleoperation System User Manual](/udexreal-docs/robotics/robot-real-time-collaborative-vrteleopmr-teleoperation-system-user-manual.md)
{% endcontent-ref %}

{% hint style="warning" icon="sack-dollar" %}
The UDEXREAL VRTeleopMR App is a paid application. When using this solution, there may be customized development work depending on the actual situation of the project; please contact UDEXREAL Business for detailed discussion.
{% endhint %}
{% endstep %}
{% endstepper %}

2. **Teleoperate using UDEXREAL GloveService**

{% stepper %}
{% step %}
Install the UDEXREAL GloveService App on the PICO headset, plug the glove receiver into the headset, and use the App to connect the gloves and perform motion calibration. The GloveService App can run in the background on the headset.
{% endstep %}

{% step %}
Develop your own teleoperation application on the headset, obtain the poses of the controllers or motion trackers, obtain the glove motion data and joystick/button data output by GloveService (<mark style="color:red;">GloveService only transmits data locally on the headset</mark>), and transmit them to the robot's main control system by yourself for teleoperation.
{% endstep %}

{% step %}
GloveService is free software. For details, please see:

{% content-ref url="/pages/8ugjFTkBaDOXQZTPDv3d" %}
[VR Headset Android Glove Service Manual](/udexreal-docs/gloves-in-vr-headsets/vr-headset-android-glove-service-manual.md)
{% endcontent-ref %}
{% endstep %}
{% endstepper %}

#### 3.1.2. UDEXREAL DPVR Teleoperation Solution

The DPVR headset is a PC VR device. Install UDEXREAL HandDriver and the DPVR teleoperation application on the PC, and receive glove data and controller poses in the robot's main control system to achieve teleoperation. For this solution, please contact our business team for details.

### 3.2. UDEXREAL Robot Teleoperation Simulation System

This system can connect to UDEXREAL DataGloves and half‑body/full‑body motion capture devices to perform half‑body or full‑body teleoperation of robots. The system can also support simulation operations on URDF models of certain robots.

<figure><img src="https://2082502898-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8s1Ia6TfFovgyJ5JsDJZ%2Fuploads%2FnP2iBnzA62Iw8ob4zYbF%2Fbf11ba57b7f9bdf07bb0fb4a6a64317d.png?alt=media&amp;token=cbc439a7-212d-4fce-a39f-bc7586240258" alt="" width="563"><figcaption></figcaption></figure>

This system is a paid customized implementation service, which varies depending on the specific robot, dexterous hand, and robotic arm. The half‑body/full‑body motion capture devices are also selected based on the customer's actual conditions (and can be combined with the UDEXREAL headset teleoperation solution). The joystick and buttons on the UDEXREAL DataGloves enable customizable teleoperation interactions.

## 4. FAQs

### 4.1. Why is there  latency with the DexHand?

From the moment the human hand starts moving to the point where the DexHand's SDK is called to initiate movement (excluding network transmission delay), the process takes approximately 20ms. The time from calling the SDK to the DexHand physically moving depends on the product from the DexHand manufacturer. This includes both software and hardware drivers, with hardware driver performance being related to the physical implementation technology of the DexHand.

Therefore, some DexHands can maintain near real-time synchronization with the human hand, while others exhibit noticeable latency. The minimum latency perceptible to an average person is around 20ms.