> For the complete documentation index, see [llms.txt](https://udexreal.gitbook.io/udexreal-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://udexreal.gitbook.io/udexreal-docs/robotics/robot-real-time-collaborative-vrteleopmr-teleoperation-system-user-manual.md). # Robot Real-time Collaborative VRTeleopMR Teleoperation System User Manual ## 1. Overview This system recommends using the PICO 4 Ultra Enterprise together with its motion trackers. The system principle architecture is as follows:
## 2. Download Released version: {% file src="/files/velQ5EgkjgGw2VCewKJN" %} Previous version (not recommended), with data transmission pause and License Key: {% file src="/files/XvHE0X8Ik0QXuy2J77wy" %} ## 3. Main Screen ### 3.1. License Key * The first time the App is opened after installation, an License Key must be entered. After entering it, click Submit. Once verified, the App can be used. The License Key is not required for subsequent use (however, it is required after uninstalling and reinstalling; it is not required for an overwrite installation). * License Keys follow the rule of one code per HMD. Please contact UDEXREAL Sales to obtain one.
### 3.2. Connection When users open the App, as long as glove and video configuration information already exists, they can connect to the gloves and video feed, and users can directly use the system. {% stepper %} {% step %} First time entering the App, status: Pending Connection (the image below is only a schematic reference, subject to the actual system screen), displayed as *No Glove, No Video*
{% endstep %} {% step %} Status: Connecting, after plugging in the receiver, a non-highlighted glove icon appears, during video connection it displays as *Video Connecting*
{% endstep %} {% step %} Status: Connected, both gloves and video are connected, gloves are highlighted and there is a remote video feed.
{% endstep %} {% endstepper %} ### 3.3. Data Transmission The data transmission switch in the Settings panel controls the data transmission status. Since v1.2.5, it has been optimized as follows: 1. When the App is opened, the data transmission switch state from the last Apply is retained. 2. The data transmission pause feature from previous versions has been removed. 3. When the Settings panel is open (including when glove and tracker devices are being calibrated), data transmission is not paused. 4. The data transmission status is displayed at the top of the main screen. {% columns %} {% column %}
Off {% endcolumn %} {% column %}
On {% endcolumn %} {% endcolumns %} ## 4. Button Operation Instructions Since v1.2.5, button operations are as follows: 1. Predefined Buttons 1. **Motion Tracker forward horizontal raise calibration complete**: the left and right joystick buttons. 2. **Settings panel invocation**: the rear button on the left and right (corresponding to controller A button). 3. **Robot teleoperation start/stop status**: the front button on the left and right (corresponding to controller B button): toggles the start/stop status. If the current status is start, pressing once changes it to stop, and vice versa. 2. Custom Buttons 1. Left and right Menu buttons: On the original glove controller 1.0, pressing the A/B buttons simultaneously triggers Menu. On the new controller 2.0, there is a separate Menu button. This button can be used to implement robot posture reset to zero (implemented by the user according to the actual requirements of the main controller).
## 5. Robot Teleoperation Status and Master Control Listener ### 5.1. Robot Teleoperation Status (Running, Stopped) 1. To facilitate users' control of teleoperation running/stopped, this App transmits the teleoperation running/stopped status externally. This status is controlled by the front button on the controller (corresponding to controller B button). Pressing it toggles the running/stopped status: when "Running", the App displays Teleop Running; when "Stopped", it displays Teleop Stopped. 2. When the App is opened, the teleoperation status is always set to "Stopped." 3. When the Settings panel is open (including when glove and tracker devices are being calibrated), the teleoperation start/stop status is set to "Stopped." 4. When the data transmission switch is turned off, the teleoperation status is automatically set to "Stopped" and cannot enter the "Running" state; when the data transmission switch is turned on, the current teleoperation status is not modified. 5. Each time the teleoperation status is set to "Stopped", FrameIndex resets to zero. {% columns %} {% column %}
Stopped {% endcolumn %} {% column %}
Running {% endcolumn %} {% endcolumns %} ### 5.2. Master Control Listener Interface 1. Since the data transmission pause feature has been removed, once data transmission is turned on, the App will continuously send data. If the master control end unexpectedly disconnects during operation, after the master control end reconnects, the robot may lose control when it resumes data from the state before disconnection. To avoid this situation, the Master Control Listener interface is provided.
2. A switch named Master Control Listener is added under Settings->General in App, and it is disabled by default. The App and the master control end work together to implement a heartbeat detection mechanism, in which the master control end sends heartbeat messages to the App. 3. The heartbeat messages are implemented by the user as needed; the App only provides the receiving interface. 4. When Master Control Listener is off, the App does not enable heartbeat detection listening, and the teleoperation Running/Stopped status is entirely controlled by the App; when it is on, the App listens for the master control end's heartbeat. If the master control end's heartbeat is not detected, the teleoperation Running/Stopped status is automatically set to "Stopped." 5. The current listening judgment interval is 3 seconds, repeating in a loop. That is, if the App side does not receive a heartbeat for 3 seconds, regardless of the current teleoperation status, the status is always set to "Stopped" and then this logic continues to repeat in a loop. The heartbeat frequency on the robot side is determined by the developer; generally, a few hertz is sufficient. 6. Interface definition: {"type": "heartbeat"}. * Master control C++ sample code {% file src="/files/x2DtN2inecPAQZIvCygJ" %} {% code overflow="wrap" %} ```cpp const std::string heartbeat = R"({"type":"heartbeat"})"; sendto(sock, heartbeat.data(), heartbeat.size(), 0, reinterpret_cast(&pico_addr), sizeof(pico_addr)); ``` {% endcode %} ## 6. Settings Settings for gloves, video, data transmission, etc. need to be configured through the settings panel. In the main screen, press the B button on the left glove to bring up the settings panel; press the button again to exit the panel. Information is only recorded and saved when explicitly selected or when the confirm button is clicked within the settings panel interface; directly exiting the panel is always treated as a cancel operation. ### 6.1. Glove In glove settings, sensor values can be displayed in real time.
Actions that can be performed on the gloves include motion calibration, controller management, tracker calibration, and setting whether the gloves are displayed with the video and whether rays are shown. #### 6.1.1. Calibration Follow the standard calibration actions of the UDEXREAL DataGlove for guidance. After the hand calibration is completed, the final step of joystick calibration is also arranged within motion calibration, allowing for one-stop completion of related calibrations. {% columns %} {% column %}
Calibration {% endcolumn %} {% column %} {% endcolumn %} {% endcolumns %} {% columns %} {% column %}
Step 1: Make a fist {% endcolumn %} {% column %}
Step 2: Extend and bring together all five fingers {% endcolumn %} {% endcolumns %} {% columns %} {% column %}
Step 3: Spread all five fingers apart {% endcolumn %} {% column %}
Step 4: Joystick calibration {% endcolumn %} {% endcolumns %} #### 6.1.2. Controller Click "Controller" in the glove settings to enter the controller management function. 1. **Controller 2.0** Supports the new glove hardware, Controller 2.0. It is basically the same as Controller 1.0 in use, but lighter. The Controller interface is as follows (while on this interface, predefined button functions are disabled and only button trial functionality is provided; after exiting this interface, predefined button functions are restored):
{% stepper %} {% step %} **Vibration Feedback Interface** 1. Basic Information * Protocol: UDP * Port: 5555 * IP Address: Local IP or assign a specific IP * Data Format: JSON 2. Vibration Feedback Protocol Format {% code overflow="wrap" %} ```json { "type": "vibration", "rate": 100, //rate:7-3000 "duration_ms": 200, //duration_ms:1-30000 "strength": 800, //strength: 600-1000 "left": { "enabled": true }, //enabled: ture or false "right": { "enabled": false } } ``` {% endcode %} 3. Example {% code overflow="wrap" %} ```json { "type": "vibration", "left": { "enabled": true, "rate": 120, "duration_ms": 300, "strength": 850 }, "right": { "enabled": true, "rate": 80, "duration_ms": 150, "strength": 700 } } ``` {% endcode %} {% endstep %} {% step %} **How to Upgrade to Controller 2.0** * Use the HandDriver Windows Standalone version bundled with the gloves to perform a firmware upgrade on the gloves. For details, please visit: {% content-ref url="/pages/ZdEZzA3UNrnjUCWcOHEn" %} [Channel & Firmware](/udexreal-docs/software-instruction/software-instruction/channel-and-firmware.md) {% endcontent-ref %} * To download and use the HandDriver Windows Standalone version, please visit: {% content-ref url="/pages/mZXVKTqkEhpFIznRkpoQ" %} [Software Instruction](/udexreal-docs/software-instruction/software-instruction.md) {% endcontent-ref %} {% endstep %} {% endstepper %} 2. **Controller 1.0**
* Press the A and B buttons on the gloves, or rotate the joystick, and the corresponding areas on the interface will highlight in blue; the joystick can also be clicked in addition to being rotated. * Click the vibration feedback button on the interface to test the vibration feedback. Click "Joystick Calibration" to enter the joystick calibration process.
Cover image
Do not move the joystick1.png
Rotate the joystick to its maximum range2.png
Calibration complete; you can recalibrate3.png
#### 6.1.3. Motion Tracker 1. **Marker Placement** Starting from v1.2.7, the marker placement has been updated as follows: the upper edge is flush with the upper edge of the Velcro, the other edge is aligned with the glove seam, and the PICO logo is oriented forward.
2. **Calibration** **This function is designed for specific scenarios.** After clicking the "Motion Tracker" button, you will enter the Motion Tracker Calibration page.
Follow the on-screen instructions and extend both arms forward horizontally. The system will then display "Calibrating...". Keep this pose and press the B button on the left glove to complete the calibration. ### 6.2. Video Settings Through cloud services, transmit video footage from the robot's perspective across regions. This feature is a custom service under development; if needed, please contact UDEXREAL sales for details. ### 6.3. Data Transmission * **Data Transmission**: There is a toggle switch. When turned on, data can be transmitted externally, and you can fill in the IP and Port (otherwise they are disabled). * Data is sent via UDP. * This setting only needs to be configured once; you do not need to click **Apply** again the next time you enter the app, unless the settings are updated. After clicking **Apply**, the settings panel will close. * **Transmission To Master Control** and **Other Target** * This application can send out glove data and headset/pose device data ([see below for details](#id-4.1.-outgoing)). The destination IP and Port for these data are entered in **Transmission To Master Control**. In a robot scenario, the destination address is usually the address of the robot's master control system. The application sends only **one** data stream externally. * **FPS** can generally be set to 90–100. If you notice the robot jittering, it might be related to the robot's inability to accept high‑frame‑rate data. You can try lowering the frame rate, but jitter is usually not caused by this—please troubleshoot on your own if needed. * The purpose of **Other Target** is: if you want to send data to another destination address, you can enter it here. As mentioned, since the application sends only one data stream externally, **Other Target** is simply a piece of information included *within* that external data stream, rather than sending duplicate data to two different addresses. In practice, after the master control system receives the data, it must forward the required data to the **Other Target** address as needed. Example use case: * The robot's arm and dexterous hand are controlled via external cabling, and they need to be connected to different computers. In this case, treat the computer connected to the arm as the master control system, and enter its address in **Transmission To Master Control**; enter the address of the computer connected to the dexterous hand in **Other Target**. The application will send \[glove data, pose data, and the **Other Target** address] together to the master control system (the arm‑connected computer). The master control system can then use the pose data for arm motion control, and also run a program to forward the glove data to the dexterous hand's computer according to the **Other Target** address, thereby controllig the dexterous hand. The dexterous hand's computer must have its own control program to receive the glove data forwarded from the master control system. * DexHand Selection: Select the compatible dexterous hand model. This setting is independent of **Other Target**. * The app currently supports sending out pose data from up to 3 independently tracked Motion Trackers. The default coordinate system is left‑handed, with +Y up, +Z forward, and units in meters (For 3 Motion Trackers, the relative pose of the two hand trackers to the body tracker will be sent). * **Do NOT use the Output Calculated Data feature**—this is only for very specific/special scenarios. * When the Settings panel is open (including when glove and tracker devices are being calibrated), data transmission is not paused, but the teleoperation Running/Stopped status is set to "Stopped". * When the data transmission switch is turned off, the teleoperation status is automatically set to "Stopped"; when the data transmission switch is turned on, the current teleoperation status is not modified.
### 6.4. General Settings for whether to display controller patterns on the screen and screen focus offset.
### 6.5. Version The version number is displayed in the lower-left corner of the Settings panel. ## 7. Data Transmission ### 7.1. Data Outbound The data transmitted by this application includes: glove motion data, joystick and button data, headset pose, controller pose, and pose data from 3 motion trackers. The transmitted data is based on Protobuf and needs to be parsed upon receipt. * Protocol Buffer Definition File {% code overflow="wrap" %} ```proto syntax = "proto3"; message Vec2 { float x = 1; float y = 2; } message Vec3 { float x = 1; float y = 2; float z = 3; } message Quat { float x = 1; float y = 2; float z = 3; float w = 4; } message TrackerPose { Vec3 position = 1; Quat rotation = 2; } message Controller{ bool menuButton = 1; bool triggerButton = 2; float triggerValue = 9; bool gripButton = 3; float gripValue = 10; bool joyButton = 4; Vec2 joyValue = 11; bool xButton = 5; bool yButton = 6; bool aButton = 7; bool bButton = 8; } message GloveHand{ bool aButton = 1; bool bButton = 2; bool menuButton = 4; bool joyButton = 3; Vec2 joyPosition = 5; Quat imu = 6; repeated float joints = 7; } message TeleopData { TrackerPose HMD = 1; TrackerPose LeftControllerPose = 2; TrackerPose RightControllerPose = 3; repeated TrackerPose Trackers = 4; Controller LeftController = 5; Controller RightController = 6; GloveHand LeftHand = 7; GloveHand RightHand = 8; int64 TimeStamp = 9; //ms level uint64 FrameIndex = 10; string MSG = 11; optional bool teleop_running = 12; //Released after v1.2.5 } ``` {% endcode %} Among them, the joints in the GloveHand data is a float array of length 23. The detailed description of the indices in the array is as follows: {% code overflow="wrap" %} ``` 0 // Thumb third joint pitch angle 1 // Thumb second joint pitch angle 2 // Thumb first joint pitch angle 3 // Thumb first joint yaw angle 4 // Index finger third joint pitch angle 5 // Index finger second joint pitch angle 6 // Index finger first joint pitch angle 7 // Index finger first joint yaw angle 8 // Middle finger third joint pitch angle 9 // Middle finger second joint pitch angle 10 // Middle finger first joint pitch angle 11 // Middle finger first joint yaw angle 12 // Ring finger third joint pitch angle 13 // Ring finger second joint pitch angle 14 // Ring finger first joint pitch angle 15 // Ring finger first joint yaw angle 16 // Little finger third joint pitch angle 17 // Little finger second joint pitch angle 18 // Little finger first joint pitch angle 19 // Little finger first joint yaw angle 20 // Thumb first joint rotation angle 21 // Index finger first joint rotation angle 22 // Little finger first joint rotation angle ``` {% endcode %} ### 7.2. Receiving Robot-side status information (text) and vibration feedback are transmitted back from the cloud. The robot side can send back text status information, and the human-side operator can view the returned text information on this App screen. **This function depends on actual requirements and should be customized and developed according to the actual situation.** ## 8. Release Notes
VersionDateContentRemarks
1.2.9Oct 28, 2026
  1. Added the adaptation of OYMotion ROH-AP002.
  2. Added the adaptation of LINKERBOT LinkerHand-O6.
1.2.82026/10/13
  1. Optimized the adaptation of BrainCo Revo2.
  2. Optimized the adaptation of TetherIA AeroHand.
1.2.72026/09/17
  1. Updated Controller terminology: changed Joystick1.0/2.0 to Controller1.0/2.0
  2. Updated Motion Tracker marker placement
  3. Fixed known bugs

1.2.62026/08/31 Released
2026/09/08 Updated
  1. Controller 2.0 vibration feedback
  2. Battery vibration alert supports Controller 2.0
  3. Tracker off - open app - turn on tracker - gloves follow
  4. Added support for TetherIA dexterous hand
  5. Optimized Other Target, with independent operation from dexterous hand adaptation
  6. Fixed: When teleoperation status is Stopped, FrameIndex does not reset to zero
  7. Supports single-hand calibration
Integration testing is not yet complete
1.2.5Skipped
  1. Optimized robot teleoperation Running/Stopped operation
  2. Added Master Control Listener Interface
  3. Optimized data transmission status
  4. Supports Controller 2.0 (vibration feedback not yet supported)
  5. Disabled predefined button functions on the Controller page; for trial use only
  6. Fixed known bugs

1.2.4
2026/08/11
  1. Adjusted the data transmission interface layout: Output Calculated Data and Other Target are now sub-items under Send to Master Control, and dexterous hand adaptation is placed at the first level, at the same level as Send to Master Control
  2. DexHand Selection now directly uses option selection instead of character matching
  3. Added support for OYMotion ROH-LiteS001
  4. Fixed the bug where data transmission was not paused when the Settings panel was invoked
  5. Implemented License key functionality
  6. Added version number
Official version naming: UDX_VRTeleopMR_vX.X.X_YYYYMMDDHHMM.apk Timestamp for this version: 202608110947