UE5 Plugin Tutorial
1. System requirements
UE 5.0 - 5.6
Windows 7 SP1 and above
The plugin packaging only supports Windows and Android
2. Development preparation and plug-in download
First, make sure that the Udexreal HandDriver software has been successfully installed.
Download HandDriver UE plugin package, download address:
2.1. UE5.5 Issue Solution
In UE5.5, there is no hand motion under edit mode when connecting to LiveLink:
It does not affect the normal use of plugin functions.
Put the hand model into a new blueprint (the blueprint created according to the manual instructions or the blueprint in the sample project) to test or check the hand model in running mode.
3. Quick Start
3.1. Run UDEXREAL HandDriver
Run UDEXREAL HandDriver on your computer and make sure the gloves are connected and used correctly. See Software Instruction
3.1.1. Data Transmission Setting
Click the gear-shaped button in the upper right corner of HandDriver, and the system will pop up a settings panel.
In the pop-up window, select the "Data Trans" and make custom settings as needed.
Check "Data Transmission" to turn on the data transmission switch.
To transmit IMU data (only supported by gloves with IMU), select the IMU option; otherwise, IMU data will not be transmitted.
The data format options—Euler, Quater, and VMC—are mutually exclusive (choose Euler for this plugin).
Select the FPS, which supports 120Hz, 90Hz, or 60Hz. The default FPS is 120Hz.
Click the "+" icon in the target address field, then enter the IP address and port number of the receiving end to enable data forwarding (use 127.0.0.1 for local forwarding).
The default port number is 5555. Unless otherwise specified, use this default port; otherwise, modify the corresponding port number in the plugin.
After configuration, click the "Apply" button to save.
Once the data sending options are set, do not exit the software—keep it running in the background to ensure gloves data can be retrieved.
3.2. Create a new project in UE
When you start Unreal Engine (UE5), the Unreal Project Browser is automatically displayed.
Select the development category that best fits your industry. Games (Games) is selected in the following example.
Select a template for the project.
On the Project Defaults panel, you can select quality/performance levels, target platforms, whether to include beginner content, and so on.
Select where you want to store the project and name it. Click Create to complete the creation.
3.3. Install the plugin
First, copy the entire UE plug-in SDK folder to the Plugins folder of the new project. If there is no Plugins folder in the new project directory, create a new Plugins folder.
Then open the project, click on Edit -> Plugins in the editor window to open the plugin management window, search for Live Link, and make sure it is checked. (Checking the LiveLink component for the first time may require restarting the UE, follow the UE prompt to restart)
Then search for the HandDriver plug-in again in the search box and make sure it is checked.
3.4. Glove LiveLink Connection
Make sure that the glove and HandDriver are running correctly.
Expand the Window tab and click on Virtual Production --> Live Link to launch the Live Link tool.
Once opened, if you expand the Source window, you will see the Hand Driver displayed as the new source.
Expand the HandDriver Source to view the associated IP and Port settings (corresponding to the IP and Port filled in MoStreamHD, which is the IP and Port of the end where MostreamHD is located), click OK, and the corresponding devices Role name (Up to eight roles) and a green dot will appear below, indicating a successful connection to the MoStream HandDriver. If the green dot does not appear, check whether the IP and port correspond or check the network.
Import the model, make sure to check the skeleton mesh and import mesh, and drag the SkeletalMesh model into the scene.
Select the SkeletalMesh model, add the Hand component under the Actor in the Details column, then add the group LiveLinkController.
3.5. Test Demo
We have prepared a test demo in the plugin so that you can experience the effect before officially using the UDEXREAL glove for the project.
Open the project with the plugin installed correctly, and open UdexrealTest in Plugins - Map.
Click Start to run it.
The models included in the map are: UE official VR hand model, standard MetaHuman female model . The hand data parameters used by the model in the scene can be used as a reference in your project.
The orange and blue sets of hand models are the same, but they use different adjustment methods. The orange hand uses the Hand component parameters (the parameters are built into the plugin) , and the blue hand uses the overlay animation (reference method: The guide of hand skeleton adjustment of MetaHuman) . You can continue reading the documentation to learn about the two hand adjustment methods to choose the one that suits you better.
3.6. Full-body motion capture integration
Full-body model requirements: Integrate with full-body motion capture. The first step is to match the hand bone definition of the data glove with the hand bone definition of the full-body model. In order to achieve the best results, the hand bones and model of the full-body model need to meet the following requirements.
The bone axis of the wrist joint and all sub-joints below is consistent (that is, when selecting each joint, the coordinate system direction is the same).
After the local rotation values of all bones in the hand model are reset to zero, the initial shape of the hand model is four fingers together and straight.
After the local rotation values of all bones in the model hand are reset to zero, the axis of the coordinate system of all bones in the fingers (excluding the wrist joint, but including all joints of the thumb) needs to point to their sub-joints.
When the data glove is used along with the full-body motion capture system, we provide a separate animation blueprint node to integrate motion capture system data and glove data: select the SkeletalMesh skeletal mesh body, right-click to create an animation blueprint
Double-click to open the animation blueprint, import the full-body motion capture node (Livelink real-time link pose) and the data glove node Udexreal Hand Pose, then import a Blend Hand Data In Mocap node, import the motion capture data and glove data into the node, output connected to the OutputPose, and finally click compile.
The yellow warning shown in the following figure is normal, and it will not affect the plugin, so just ignore it.
Note: For those projects with complex animation blueprints, if you are unable to set the animation blueprint correctly, first it is recommended to maintain the node order of "Udexreal Hand Pose" - "BlendHand Data In Mocap (Hand)" - "OutputPose", and associate other nodes with "BlendHand Data In Mocap (Mo Cap)". If you can successfully drive hand movements, then try to adjust the animation blueprint according to the actual situation of your project. An example is shown in the figure below.
Select the SkeletalMesh model, configure the skeleton model in the blueprint as an animation blueprint driver, and select the animation blueprint just created in the animation class.
Match the hand bone name defined by the data glove with the hand bone name of the whole body model (It will be automatically matched after starting the plug-in. If the matching fails, please perform manual matching). Add the Hand component and select the Hand component, copy the bone joint names of the left and right hands of the model one by one to the configuration box corresponding to the Hand component to complete the name matching. Similarly, referring to the above method, bone joint point matching can also be established in the UE through the blueprint class, and the character model can be placed in the scene (please use streaming in the scene. You may not be able to observe motion in the blueprint class depending on different UE version).
3.7. Coordinate system calibration: Calibrate the Pitch Axis (pitch angle), Yaw Axis (yaw angle), and Roll Axis (rotation angle) of the model finger joints.
It is suggested to calibrate the pitch angle of the finger first, then the yaw angle and rotation angle;
When calibrating the pitch angle, you can first open the skeletal tree diagram of the model, change the coordinate system to the object coordinate system, then randomly select a finger joint, hold down the W key, and confirm the coordinate axis corresponding to its pitch angle (the same goes for yaw angle and rotation angle). For example, if the axial direction corresponding to the downward rotation angle of the finger joint points to the positive direction of the X axis, that is, the pitch angle X of the finger joint, select the corresponding axis in the coordinate system column: Pitch = X; if the direction of the five fingers opening outward points to the negative direction of the Y axis, then the yaw angle of the finger joint is -Y, select the corresponding axis in the coordinate system column: Yaw = -Y; then the rotation angle corresponds to the Z axis, and from the coordinate system of the finger joint, the Z axis points to the negative direction, so the rotation angle of the finger joint is -Z, select the corresponding axis in the coordinate system column: Roll = -Z.
After the calibration is completed, the hand movements can be displayed. If the joint rotation direction is incorrect, it can be repeatedly calibrated until the correct result is obtained (the values of yaw angle and rotation angle are usually different, but in some cases, they need to be consistent to achieve the correct effect, as shown in the figure below).
3.7.1. Reference video
Note: The XYZ axis in this video is represented by the following numbers (-X: -3, -Y: -2, -Z: -1, X: 0, Y: 1, Z: 2), and its principle is the same as the above.
3.8. Thumb parameter adjustment:
3.8.1. The Thumb Coefficient:
Adjust the inward bending coefficient of the thumb during real-time streaming (coefficient range 0-1) , the lower the value of the thumb root node coefficient, the smaller the degree of inward bending; the higher the value of the thumb root node coefficient, the greater the degree of inward bending.
3.8.2. Thumb root node deviation (Thumb Pitch/Yaw/Roll Offset) :
Adjust the deviation coefficient of the X, Y, and Z axes of the thumb during rotation (This deviation is actually an angle, you can drag the cursor into the input box, or directly fill in the angle value to observe the effect), During real-time streaming, you can adjust the rotation direction of the thumb by adjusting the values of the three axes to achieve the optimal effect (Note: To see the adjusted effect, you need to release the mouse after adjusting the value).
3.8.3. Thumb root node rotation coordinate system (Thumb Pitch/Yaw/Roll Axis):
After selecting Adjustment for Thumb Axis, the rotation coordinate system of the thumb Pitch, Yaw, and Roll can be individually calibrated without affecting the rotation coordinate system of other four fingers.
3.9. Parameter saving and reuse
After adjusting the parameters, the data can be saved, which is convenient for data migration and reuse. The following is a reference video of how to use it.
3.9.1. Data Saving
Click the Save Setting button.
Select a folder under Content to save.
You can see the latest saved data under the corresponding folder.
Right-click the saved data, which can be exported as a csv file for reuse among different versions of UE.
3.9.2. Use the data
When importing data in csv format, be sure to select the HandDriverStructSetting row type.
You can switch data by selecting the required data table in the HandDriverSetting attribute in the Hand component.
3.10. Drive the hand model by retargeting
If the finger bone axis does not correspond to our standard model, or if you do not want to adjust the parameters, you can use retargeting.
Finger bones are oriented along the finger axis, CORRECT
Finger bones are NOT oriented along the finger axis, WRONG. Please modify bone axis or retargeting
Find the f_med_nrw_body in the test demo of the plugin, which is the standard skeleton of metahuman, and we have completed the parameter adjustment in the hand component for it. You can directly use this as the source of retargeting driver.
You can refer to the official documentation for creating and using Ik binding: IK Rig Retargeting and runtime IK Retargeting . It should be noted that the fingers of the source model are not in a straight state. When adjusting the retargeting device, pay attention to adjusting the curvature of the finger joints to achieve the target effect
3.11. Adjustment of Thumb for VRM Model
The standard bone node of VRM model is special, and the bone axis of thumb is not along the direction of finger, so it is difficult to achieve the expected effect only by adjusting the axial parameters of thumb.
For the VRM model, you can turn on the option of UseVrmModel in the Hand component. After turning on, it will automatically adapt to the thumb bone of the VRM model, and there is no need to align the thumb separately.
3.12. Parameter for MetaHuman models (also applicable to all models)
When using the MetaHuman model, because its default initial state of five fingers are bent and open, finger parameter adjustment is required. If other models have similar situations, this parameter adjustment can also be attempted.
Select UDEXREAL_MetaHuman in HandDriverStructSetting to import preset parameters quickly.
Other (L/R) Fingers Yaw Offsets
Index: index finger yaw adjustment;
Middle: middle finger yaw adjustment;
Ring: ring finger yaw adjustment;
Pinky: pinky yaw adjustment;
The above values can be adjusted when the fingers are straight and close, that is, the distance between the fingers when the fingers are straight and close. The reference experience values are shown in the above figure (-10, 0, -12, -18), but please adjust according to the actual situation.
Overall Fingers Adjust - StraightenFinger
This option is "Straighten Finger". When enabled, it will automatically straighten the bent finger (only for MetaHuman, the effect is not guaranteed for other models). Please choose whether to enable it according to the actual situation.
Overall Fingers Adjust - (L/R) Finger Pitch Offset
This parameter applies to the situation where the finger cannot be straightened, that is, the user's actual finger is straightened, but the model is not straightened. This parameter applies to the pitch of the root joint and the middle joint of the four fingers. This plugin has enabled the "StraightenFinger" option for the MeteHuman model. Usually, there is no need to adjust this parameter.
Overall Fingers Adjust - (L/R) Finger Coefficient
This parameter is "finger spacing parameter". It is recommended to adjust it when making a fist. The reference experience value is 5, but please adjust it according to the actual situation.
Additions: The above parameter adjustments can basically satisfy the state of fingers being straight and close together and making a fist, but there may be some differences in performance due to different hand shapes, calibration accuracy, glove matching, etc.
MetaHuman experience values for hand parameters (using the left hand as an example)
After adjustment, we can obtain a set of parameters that are more suitable for the MetaHuman hand, which can be used for reference.
3.13. Controller Management
Starting from V2.1.3, HandDriver supports the glove with controllers (joystick and buttons), ONLY the plugin after V1.3.0 supports the controllers.
3.14. About multiplayer
When you need to run more than two gloves on one terminal scene, you can run the HandDriver on multiple computers to send the data to that terminal.
Send data from other computers to the unused port of the terminal.
Add a new source in LiveLink, make sure the port number is filled correctly.
At this moment, multiple devices can be seen in LiveLink.
Select the Actor in the scene, find the LiveLinkComponentController component in the Details panel, and select the device you want to drive this Actor in the Subject Repersentation drop-down panel.
3.15. Close the source when exiting the UE
When you need to exit the UE to complete the operation, please follow these steps to close the HandDriver plug-in:
Expand the Window tab and click Live Link to launch the Live Link tool.
After opening, click the delete icon after HandDriver LiveLinklocalhost to delete HandDriver LiveLink.
3.16. For cases when the LiveLink source cannot be saved after packaging
After restarting the UE, the LiveLink "source" will be cleared, and the "source" character needs to be re-added to move.To solve this problem, please follow the below steps (this solution is just a reference, if there is any problem, please feel free to contact us) :
3.16.1. Option 1: Add it yourself step by step
Make sure that the Subject Representation in the LiveLinkComponentController component of the Actor that requires hand capture has selected Device1 of the glove (if it shows None, you need to operate the glove LiveLink connection in the above step first)
Open the blueprint of the Actor that needs hand motion capture, and add nodes "Delay--Create HandDriver Live Link Source at Runtime" in turn after the Event BeginPlay
Right-click on the return value of Create HandDriver Live Link Source at Runtime to promote Handle to a variable, modify the variable name to HandDriverSource , and connect.
Enter 0.0.0.0:5555 in the variable Connect String of the Create HandDriver Live Link Source at Runtime.
After the Event End Play, add the node "RmoveSource" and connect the HandDriver Source as a parameter.
Compile and save, then it can drive the Actor's hand even if there is no HandDriver source in the Live Link list.
3.16.2. Option 2: Use the plugin’s preset object
Drop the plugin-supplied blueprint object (BP_UdexLiveLinkSource / BP_UdxLiveLinkSource) into the level. It is recommended to place only one instance per level.
Copy the code below and paste it directly inside UE, or use the method provided in the sample Pawn. When running on Windows, modify the information in the WinRole variable; on Android, keep the sample defaults.
(Subject=
(Name="AndroidService"),Role=/Script/CoreUObject.Class'"/Script/LiveLinkInterface.LiveLinkAnimationRole"')
4. FAQ
4.1. The UE has been connected, modifying the role name in HandDriver 2.1.0 will cause the UE to re-identify a new source, and the previously connected source is disconnected by default
Answer: The reason for this situation is that a cache has been generated during the previous connection. Because the identification is based on the role name, a new source has been generated.
Solution: modify the old role name that was previously bound to a new role name.
4.2. Modify the latest source at the role data source, and the role cannot be driven in real time (if you find that the model will not move according to the process, please reset the current character)
Answer: Select the role of the source that needs to be switched in the UE scene, click the icon behind the switch source to reset the source interface, and then re-perform step "f" of the glove LiveLink connection operation in this manual to give the latest source to this role.
Last updated