Unity Plugin Tutorial
1. System requirements
Unity 2021 LTS and above version
Windows 10 and above.
2. Development preparation and plug-in download
First please make sure that the Udexreal HandDriver has been successfully installed.
Download the unity plugin package:
3. Quick start
3.1. Configuring the Unity environment
3.1.1. Import HandDriver Unity plugin
Create a Unity project and import HandDriver.unitypackage into the Assets package.
3.1.2. Solution to errors reported after plugin import
Delete New tonsoft.Json.dll under HandDriver
If after importing the HandDriver plugin in Unity, the console column appears Multiple precompiled assemblies with the same name Newtonsoft.Json.dll included or the current platform. Only one assembly with the same name is allowed per platform, please fix this as follows:
Delete Newtonsoft.Json.dll in Assets/HandDriver/Plugins folder
Modify API Compatibility Level
After deleting Newtonsoft.Json.dll in the Assets/HandDriver/Plugins folder, if the console column still reports an error, please fix it according to the following methods:
File -> Bulid Settings -> PlayerSettings -> Player -> Other Settings -> Configuration -> API Compatibility Level, set it to .NET Framework, then wait for Unity to complete the compiling.
3.2. Run UDEXREAL HandDriver
Run the UDEXREAL HandDriver on your computer and make sure the gloves are connected and used correctly. Please refer to Software Instruction
3.2.1. Data Transmission Setting
Click the gear-shaped button in the upper right corner of the HandDriver, and the system will pop up a settings window.
In the pop-up window, select the "Data Trans" and make custom settings as needed.
Check "Data Transmission" to turn on the data transmission switch.
To transmit IMU data (only supported by gloves with IMU), select the IMU option; otherwise, IMU data will not be transmitted.
The data format options—Euler, Quater, and VMC—are mutually exclusive (choose Euler for this plugin).
Select the FPS, which supports 120Hz, 90Hz, or 60Hz. The default FPS is 120Hz.
Click the "+" icon in the target address field, then enter the IP address and port number of the receiving end to enable data forwarding (use 127.0.0.1 for local forwarding).
The default port number is 5555. Unless otherwise specified, use this default port; otherwise, modify the corresponding port number in the plugin.
After configuration, click the "Apply" button to save.
Once the data sending options are set, do not exit the software—keep it running in the background to ensure gloves data can be retrieved.
3.3. App Options
3.3.1. Description
Besides connecting with HandDriver, the plugin also supports receiving data by connecting to the Android Service - UDEXREAL Glove Service, and can be used in Android devices such as VR headsets after packaging for Android. When using it on Android devices, both Glove Service and the application developed based on this plugin need to be started simultaneously. After Glove Service is started, it can run in the system background.
Need Real Transform: Checked by default.
Using Network: Used for the plugin to receive data via HandDriver and postback vibration feedback.
Using Android Service: Used to receive data via Glove Service in Android devices such as VR headsets.
"Using Network" and "Using Android Service" cannot be checked simultaneously and the application scenarios need to be distinguished.
3.3.2. Vibration Feedback Method
The vibration feedback for the Android service scenario is in the HandDriver script. After checking "Using Android Service", just call the ServiceVibrationControl() method.
void ServiceVibrationControl(int VirbatorIndex = 1, int DurationSecond = 20, int Strength = 10)
Interface Function: Send vibration feedback to Android service.
Parameter Description: The parameter description of this interface is as follows:
VirbatorIndex
int
No
1
Vibrator position, 1 - below the button, 2 - below the joystick, 3 represents enabling both vibrators simultaneously
DurationSecond
int
No
20
Vibration duration coefficient (based on 10ms), vibration duration=10ms * vibration duration coefficient. The sample value represents 200ms.
Strength
int
No
10
Vibration amplitude level, with an optional range of 4 to 10 levels
3.3.3. Precautions
When receiving data by the Android service, the role name must be "AndroidService".
Remove redundant scripts: The HandDriver/Scripts/Vibration Controller.cs script in the original plugin should be deleted to avoid errors.
3.4. Driving glove in Unity
3.4.1. Run Demo Scene
Complete the glove calibration in the HandDriver, double-click the Sample scene to enter the Demo scene from Project - Assets/HandDriver/Scenes/Sample.
Click the "Run" button, and you can see the hand model driven by the glove in the Game view. At this time, you can control the glove to do the action to observe whether the motion capture is accurate.
3.4.2. Custom scenes
Create an empty scene
Create an empty scene
Place the left and right hand models in the scene , create a new GameObject, and mount the HandDriver script.
Place your own hand model in the scene and move the camera to the appropriate position for observation
Method 1: Mount a HandDriver script on each hand
This method is suitable for two independent hand models, and it is more intuitive to mount a HandDriver script under each model.
Select the model of one hand, click Add Component in the Inspector panel, enter HandDriver in the search box, and click Mount HandDriver script.
Click Hand to expand the branches. If the currently selected model is the left hand, select Left; if the currently selected model is the right hand, select Right.
Under the HandDriver script, drag and drop the corresponding bones for matching
Create a new empty GameObject and change the name to "Network".
Click Add Component under the Inspector panel, enter Network in the search box, and click Mount Network Script.
Enter the Port corresponding to the MoStream HandDriver (the default port is 5555).
Select the hand model and drag the Network object into the Network box under the HandDriver script.
Repeat the above operation with the other hand
Method 2: Mount two HandDriver scripts (controlling the left and right hands individually) on an empty GameObject at the same time.
This method is suitable for full-body skeleton models, directly creating a new game object and dragging it into the model skeleton, making the operation simpler.
For this empty GameObject, click Add Component in the Inspector panel, enter HandDriver in the search box, and click Mount HandDriver script (Please mount two HandDriver scripts to control left and right hand respectively).
Under the HandDriver script, drag and drop the corresponding bones for matching
Create a new empty GameObject and change the name to "Network".
Click Add Component under the Inspector panel, enter Network in the search box, and click Mount Network Script.
Enter the Port corresponding to the MoStream HandDriver (the default port is 5555).
Select the hand model and drag the Network object into the Network box under the HandDriver script.
Coordinate system calibration: calibrate the pitch (pitch angle), yaw (yaw angle), and roll (rotation angle) of the model finger joints.
There are six expressions, X/Y/Z/X_n/Y_n/Z_n, and "_n" means the negative direction. The "Pitch" can be corrected first by making a fist and bending the fingers to select an axial direction (pay attention to the positive and negative), such as Z_n, then opening five fingers to correct the axial direction of "Yaw", and the rest is the axial direction of "Roll".
IMU(On/Off): Whether includes IMU
Thumb parameter adjustment:
(1) Thumb Root Coefficient:
Numerical range 0-1, adjusting the inward curvature coefficient of the thumb root node during real-time streaming, the lower the value of the thumb root coefficient, the smaller the degree of inward curvature; the higher the value of the thumb root coefficient, the greater the degree of inward curvature.
(2) Thumb Root OffSet:
Adjust the deviation coefficient of the X, Y, and Z axes of the thumb during rotation. During real-time streaming, the rotation direction of the thumb can be adjusted by adjusting the values of the three axes to achieve the optimal effect (the value of the X, Y, and Z axis rotation should be between -360 ° and 360 °) .
Adjustment proposal:
During the product testing process, it was found that if the thumb root node of most models finds that the rotation direction is opposite to the actual needs, the value of the axis that needs to be reversed can be subtracted by 360° or added by 360° to ensure that the value is between -360° and 360°, and the appropriate effect can be achieved (for example: the current value of the X-axis is 250°, and it is found that the reverse rotation of the axis will be better when adjusting the parameters. At this time, 250° -360° is used to obtain the final value of -110°, and then fine-tuned; if the current value of the X-axis is -250°, -250° + 360° is used to obtain the final value of 110°). The actual adjustment of the value is subject to actual use.
You can choose center or pivot according to the actual performance in the Toggle Tool Handle Position.
3.4.3. Controller management
After HandDriver2.1.3, controller management and vibration feedback were added. The added content is located below the inspector of the HandDriver.cs on the left and right hands in the example scene.
Controller values
Corresponding from top to bottom is:
Joystick X value (range from -1~ 1 floating number), joystick Y value, A button (triggered or not), B button, joystick button, menu button (pressing button A and B at the same time).
Controller vibration
Vibration feedback needs to be sent back to the HandDriver, so you need to manually set the IP address of the HandDriver in the "Send Back IP", the default value is the local IP 127.0.0.1 at runtime.
Vibrator 1 is under the buttons and Virbrator 2 is under the joystick. "Duration" is the duration of vibration, which can be set as a floating number. The minimum detectable duration is 0.04 seconds. When the duration is lower than this value, the minimum effect of 0.04 seconds will be automatically triggered. The maximum duration is 2.5 seconds, and when it is higher than this value, the maximum effect of 2.5 seconds will be automatically triggered. "Amplitude" is the vibration intensity, which can be selected from 4-10. The larger the number, the higher the vibration intensity.
When streaming data is received, and the specified character name and correct "Send Back IP" address are filled in, clicking the "Vibrator_1 Active" button can trigger the vibration effect of the specified glove's vibrator 1. "Vibrator_2 Active" corresponds to the vibration effect of vibrator 2. The "Both Active With Vibrator_1 Parameters" button can trigger two vibrators at the same time, and the duration and amplitude follows the parameters of vibrator 1.
4. Multiplayer scenes
4.1. HandDriver2.0.5 R1 and previous versions
Currently, the HandDriver software can connect up to two pairs of gloves. When encountering scenarios with more than 2 people, it supports opening HandDriver on multiple computers, and then forwarding the data to the same Unity end point to form a scenario where multiple people use it at the same time. At this time, different ports need to be configured at the HandDriver side as sender (such as 7878), and corresponding different ports need to be configured at Unity side as the receiver. As shown in the following example, create another GameObject of the Network (Network2) and set its port correspondence to be different from other ports of 5555 (such as 7878).
Then mount to achieve multiple senders to the same receiving end point.
4.2. After HandDriver2.1.0
This version mainly upgrades the multiplayer scene, which can support up to 8 people online at the same time. The difference from previous versions is that the port number only needs to be set in one network (default port 5555), without having to re-create a network. You only need to copy the saved model and enter the corresponding character name in "Character Name" according to the character name created in HandDriver (for example: In the following image of HandDriver, the character name is [1], so you can stream into the specified model by entering the character name [1] in Character Name):
5. Note
5.1. The standard of Hand Model
If you want to replace the hand model, please follow below requirements for the hand model:
Each finger needs to have 4 joint points (3 phalanxes), otherwise it cannot be driven normally.
All sub-joints below the wrist joint have the same axial direction and the positive and negative directions are also consistent.
The initial state of the model needs to be four fingers close and straight, and there is an angle between the straight thumb and index finger. The angle degree is based on the actual model bones.
The coordinate system axis of the finger bone X or Y or Z should point to the sub-joint (in the Local coordinate system).
5.2. If the joint axis of the model wrist is incorrect, the HandDriver script can be modified(IMU not supported yet)
In the demo, the joint axis of the model wrist is: Z - pitch, X - yaw, Y - roll.
If your model wrist bone is inconsistent with ours, don't worry, double-click to open the HandDriver script under Asset/HandDriver/Scripts.
Find these two lines of code in the script.
The three values in the Rotate method (X, Y, Z) correspond to the pitch, yaw, and roll of the wrist joint. If the axis is incorrect, adjust the position and positive/negative direction of the values in the code (X, Y, Z).
5.3. Wrist joint and finger joint modification
Modifying the coordinate system deviation under the HandDriver component in Unity only affects the rotation of the model's finger joints.
Modifying the code in the HandDriver script only affects the wrist joint rotation of the model.
Last updated