Hi everyone. Since there are a lot of people asking me about how to build a new hand with their own data in How could I build a new hand in unity using my own frame data. To make the step more clear, I decide to write the detailed procedures in this topic and try to make it a tutorial. I hope this topic would help more people who want to build a multi-leapmotion project or collaborative project. OK, let's multi-leapmotion together!!
My project is working in unity 2017.1.1f1, and using C# language, so the tutorial is all based on this platforms. I think Leapmotion SDK may have the same structure in other platform, if someone working on a project which is not in Unity, the tutorial may be a reference but the details may not be same.
Well, the first thing that we need to know is that all data comes from Leap Motion Controller will be firstly sent to the 'LeapProvider' script. This script contains a class with the same name, which is used as the parent class for other provider-like class that Leapmotion SDK uses. So we can build our script based on this class. The most easy way is to find a similar function script given by SDK and copy the content into our script. In this tutorial, we use LeapServiceProvider as an example.
First, we build a C# script in unity asset, which could be like:
Then, we copy the content from LeapServiceProvider to our own script, and remember to change the class name after copy. The LeapServiceProvider is like this:
It can be found in this folder:
After copy, our script becomes this:
Next step, we need to find where SDK update the frame data to unity. In the 'Unity Events' region, we can find 'Update()' function and 'FixedUpdate()' function, and here are the place where the data is transmitted into unity.
Now, we can start to insert our data into the SDK's process. In each update function, by searching the 'Dispatch', we can find a line of code named DispatchUpdateFrameEvent(). (in FixedUpdate(), it's DispatchFixedFrameEvent).
Upon the code, the parameters, 'untransformedUpdateFrame' and 'transformedUpdateFrame', are the original hand data from Leap Motion Controller, the former one is in the coordinate system of Leapmotion, the latter one is in the unity's coordinate system. What we need to do is to create a new hand and replace the original one. Here I give a example of offsetting the left hand 50 mm upward.
First, we create a new empty frame with the code
Frame myframe = new Frame();
If we look inside, the parameters of the function are:
We can input the parameters right now, or we announce an empty object first and then add the parameter into it. Here, since we don't have the hands data right now, we announce an empty object at first, We then copy ID, timestamp, and fps into our frame from the original frame.
Then, we need to build hand data. First, we announce a hand list and copy the hand data into it from the original frame. After that, we apply a transform operation to the left hand data. Just like this:
Then, we add the hand list into our frame and use the 'transformFrame()' to transform our frame into Unity coordinate system. Finally, use 'DispatchUpdateFrameEvent' to send our data into Unity process.
And there will be some bugs due to that we change the inherit relationship, we can just comment out them.
Now we have completed the script, let's see if it works. We create a new scene in unity, and add two LeapHandController prefab into the scene.
Then, we replace the 'LeapServiceProvider' script in the second prefab with our script, and set the prefabs like this:
Start the application and we can see the left hand has been offset.
Some discussions about my method:
1. Because I modified the data at the end of the Update process, so the modified result can not be optimized by Leap Motion SDK, such as interpolation. So this method may have lower robustness of interaction than the original hand. And I don't know how to insert the modified data in the beginning of the update process, I will be very grateful for someone could tell me.
2. I'm not very clear about the difference between Update and FixedUpdate. According to the comment, it seems that the FixedUpdate is used for providing more accurate interaction. In my method, after modifing the data in the Update process, I just copy it into the FixedUpdate process. So the two processes have same hand data, but I didn't see any differences with the original setting. Also, if someone knows the trick in this two processes, please tell me, thanks!
3. I'd like to know that is there any code in Leap Motion SDK used to evaluate the tracking quality? I mean the general 'health' of the hand data, like how likely the hand could be regarded as a left hand, or how confident we can assume that current tracking is good. I know there is a confidence value of gesture, but it's not enough to evaluate the tracking quality. So if someone knows the code or is working on a project about this, I will be very appreciate for sharing with me.
Well, that's all about how I build a new hand with my own data, and hopefully this tutorial is clear enough and could help you. Any questions and suggestions are welcomed. Thanks and good luck with your project.
