How could I build a new hand in Unity using my own frame data?

[fatislord]

Hi everyone, I want to build a multi-hands network system, in which I could sample my hand data on the client PC which is connected with a leapmotion, and transfer the data to the host PC through network, then rebuild the hand on host PC's unity. I have finished the data transmission work, I write a script to serialize the data and successfully deserialize them into a frame data on host PC. The data is valid. The problem is, I don't know where I could insert those data into host PC's unity. I mean how do I use my own frame data to make hand model move on host PC. I have tried this in several scripts of leapmotion SDK of unity, such as LeapServiceProvider and HandModelManager, but the results look not very good. And I don't want to write my own hand model driving script, which only could make the graphic model move without interaction function. So does anybody know how to solve my problem? Thanks.

[fatislord]

Problem solved! The key is hidden in the script of LeapServiceProvider script. In the update and fixedupdate function, there is one code called DispatchUpdateFrameEvent(). Just create a frame with your data, which would be a tedious work, and smash the frame into that code, then everything is done!

[gameshire]

Hey, could you please elaborate on how you constructed the new frame? I'm trying to bring in a new hand which is offset by some distance (other than that it mirrors existing hand). Is there a more standard way to introduce offsets or add custom motion through scripting?
For reference, I'm trying to achieve something like the Experiment #3 in this article (can be called Extendo Hands )

[fatislord]

Hi, glad to answer. I don't know the language you working with so I just use c# in unity as a case. If you announce a new frame object, just like

Frame myframe = new Frame();

you will see the parameter that building a new frame data needs, for example frameID, frame rate and handlist ( this is the keypoint ). Just build the data and assembly them together level by level, you will get a valid frame data. Then find the LeapServiceProvider script or the same script that your platform uses, and generally there will be a line like

DispatchUpdateFrameEvent()

in update and fixedupdate progress. Replace the input of the above function with your own frame data and everything is done.

In the case of offset, you can refer to a build-in function, which is called transformframe ( likely, I can't remember clearly, you can find this function being called before the script inputs the frame data into DispatchUpdateFrameEvent() ) in LeapServiceProvider. This function uses the LeapTransform class function to transform the hand data, finger data and else. And it seems the LeapTransform class provides a function to help you offset a hand data.

Recently I found Leapmotion SDK gives us a class named HandFactory, which could make hand data rebuild much more convenient.

Cheers

[gameshire]

Thanks a lot, @fatislord! Yes, I'm using C# with Unity 2018.3.4. Thanks to you, I understood where to transform the Hand data. I'll be posting the code for that in the thread that I started for this here. Now I'm working on creating a duplicate hand object.

[janah]

Thanks for this helpful Informations. I am about to create a Multiplayer Game. How did u create the new frame? Could you maybe share that part of your Code. Thanks.

[fatislord]

Hi there, it's just like creating a new object and filling all parameter into the constructor function, and you will see the parameter list when you do this.

maybe like this:

Frame myframe = new Frame(……blahblah)

Well the main work is to create the corresponding parameter. The most important part is to create the handlist, which contains the available hand data in this frame. It's kind of tedious but the class HandFactory would make works easier.

Unfortunately, the code is still under development. But I'm glad to share it after the project is finished. I will upload it onto github as soon as it's completed.

cheers

[rercVR]

Hi @fatislord,
I was wondering if you might be able to help me by elaborating a little on where exactly you put this new code. I'm just very new to Unity, C#, and Leap Motion so I'm a little lost. Could you possibly show a picture of your code where you made this modification so I know I am positioning it correctly? Also, I'm working on a project similar to this and was wondering if you could help.

I am looking to create a game in Unity (with LM) which shows the left-hand as being unaffected while the right-hand's movement is amplified. It is meant to help someone play Leap Motion games whose left hand is healthy while the right hand is impaired. So essentially I want to be able to modify the position of the virtual hand to move 1.5 times further/faster than it is actually moving in real life. I am also hoping that at that modified position, it is still able to interact with objects.

Like I said, I am very new to Unity, C#, and Leap Motion so I am unsure what code to modify to change this. If you can show me the typed code you would try, that would be very very helpful! I don't even know where to start with modifying all the different scripts. Thank you so much!

Notes:
I am not using the LM with VR headset but just in Desktop Mode. I'm willing to try any version of Unity you suggest (I have been using 5.6.1). I have also been using Leap Motion Core Assets 4.4.0 and Leap Motion Interaction Engine.

[fatislord]

Hi, I just wrote a tutorial about how to do this. Hope it will help you. Cheers and good luck!
Building a new hand with your own frame data: a tutorial