Manipulate yaw of the leap motion hand based on measured sensor data

[Niraya]

Hello

Currently I am working on my master’s thesis, which is about developing a sensor equipped smart glove for industry (only scientific purpose for university, no commercial use). Therefore I want to set up a 3D visualization of my smart glove with Leap Motion using Unity3D and C#.

I managed to manipulate the hand in such a way that I can visualize the finger bending of my glove. But for the rotation of the hand, especially for the yaw, a problem occurs:

What is happening with the bottom right joint?

(image source: http://www.kolor.com/wiki-en/images-en/2/27/Leapmotion_utilization_3.png)

When doing the yaw rotation, I manipulate all the joints and vectors, which I can access through the frames hand object, by using a rotation Quaternion which performs a rotation around a specified rotation axis (in this case the palm position) with a certain angle (measured by an orientation sensor on the glove).

No matter what I am trying, I am somehow not able to access the joint on the right bottom of the hand (see above picture). It seems to be calculated relative to some other values?!

Can somebody tell me how the position of this joint is calculated?

Is there any possibility to access this joint like the other joints? I thought this has to be the previous joint of bone zero of the pinky, but this seems not to be the case, since manipulating this joint, doesn’t change anything.

What am I missing?

Thank you!

Patricia

[Joe_Ward]

The corner of the hand is calculated in the script CapsuleHand.cs based on the other bones reported by the API.

[Niraya]

Hey Joe_Ward,

Thanks for your reply

So the important lines are the following:

CapsuleHand.cs lines 169-172

Transform thumbBase =_jointSpheres[THUMB_BASE_INDEX];

Vector3 thumbBaseToPalm = thumbBase.position - hand_.PalmPosition.ToVector3();

mockThumbJointSphere.position = hand_.PalmPosition.ToVector3() +Vector3.Reflect(thumbBaseToPalm,hand_.Basis.xBasis.ToVector3());

The mockThumbJointSphere.position is the position which gets messed up. This is calculated by calculating the vector from palm position to thumb base, and reflecting it by a plane which goes through the x-basis of the hand.

The basis of the hand is a LeapTansform. The documentation about LeapTransform is not very detailed
(https://developer.leapmotion.com/documentation/csharpapiLeap.LeapTransform.html#csharpstruct_leap_1_1_leap_transform). Thats why I am not sure if I am getting it or if I am missing someting. How is the LeapTransform calculated? Where does for example the rotation come from? the basis coordinates? Where exactly is this xBasis of the hand?

Since I am rotating the palm position and also bone 0 of the thumb, I assume that my problem is caused by the hand.Basis value. I can not set the transforms rotation manually, I dont see a way to manipulate/rotate the basis with the rotation angle I get from my sensors.

Do you have any hint?

Thank you very much!

Best regards,
Patricia

[Joe_Ward]

This is calculated by calculating the vector from palm position to thumb base, and reflecting it by a plane which goes through the x-basis of the hand.

The vector is a displacement vector so essentially, you are moving the thumb base to the origin. Unity's Reflect function reflects across the plane defined by the normal specified by the second parameter -- so it is reflecting across the plane formed by the y and z basis vectors -- i.e along the x basis vector. After the reflection, the code then adds the hand position back so you get the offset of the mock thumb to the opposite side.

How is the LeapTransform calculated? Where does for example the rotation come from? the basis coordinates? Where exactly is this xBasis of the hand?

The transform is derived from the orientation of the hand (or whatever bone it represents). The rotation is a quaternion that represents the rotation from the coordinate system basis to the hand's basis. The basis is an orthonormal set of vectors -- essentially the local coordinate system of the hand. The x basis runs laterally across the hand, the y basis is normal to the hand, and the z basis runs forward. Since these are direction vectors, they don't have a position.

Possibly, you could apply your rotation to the xBasis vector and use the result for the reflection.

Otherwise, it might be easiest to rewrite the capsule hand script to use the base of the pinky metacarpal bone instead of the mock thumb joint. I suspect the positions won't be that different.

Alternatively, you could calculate the quaternion for the rotation you want to appy, use it to create a LeapTransform object, and apply that to the hand data before Capsule hand updates itself with the Hand.TransformedCopy() function. (You can probably do this in CapsuleHand.SetLeapHand(), but I haven't tested it.)

[Niraya]

Thank you very much! It works

I am no longer using the mock thumb joint, instead I am using the prevjoint of the pinkys metacarpal bone and shift it a little bit in the negative bone direction, so that it looks more natural It looks not 100% perfect, but well enough for my purpose

Thank you!