Pointing Gesture: Finger Count struggle

[Apidcloud]

Hello,

Since the new API detection is way better than before, every finger is detected almost the entire time. Thing is, let's say I want to check if only 1 finger is doing a "pointing gesture". What would be the best approach to detect that?

I mean, we can't use fingers count, unlike before, since it's 5 almost all the time...

Thanks

[Joe_Ward]

Pointable.isExtended is true when the finger is roughly straight. If a finger is extended in the new API, then it would have been detected in the old API; conversely, is isExtended is false, the finger would not have been detected in the old API.

If you need a more rigorous definition of "pointing" then I suppose you would have to look at bone direction alignment.

[Apidcloud]

I think that isExtended does the job, I'll be sure to post here the result later today.

I wasn't aware of that property, thanks

[yosun]

Is there a way to detect which finger is the pointable.isFinger/isExtended?

[Joe_Ward]

If you have a Pointable that is a finger, you can convert it to a finger object and then read Finger.Type.

[yosun]

Hmm other than looping through all the other fingers (to check that they are not pointing), is there a quick way to find if just the index finger is pointed?

[Joe_Ward]

Check if the extended count is == 1 and then check that the finger is the index finger.

[yosun]

Cool!

Finally - trying to get the position and direction of the index finger tip... hm = hand model... except the tip position seems to be offset?

hm[i].transform.TransformPoint(pointable.TipPosition.ToUnityScaled())
hm[i].transform.TransformDirection(pointable.Direction.ToUnityScaled())

[Joe_Ward]

All the coordinates from the Leap SDK are real-work relative to the Leap device. So finger tip positions aren't relative to the hand. If you look at the FingerModel script, you can see that the positions are transformed relative to the HandController, but there is also an offset calculated:

  Vector3 local_tip = finger_.Bone ((Bone.BoneType.TYPE_DISTAL)).NextJoint.ToUnityScaled (mirror_z_axis_);
  return controller_.transform.TransformPoint (local_tip) + offset_;

(The Unity Assets use the end of the bone, rather then the Finger TipPosition -- they are slightly different.)

To maintain consistency with the calculations done by our Unity scripts, it is probably easier to use FingerModel.GetTipPosition() instead of rewriting the calculations using the values directly from the underlying Leap API.

Also, for direction vectors use ToUnity() not ToUnityScaled(). Directions should be unit vectors and scaling them violates that assumption.

[yosun]

FingerModel.GetTipPosition

How do you access .GetTipPosition() or anything from FingerModel? Is it accessible from finger or pointables?

HandModel[] hm = hc.GetAllGraphicsHands(); 

		for(int i=0;i<hm.Length;i++){
			Hand h = hm[i].GetLeapHand(); 
			PointableList pointables = h.Pointables; 

			for(int p=0;p<pointables.Count;p++){
				Pointable pointable = pointables[p];
				Finger finger = new Finger(pointable);
               
                if (pointable.IsFinger)
                {// print (pointable.ToString());
                    if (finger.Type == Finger.FingerType.TYPE_INDEX)
                    {
                        if (pointable.IsExtended && pointables.Extended().Count == 1)
                        {

[Joe_Ward]

Remember that there are essentially two, separate APIs that you are dealing with here. One is provided by the Unity scripts in the asset package and the other is in the Leap Motion C# API proper. FingerModel is part of the Unity scripts, not part of the base C# API, so no, you can't get to it from Leap.Finger or Leap.Pointable. Instead, you get the FingerModel objects from a HandModel object.

[yosun]

ok so there isn't an equivalent of frame.GetGraphicHands() to get the finger model from a frame? (etc)

[yosun]

Also even using GetTipPosition() there still appears to be an offset?

[Joe_Ward]

How much of an offset? Which hand model are you using? The models have to be set up very carefully for the tip position to coincide exactly with the real tip position. It should be close though, like within the nail area of the finger.

[yosun]

The offset is quite a bits off - actually maybe I'm using the wrong methods - just trying to raycast hit from index finger to get what's being pointed at? Is there a more direct way than getting finger tip position and direction to cast the ray?

[Joe_Ward]

The picture above shows the position returned by FingerModel.GetTipPosition(). (I positioned a sphere at the vector returned by GetTipPosition()).

Direction is a little trickier because a finger is not always straight. The Leap API uses the intermediate bone of the finger as the finger direction, but you might want to use the distal bone instead.

[yosun]

Hmm... the Debug.DrawRay for the raycast definitely did not show the ray coming out from anywhere near the finger...

Is this true also if you scale the hand prefabs?

[Joe_Ward]

I get the same results whether the prefab or the hand controller are set to a different scale.

If I add the following code to the Update finger function in the finger script for my hand prefab (SkeletalFinger.cs in this case):

public override void UpdateFinger() {
  SetPositions();
  Ray ray = this.GetRay();
  Debug.DrawRay(ray.origin, ray.direction, Color.green, 0, true);
}

I get this result:

I'm not sure why your results are different.

GetRay() is defined in FingerModel and uses GetTipPosition and the distal bone:

 /** Returns a ray from the tip of the finger in the direction it is pointing.*/
  public Ray GetRay() {
    Ray ray = new Ray(GetTipPosition(), GetBoneDirection(NUM_BONES - 1));
    return ray;
  }

[yosun]

ok this works - earlier, I was not using UpdateFinger and FingerModel as the base class and there might have been ordering conflicts in the update loop rasterization

[atiyeh92.2]

Hello Yosun , is this code working ?