Misclassification of Ring and Middle Fingers

[austinsnider0617]

Hello All.

Our company is working on prototypes requiring XR gestures, which in turn require fairly precise hand tracking. Based on our understanding of what the LeapMotion was capable of, we have been using the LeapMotion with the Latest Orion Beta (4.0.0+52173) in Unreal Engine 4.22.1 to track individual digits and determine whether fingers of the hand were "up" (extended, but with some leeway for natural curvature) or down (not extended).

The tracking works very well in most cases; however my team has been complaining about regularly getting false detection on the ring and middle fingers (i.e. sometimes ring finger is down (curved or drooping) and FLeapFrameData claims that the middle finger is down, or vice versa -- but usually the former.) This usually happens as a result of very light occlusion of the ring finger by the middle finger, or by the hands being not directly centered inside the sensor field.

We experienced the same issue when just using the bIsExtended boolean from the FLeapDigitData in the Unreal Engine LeapComponent, and we also get the same behaviour outside of our app when using the VR Visualizer or the Diagnostic Visualizer.

These mis-detections are critically impairing the usability of these prototypes, but in all the videos and things I've seen I have been unable to determine if this is a limitation of the device or SDK or if there are any steps we could take to fix these issues. If anyone has any information that helps to resolve these tracking anomalies it would help us greatly and we'd extremely grateful. Thanks in advance.

[austinsnider0617]

It's been a week with no response. Bumping here, since I really need to know if we're the only ones encountering this problem. If anyone else has encountered this problem I would really like to know, since it seems to be a limitation of the hardware/SDK implementation, but we cannot be certain.

[AlexColgan]

Unfortunately this does sound like a limitation of the core tracking. What kind of gesture recognition approach are you using? I haven't heard of too many implementations that required ring and middle fingers to be differentiated; is it ASL?

[austinsnider0617]

@AlexColgan Not quite. We're basically just checking independent "Down" states for each finger -- whether the fingertip is lower than that finger's metacarpal; when we have our hands in certain positions (specifically if there is ANY overlap of middle/ring finger) the detection of ring finger often gets treated as a middle finger. Occasionally the middle finger will be seen as a ring finger, but more often than not it's the former problem. As demonstrated by the video of the VR visualizer, this happens completely outside of our application. I'd love to know when new improvements come online.

[yuxinl]

Same issues here! Our team is currently working on a software for physical diagnosis that we requires the patients to flex the finger which we ask them to flex. I also discovered the same problem that you described. Sometimes, it false detected the ring and middle fingers, or mismatched the virtual hands and real ones (for example, the user flexed his index finger almost 180 degrees, but the virtual finger only flexed roughly 90 degrees). It was terrible! If you have figured out how to deal with it, could you please give me some advice as well? Thanks so much in advance!