How to align Leapmotion with external webcam?

[AugmentAd]

Hi everyone,

Just got your device and am attempting to create an augmented reality application with it. More specifically, I am trying to align the Leapmotion hand model in Unity with a live video taken from a webcam. The Leapmotion is in desktop orientation and underneath the hands, while the webcam is looking down and angled approximately 45 degrees.

Having read some research papers, the leapmotion device is supposed to provide sub-millimeter precision so I think this should be doable. However, I can't seem to get the tracking and the image aligned, other than in a single position. I'm wondering if there are some best practices or advice (perhaps from someone on the Leapmotion team? I can't be the first one doing this)?

I'm using Orion and Unity 5.6, btw.

And thanks for a great device!!

[AlexColgan]

The critical piece is that the camera in Unity needs to be aligned with the physical camera in the real world. Can you send screenshots of the misalignment?

[AugmentAd]

Hi Alex,

I'll try to describe the geometry of my setup with some detail.

The leapmotion is lying flat on the desktop. The webcam is mounted on a tripod 5" (.127m) behind the leapmotion device and 18" (.4572m) above it:

View post on imgur.com

Here's the view from the webcam. I've overlaid a small rectangle (not showing up for some reason) on the bottom of the image and I've positioned the leapmotion device squarely in it. This ensures that the device is exactly in alignment with the camera and perfectly straight:

View post on imgur.com

In Unity, the position and rotation of the leapmotion device are set to zeros and the scale is 1. I figure this should be left alone and all adjustments should be made to the camera (right?).

Anyway, this is what I'm getting. I've been messing with the rotation and this is about as close as I've gotten. The fingers and watch misalign pretty frequently though.

View post on imgur.com

Any suggestions?