HoloLens and LeapMotion Hand-Alignment

[MaX_van_HeLL]

Hi there,

I am using the HoloLens and the LeapMotion together for a research project. Currently I am aiming for aligning the virtual LeapMotion hand skeletons with my real hands. The LeapMotion is mounted ontop of the HoloLens. Thus I have to consider a Y-Translation aswell as a Z-Translation to align the LeapMotion sensors with my biological eyes. However, only applying this seems not to be sufficient, the hands are still off and I guess this is related to the different baseline of the LeapMotion cameras and my eyes. The mismatch of the hands is different depending on the Position within the FOV.

I have already read through the article about the alignment problem [1] where the challenges between physical-, virtual- and biological cameras are pointed out. As far as I understood, the suggested solution here was to match the VCS with the ICD(40mm).

However, since I am using the HoloLens, my real hands are measured with my biological eyes (IPD ~65mm) which means the solution here does not fit in my case. So what I would need is a way to solve the alignment problem between IPD and ICD or maybe I am overseeing something else.

Currently I am using the LMHeadMountedRig and the Leap_Hands_Demo_VR as a template. The Y and Z Translation is applied in the LeapServiceProvider::transformFrame() by me.

Maybe I am overseeing something else, so I am thankful for any hints.

Best wishes!

[1] Alignment Problem: http://blog.leapmotion.com/alignment-problem-position-cameras-augmented-reality/

[Update 1]
I think LeapVRCameraControl::_overrideEyePosition seems to be intended exactly for that alignment purpose and I will now try to apply the correct shift for the HoloLens here.

[Update 2]
Actually I think overriding the eye Position only makes sense if the real world is measured throug the AR LeapMotion cameras. Should the LMHeadMountedRig already fully handle real-world scale? However I seem not to be able to transform the virtual Skeletons perfectly onto my real Hands using any Translation without any drift or scale problems. I am definitely overlooking something here.

[Update 3]
I could reach a really nice alignment of the LM Skeletons with my real Hands. One Thing I did was applying specific offsets along all 3 Axes within the LeapSpace Unity object to count for the physical mount. However, since the HoloLens SDK does the Stereo Rendering by using the IPD value from within the Device Portal (~58,2mm] I applied an X-Axis offset of 18,2mm to the left eyes' view Matrix. The results are very nice so the Skeletons match my real Hands in Position, Rotation and Scale. Interacting with virtual objects feels very natural and collisions appear exactly when I expect them to happen (while disabling the graphical LM Skeleton meshes). Only in a very near viewing frustum the alignment gets off around 1cm (worst case) which is anyway totally out of the expected interaction area. In all other cases / distances the alignment is very satisfying.

[Resod]

Raising my hand here as I'm working on a similar project with Hololens and Leap Motion and will soon get on that problem.
My needs are however slightly different as I need to match a V2.3.1 tool tip with it's real position.


Anyway I'm really interested in the solutions to solve this problem so please keep updating your progress.
I can also show here some things I might find interesting while working on my project, if that's OK.


As for now, you may already have seen it, but here's a guy which worked on the same problem in the last summer.
https://forums.hololens.com/discussion/1762/hololens-with-leap-motion-vuforia
His source code is currently available and he seems to get a quite nice placement (if the mixed reality capture is not deceivingly altering the result).
I have not looked at his approach in details for now, but it might bring some answers.

[MaX_van_HeLL]

Hi,
I was trying out yesterday another approach pointed out in my
"[Update]" section. I already got alot better alignment. So instead of
transforming the skeletons I tried to override the eye position and
it worked way better, already seems quite nice actually.

However its not yet perfect, so I am still searching for very good
parameters, but I am pretty sure this is the right approach (overriding
the eye position). Thanks for the link, I saw it already but didnt
checked out the code so far.
Of course I am very happy in collaborating here, so I will definitely
keep it updated. And when you reach this point, feel free to contact me
again so we might can share parameters or configurations on that
Thank you for contacting me, am very happy collaborating here! Wish you a nice weekend!
Best,

• Markus

[Resod]

Well that already shows some flaws in my approach

For now I'm still on the network side of things and made a java UDP server on the PC that processes the leap motion data with the SDK and sends only the final positions to the Hololens.
To avoid latency I send only the tool infos and I thought too that only a few transformations would make the position fit.
As it don't seems to work that way and there's no option to override eye position in the SDK documentation,
I guess the only way to stream leap motion data to the Hololens is with the websocket service ?


I mean is latency a problem that requires optimizations to provide a good experience ?


For now I will push my solution to the end anyway as it just needs a few hours to fully work and will share the results when it's done (even if it seems doomed to fail).
Nice weekend to you too

[MaX_van_HeLL]

There is actually another way of getting the HoloLens and LeapMotion work together, in case it is not required to deploy a UWP build at least. Microsoft worked on a tool called "Holographic Remoting Player" [1] available at the Microsoft store which lets you bypass the UWP deployment entirely for development purposes. It is also a Unity plugin, which Transports all the computation to the development Computer but you can use all the data (RayCasting, Spatial Mapping, HeadTracking etc.) from the Hololens still. The rendered frames get automatically sended in stereo to the HoloLens (VR mode), as you would have deployed a UWP build actually. I am using this and it works like a charm. Using this makes it possible to use the whole native API from LeapMotion of course. So depending on the aspect if your aim is more research- or production oriented, this might could be interesting for you aswell

I will look into the overrideEyePosition any further. It Looks good when the skeletons are infront of the HMD like in a "VR mode" or like in the Video you showed. However they get off when a view angle is applied - like interacting on AR objects located at a real table where you have to look downwards.

so far, cheers

[1] Holographic Remoting Player: https://www.microsoft.com/en-us/store/p/holographic-remoting-player/9nblggh4sv40

[Resod]

I was soo used to deploy my app to UWP that I totally forgot that option

I might give it a try soon since the client part of my app is becoming quite a nightmare with the .Net/Unity.
UWP needs to totally rewrite the network part of a .Net app.

But Unity doesn't seems to handle it correctly since my app works on a basic UWP project but get some unexpected behaviors/crashes while built through the Unity pipeline.

I'm guessing that's more my implementation of the UWP client that have some flaws I'm quite slow to find as the UWP architecture is brand new(to me at least).

In the meantime that's a really good solution thanks !