Orion SDK Update

Tags: #<Tag:0x007fb6c08f05b0> #<Tag:0x007fb6c0943e90> #<Tag:0x007fb6c0943d00>

AlexColgan 2018-06-11 12:38:37 UTC #1

Hi everyone,

Today we’re excited to announce the latest major release of our Orion VR tracking software, which is now available for public beta on Windows. This is the fourth generation of our core software overall, featuring a range of feature improvements across the board:

Better finger dexterity and fidelity
Significantly smoother hand and finger tracking, with motions that look and feel more natural
Faster and more consistent hand initialization
Better hand pose stability and reliability
Improved tracking fidelity against complex backgrounds and extreme lighting
More accurate shape and scale for hands

We’ve also taken several steps to streamline the developer experience, reflecting deeper changes in our SDK over time. This includes updates to our Unity and Unreal integrations. Read more about the release and SDK updates on the blog. As always, please report any bugs you find along the way!

Thanks,
The Leap Motion team

AlexColgan 2018-06-11 12:38:52 UTC #2

kries720 2018-06-12 02:03:11 UTC #3

This is amazing! Thanks for the update.
I was working on my research presentation for next week about interaction in VR and currently preparing for the demo, and the timing for this release couldn't be better.
Just a quick question, with Unity recently release the Lightweight Rendering Pipeline and Shader Graph, do you reckon it would work with the Leapmotion SDK?

AlexColgan 2018-06-12 15:43:33 UTC #4

That's awesome Not sure about compatibility there but there's one way to find out!

olivergb 2018-06-12 17:06:38 UTC #5

Hello Leap team!

I'm excited about the updated sdk, and the new simplified setup seems cool so far.
One question regarding scaling the tracking space/hands:

The way I've been able to do this before without getting deformed hand models (skinny fingers or super tiny hands) was to set the scale of the hand objects themselves to the same scale as the service provider/hand controller object. This still works for the capsule hands in 4.4.0 but not the LoPoly Rigged Hands which I'd rather use! I get extra long fingers like so:

So I'm wondering: is there a more appropriate way to scale the leap components to avoid this?
Thanks a bunch for any info!

P.S. the driver download link at https://www.leapmotion.com/setup/desktop/windows still points to the previous version, maybe y'all are waiting to roll the new one out but it certainly caused some confusion for me as the old runtime still claims to be the latest version!

AlexColgan 2018-06-12 17:21:38 UTC #6

Can you advise why you're tweaking the scale? If it's to solve a problem there may be a more optimal approach.

Just updated the Setup page since there haven't been any catastrophic bugs reported with the beta yet

olivergb 2018-06-12 17:40:46 UTC #7

Thanks for the quick reply Alex,

I'm tweaking the scale simply because I need the hands to be bigger in the scene. If the optimal approach would be reworking the whole project to work at a smaller scale, that's not really viable at this point for a couple of reasons and I'll have to stick with the previous version. I'd think 3d scaling is expected functionality for assets in a 3d engine!

AlexColgan 2018-06-12 17:56:52 UTC #8

That's fair! I'll see if we can reproduce here and get back to you.

nbenson 2018-06-12 18:54:48 UTC #10

Hey olivergb, I'll be looking into the scale trouble. We added a change that improves fingertip accuracy at world-scale, but it looks like it could be buggy at non-world scales.

VR and AR have real humans involved, so we build everything at world-scale. But if you're stick in an old application context that wasn't built to scale, I get how you're gonna want to scale your rig.

Here's the ticket I set up for this. Do you have a precise set of repro steps or an example project you could share? It would speed things up a lot. Please share them in the ticket thread if you've got 'em.

nbenson 2018-06-12 19:01:56 UTC #11

One workaround you might use in the mean time is a cleaner set of capsule hands that may be more aesthetically-appropriate for your application than the default capsule hands. We used a modified capsule hand in our Paint application:

github.com

leapmotion/Paint/blob/master/Assets/LeapPaint%20(v3)/Scripts/Paint_CapsuleHand.cs

/******************************************************************************
 * Copyright (C) Leap Motion, Inc. 2011-2018.                                 *
 * Leap Motion proprietary and confidential.                                  *
 *                                                                            *
 * Use subject to the terms of the Leap Motion SDK Agreement available at     *
 * https://developer.leapmotion.com/sdk_agreement, or another agreement       *
 * between Leap Motion and you, your company or other organization.           *
 ******************************************************************************/

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using Leap.Unity.Attributes;
using UnityEngine.Serialization;

namespace Leap.Unity {

  /** A basic Leap hand model constructed dynamically vs. using pre-existing geometry*/

  public class Paint_CapsuleHand : HandModelBase {

This file has been truncated. show original

nbenson 2018-06-12 21:41:59 UTC #12

olivergb, the fix PR for scaled rigs is here. We'll get it reviewed and released after QA, but check the PR diff and feel free to apply it to your local copy of RiggedFinger.cs to get your Rigged Hands working in your project right away.

herbst 2018-06-13 13:53:44 UTC #13

Thanks for the update! However, it seems that you optimised for specific cases (the typical first-user thing, wiggling your fingers while looking at the inside of your palms) and on the way broke some other cases that we are relying heavily on.

We spend some time today rolling back and forth between v3 and v4, and our findings are that:
* v4 looks nicer for cases where you actively look at your hand
* v4 more aggressively removes overlapping hands (in v3, clapping your hands works pretty reliably, in v4 it basically doesn't work at all)
* v4 initialization is faster for many common poses, but slower for extreme poses
* v4 has more issues with extreme poses such as doing a "karate move" with the edge of your hand moving down from the side of your head - fingers start curling in weird ways
* v4 robust mode is predicting movements so much that sometimes hands just fly away in very weird ways - try punching fast like a boxer, your hand will fly away for a while and wiggle back
* both robust and non-robust mode in v4 have more of these issues than v3, with the robust mode of v4 being especially bad at them

We documented these findings in this playlist here (comarison using the Visualizer, first with v3, then uninstalling, then with v4, with identical lighting conditions):

(note that there are four videos in this playlist - robust mode and non-robust mode for both v3 and v4)

For our application, that uses clapping and holding your hand at specific angles (to create sections of stuff) we had to roll back to v3. People using v4 will have a much worse experience than people using v3. Any ideas?

kries720 2018-06-14 15:37:06 UTC #14

Just update from testing v4 with Unity Lightweight render pipeline. With normal render pipeline, Leapmotion works alright.
However, with Lightweight Render, the hands just disappear whenever I hit play or more likely the hand models fly off to non-existent. Even though on Interaction manager, it still track my hands.

And I just found out the solution is to disable Temporal Warping Mode in Main Camera under Leap Rig and it should works!
P/s: Sorry for my bad English

AlexColgan 2018-06-15 14:16:38 UTC #15

@herbst Good feedback here! Really thankful that you've put the tracking through its paces in a rigorous way; I would expect nothing less from a 3D Jam double-hitter. Some thoughts below:

Framerate: Not sure what’s happening here, but your data framerate is in the 40s. This should be around 110fps and will have a large impact on the tracking experience. Do you have it hooked up to the Vive USB port?

Robust Mode: There is a very narrow set of circumstances where robust mode performs better than non-robust mode. For this reason, we've disabled robust mode by default, and anticipate that tracking quality will be degraded with it on. We left the option to enable robust mode in the control panel for safety purposes.

Two Hands: Two-handed interactions have become slightly worse in some circumstances. This is definitely a focus for an upcoming release.

Extreme Poses: We know that tracking isn’t perfect yet, and we’ve found that users prefer clean and consistent failures. For some of the extreme poses that lose tracking in V4, it is better for the hand to not be tracked at all than to be tracked poorly.

V3 Rollback: That's understandable, and to some extent the interaction design of any application has to be built with the strengths and limitations of a particular tracking paradigm in mind. Given that you're trying to make sections, Cat Explorer may be be a good reference point.

herbst 2018-06-18 10:51:47 UTC #16

Hi Alex,

thanks for your hints. Here's some more remarks.

Framerate: for us, the framerate was stuck at 40-55fps no matter whether USB is running through the Vive or directly into the PC. Tried different cables, different ports, different lighting conditions, and different Leap Motions. Only after uninstalling and reinstalling the whole Leap software stack, we can get 115fps both for Vive and direct USB ports, and it sometimes (without clear reasons) it goes down to 55 or even 22-25.

Any way to control the behaviour of when Leap goes into a lower framerate better? Sometimes, the framerate seems to get "stuck" at a lower rate and never gets up again, even if lighting is pretty dark. Leap has to be restarted to get to a higher framerate again...

Robust Mode: we'll leave it off for now, but so far it was necessary to have it on to get somewhat reliable performance for trade shows etc., where lighting conditions can vary greatly. Can you explain a bit more what this actually does? More interpolation/filtering for sure, but what else?

Two Hands: any eta on improvements in that area? Is "upcoming release" more like weeks to fix these blockers, or more like months/a year?

Extreme Poses: I strongly believe this should be up to the application developer. Any way to control that behaviour? Otherwise we'll have to "hack" it on top of the existing software implementation (just keep a hand displayed in the last known position if it's not found anymore), but I'd very much like to avoid that and tweak some numbers (even if it's registry changes or whatever).

AlexColgan 2018-06-21 16:48:52 UTC #17

@herbst That framerate issue is really strange. Can you try reproducing the behavior and then use the log locator (download here: https://warehouse.leapmotion.com/apps/4630/download)? We'd like to see if we can identify the cause.

Merglasch 2018-06-23 14:00:45 UTC #18

If you mean the data framerate shown in the visualizer, well I also only get around 43 there. In another thread I wrote about jitter/shake problems I have, might that be connected?
Tried with different leap motion devices and several recalibration attempts with values around 95. Also tried reinstalling v4 completely.
Always get a data framerate of only around 43

EDIT: I ran the log locator and then started the unity application while it was still running. Hopefully that was correct. Here is a link to an archive containing the logs.

AlexColgan 2018-06-25 12:48:28 UTC #19

@Merglasch Sounds like it could be USB bandwidth limitations. Have you tried switching different USB ports, or running the controller and nothing else?

Merglasch 2018-06-25 17:56:30 UTC #20

@AlexColgan Yes I've tried different ports, shorter and longer usb cables and running just the leapmotion or several devices. Also tried a different Leapmotion. None of that had a noticable impact on the data framerate for me.

AlexColgan 2018-06-25 19:46:25 UTC #21

It looks like you have low resource mode enabled. Can you disable it? (It's in the Leap Motion Control Panel, under the Troubleshooting tab.)