Unity Assets: AR/VR Update

[AlexColgan]

Update: The new core assets are here, and Oculus 0.7 compatible! https://developer.leapmotion.com/downloads/unity

Hey everyone,

As we posted back in June, Unity 5.1 and Oculus SDK 0.6 introduced a radically different Oculus plugin structure, including major changes to the VR cameras. These changes effectively abstracted away many of the things that we used to implement image passthrough in Unity. Since then, we’ve been in discussions with Unity about how to make sure that the assets work properly.

Tomorrow’s release of the Oculus 0.7 SDK includes breaking changes so that the runtime won’t support applications built before 0.6.0.1. For this reason, we wanted to give you a clear path for development. Here are your options, depending on what type of experience you want to build.

Augmented Reality

Any Unity project that uses image passthrough will need to use the older version of the Oculus plugin, which is only possible with SDK 0.5.0.1 (and some earlier versions). As a result, users trying the demos will need to have the 0.5 or 0.6 runtime installed. Here is the best setup:

• Unity 5.0 or below, OR Unity 5.1+ with the integrated VR support disabled
• Leap Motion SDK 2.3.0
• Oculus SDK 0.5

Virtual Reality with Rigged Hands

The VR cameras issue does not affect pure virtual reality, including our traditional rigged hands assets. Right now, developers using pure virtual reality with rigged hands can use:

• Unity 5.1+ with the integrated VR support
• Leap Motion SDK 2.3.0
• Oculus SDK 0.6.0.1

We’re excited to start digging into the Oculus 0.7 SDK and will be working to update our Core Assets support.

Virtual Reality with Image Hands

Since the Image Hands rely on passthrough, they will not work properly except with the augmented reality setup:

• Unity 5.0 or below, OR Unity 5.1+ with the integrated VR support disabled
• Leap Motion SDK 2.3.0
• Oculus SDK 0.5

Fortunately, we have a completely reimagined asset in the works that is being rebuilt from the ground up, and which we expect to release in the near future. In the meantime, we recommend that developers either build with older, stable tools, or use rigged hands as stand-ins until we launch the new Image Hand asset.

It’s also worth mentioning that since Oculus no longer supports Mac OSX, future releases of our Unity Core Assets will be Windows-only. Older releases that are OSX-compatible will remain available through our Unity downloads page.

Virtual reality is a rapidly emerging field, and the third-party engines and APIs that we use to build the next generation of hybrid experiences are changing quickly. Thanks for your understanding and support as we continue to adapt to these evolving platforms.

[AlexColgan]

Good news everyone!

Thanks to the latest Unity release, we will soon be able to provide full support for image passthrough and the upcoming Oculus 0.7 SDK. With Unity 5.1.3, we can render separately for each eye and customize the separation of the VR cameras. These features are essential to enabling augmented reality using Leap Motion passthrough, and until now were only available through the v2.1.3 Oculus Unity plugin (which is no longer supported by the latest Oculus SDKs).

The level of VR support that Unity has been providing is great news for everyone experimenting on the leading edge of this space. We’re excited that they were able to roll out support for these extra features so quickly.

Our Front End team will soon start work on the new Core Assets for Unity 5.1.3 and the upcoming Oculus 0.7 SDK. We’re targeting them to land before the 3D Jam starts on Sept. 28th. Stay tuned to this thread for the latest news.

[neohaptic]

Thanks for the info. But we are currently facing the following dilemma:

Our project has been migrated to Unity 5.1.3, using the integrated VR support, Oculus SDK 0.7.0.0, and the new Oculus Utilities for Unity 5. This works fine so far - the only problem are the latest Leap Motion Assets (v2.3.1). Since the Leap Motion Assets provide their own OVR subfolder with an older, incompatible implementation of Oculus-related prefabs and scripts, they break the project (lots of compiler errors).

I understand that you guys are working on the new Core Assets for Unity 5.1.3 / Oculus 0.7 SDK. But a target date around Sept. 28 is still a month away, and until then, we're stuck with a broken project, unless we remove all Leap Motion support.

So I was wondering: Is there a way, in the meantime, to continue using Leap Motion? We're not using any image passthrough functionality, and could even live with a desktop Leap Motion sensor (instead of HMD-mounted) for the time being. All we need is a way to recognize hands and bring them into the 3D world as rigged hands (independently of any OVR-specific components). We weren't able to find any standalone Leap Motion Unity package that does not depend on any OVR implementation. Is there anything like that, and we simply overlooked it?

Alternatively: Is there a way to import the v2.3.1 assets into Unity 5.1.3 and exclude anything related to OVR, but still get functional, stand-alone Leap Motion support? If so, what exactly needs to be excluded?

Thanks for any clarification!

[AlexColgan]

Hey neohaptic, we hear you. If you build with the Oculus 0.6.0.1 SDK and for direct mode, it should still be compatible with the Oculus 0.7 runtime. As for projects that don't involve VR, these can be found in the LeapMotion folder: https://github.com/leapmotion-examples/LeapMotionCoreAssets/tree/develop/Assets/LeapMotion

[neohaptic]

Thanks, Alex. Building with the Oculus 0.6.0.1 SDK doesn't really solve our problem (being able to test/debug in the Unity player).

We downloaded the Assets/LeapMotion folder from GitHub, but it seems like it's basically the same as importing the LeapMotionCoreAsset_2_3_1.unitypackage, and selecting only the 'LeapMotion' branch. However, this won't work, since plugins are missing. What we found to work (somewhat) is selecting exactly the following branches/files upon import:

But it's still messy, and hard to get the HandController to work properly in a VR setup. We're definitely looking forward to the new Core Assets for Unity 5.1.3.

[Joe_Ward]

You should be able to build your own LeapMotion+Oculus camera rig fairly easily. The general idea is to parent the HandController to the Center Eye Anchor, rotate it to face forward and position it so that it is in the same position in the virtual world relative to the cameras as the real Leap device is relative to your real eyes. I.e. if the Leap is mounted square to your HMD with the Z axis (in Leap terms) intersecting the midpoint of the line between your pupils, then you need to move the HandController forward by the distance that the Leap is forward of your eyes -- usually about 7 centimeters. You also should set the HandController.IsHMD property to true in the Inspector. This should work fine for VR apps, but has significant problems for AR apps that use the Leap Motion camera images.

[neohaptic]

Thanks, Joe... That was very helpful.

I ended up parenting the HandController to the CenterEyeAnchor, with the following transforms:

Position: X=0, Y=0, Z= 0.08
Rotation: X=270, Y=180, Z=0
Scale: X=1.6, Y=1.6, Z=1.6

It works pretty well so far. However, I have the impression that hand posture recognition is a lot less reliable than before (the recognition FOV/frustum seems to be very narrow, and fingers/hands/arms flip around a lot). I wonder if that's something that can be improved with settings in Unity, or if there's a different reason why the recognition quality suffers so much when just using the Leap Motion assets and rolling our own camera rig, instead of the previous Leap Motion + OVR prefabs?

[Joe_Ward]

Did you remember to check the IsHeadMounted checkbox on the HandController object?

[neohaptic]

Yes, it's checked. But now that you asked, I verified again, and tried both variants, and I'm surprised to report that it doesn't make any perceivable difference whether it's checked or not.

So that makes me wonder what exactly the effect of the IsHeadMounted property is supposed to be?

[Joe_Ward]

The IsHeadMounted setting primarily affects the initial hand recognition code and should improve the success rate in classifying a hand as right or left handed and facing the correct direction. So while hands are in view and successfully classified, there won't be much if any difference with the mode on or off.

The only other things that the LeapMotion+OVR camera rig prefabs are doing is setting the distance between the cameras and doing some image latency adjustments. Neither of these are needed for VR. I'm not sure why tracking would seem worse.

[naruse]

Hey guys,

I can second this, I've just tried what @joe_ward said (assembling a custom leap + oculus rig) and as @neohaptic says I get a not-so-reliable hand recognition when the leap is mounted unfortunately :(. Checked it with both isHeadMounted true and false flags) I wonder if this is a problem with the hands being inverted (facing downwards) (newbie here tho)

[neohaptic]

After some more experimenting, I came to the conclusion that the less reliable tracking in HMD-mounted mode is not due to differences in software, but simply due to the fact that in HMD-mounted mode, the sensor is facing the monitor, and I'm probably too close to the monitor. As mentioned here (under "Background interference: 3 rules of thumb"), the distance between sensor and hands should ideally be less than the distance between hands and monitor - which is easy to neglect when testing in front of a monitor.

When respecting that rule, tracking improves a lot.

On a different note (hoping not to sound too impatient ): @Alex, any news about the new Core Assets for Unity 5.1.3 / Oculus 0.7 SDK? I'm really looking forward to those.

[AlexColgan]

The new Core Assets are here! https://developer.leapmotion.com/downloads/unity

[neohaptic]

Awesome - thanks so much!

[neohaptic]

After a week of using the new Core Assets, we're running into a few problems - most of them related to the Leap Motion assets not playing nicely with the Oculus Utilities for Unity. I was wondering: Does it make more sense to post about it here (in what is basically an announcement thread), or open a separate topic? Or maybe there is already a thread related to bugs/problems in the Core Assets, and I didn't find it? Thanks for a suggestion.

[AlexColgan]

This thread is cool; originally it was posted because there was a thorny problem with the assets, so this seems like a good spot for support.

[Joe_Ward]

You should also post issues to the Github site for the Core Assets to prevent them from getting lost in a forum thread: https://github.com/leapmotion-examples/LeapMotionCoreAssets/issues

[Tim_B]

Is the Core Asset v3.0.0 just for Unity 5.2+ with the integrated VR support enabled
Leap Motion SDK 2.3.1+
Oculus SDK 0.7.0 for VR

Because on the download site it tells that it is compatible with Leap Motion v2.3.0 / Oculus Rift 0.6.0.1 / Unity 5.1+
but it is not working very well.
By the way you always say to use SDK 2.3.0 do I really need to downgrade the SDK?

I thought I need to took this asset because I run my project with 2.3.1+31549 / 0.6.0.1 / Unity 5.1.1P3 and I need the Image Hands which Alex posted 2 months ago isn´t that easy

[AlexColgan]

@Tim_B That appears to be a typo in the Archived Assets table, as 3.0.0 was added to the left column but the new requirements weren't added to the right column. Thanks for pointing that out -- we'll fix it.

The 3.0.0 version is built for Oculus 0.7 and includes full Image Hands support. If you can update, I strongly recommend it -- but otherwise you can still use the Core Assets v2.3.1 with Leap Motion v2.3.0 / Oculus Rift 0.6.0.1 / Unity 5.1+. As for whether you can build with Leap Motion v2.3.1, while we haven't tested it thoroughly, there likely aren't any breaking changes between the two releases. I'll see what I can find out today.

[iKriz]

I'm using v3.0.0 in Unity 5.3 beta but the image hands asset is completly botched it seems? The skin mesh renderer is really deformed. Thus using the demo doesnt work properly (how is support for 5.3 beta?)