Tool Support Abandoned?

[electricmonks]

Hey, It seems like Tool Tracking functionality has been removed, and I just wanted to express my disappointment for that decision! We've been successfully using that in a commercial industrial prototype development project for over a year and a half and the decision to not support the Tool tracking means we will have to abandon the Leap as a development tool on this project. I'm otherwise very impressed with the Leap but this has been a huge disappointment. Would it be possible to offer it as a separate download or something? An add on package even? We just wasted a bunch of time updating the oculus runtimes on the system, the project, etc updating everything until we realized we can no longer use the Tool tracking and now have to downgrade everything to 2.3.1 and use the old Oculus libraries.

If I'm mistaken about this I would love to hear a solution but right now I'm going on the assumption that tool tracking is not a priority and we will have to find new hardware that can handle that task.

Thanks

Dylan

[SChin]

I hear what you're saying. With the Orion beta we're focused really heavily on VR hand tracking, and tool tracking isn't on the immediate roadmap.

However, you may be able to move your project forward by updating the old (2.3-compatible) Unity assets to work with the latest versions of Unity and the Oculus SDK. Since they were built around the native VR plugin in Unity, repairing any broken scripts that arise would probably be a lot faster than starting with new hardware.

[electricmonks]

Thanks, we're having issues with compatibility with the libraries. Unity crashes the second time we run each time with both the library you send us to as the downloadable 2.3.1. We'll dig in and see what we can do with it though. Thanks!

FYI This is the issue we're currently getting with 2.3.1 and oculus 1.3.* https://community.leapmotion.com/t/leapmotion-breaking-vr-mode-in-unity/5130

[electricmonks]

I am also wondering if the tools are something you have plans to support in the future? We can put it off for a little while but if its going to be dropped completely we may have to change directions. Its a shame because the tools support was really nice and seems much easier to implement then the hands would be. Thanks for the quick response though!

[SChin]

Unfortunately tool support is not on the roadmap at this time. Hope you can get 2.3.1 to work!

[electricmonks]

Thats too bad. We may have to find a new device then. It seems like it would be easy to add? Its just a simplified finger with no joints and no hand? I would just use a hand as all we really need is the location but the hand goes all crazy trying to solve for the tool. Anyways, Thats too bad but thank you for getting back to me about this,

Best

Dylan

[cameronpenner]

Is there any way I could track anything other than a hand? My users will be holding a device while using the application, and the V2 SDK isn't compatible with the Oculus 1.3 runtime. Or is there another runtime that would work? Or an open source leap SDK I could modify?

[Joe_Ward]

The V2 SDK is compatible with Oculus 1.3 -- well really, it has nothing to do with Oculus compatibility. What isn't compatible are our Unity Assets that go with the V2 SDK. And those assets are open source, so you can modify them as you please: https://github.com/leapmotion/LeapMotionCoreAssets.

[cameronpenner]

Alright, I've been working at this for a while now. I took the project at https://github.com/leapmotion/LeapMotionCoreAssets and narrowed down the problem. If I ever do var controller = new Controller(); it will end up breaking the project.

I made a new project, only loaded the "Plugins" folder found in the LeapSDK/lib/UnityAssets which only contains DLLs. Then I made a script with this in it:

using Leap;
using UnityEngine;

public class LeapTest : MonoBehaviour
{
	private void Start()
	{
		var controller = new Controller();
	}
}

The VR is still breaking.

This seems to be the simplest project I could make. I've uploaded it to a repo on github: https://github.com/cameronpenner/LeapV2VR

[Joe_Ward]

I just took the assets from here: https://github.com/leapmotion/LeapMotionCoreAssets/releases, imported them into a new project, turned on VR Support in the Unity Project Settings, and ran a few scenes. Everything worked fine.

[cameronpenner]

Thanks for the quick responses, I really appreciate the help. Could you tell me which versions of Unity and Oculus you're running? I'm still struggling to get this running with the Oculus 1.3 runtime (I've tried Unity 5.3.5 as well as the 5.4 beta). The project will usually work the first time I run it, but I'll need to close Unity and reload the project before I can run it again.

Thanks again!

[Joe_Ward]

I'm using Unity 5.4.0b19 and Oculus 1.6.0.250620 (at least that's what is listed in the oculus-runtime.json manifest).

[electricmonks]

Hi There,

Thanks for your helpful support! I've got it running somewhat but have run into a strange issue where it crashes the Oculus runtime the second time I try to run the scene in unity. It appears to lock up the Oculus runtime even when the unity instance has been closed. I saw a similar issue on this thread (a couple down, question by GregAltspaceVR and Alex from Leap is replying) https://www.reddit.com/r/oculus/comments/4cn6nn/upgrading_leap_motion_unity_projects_to_13_is/

Alex mentioned it had been fixed so I was wondering if you had any insight into how we could avoid this ourselves. We're currently running Unity 5.4.0, Oculus Runtime 1.6, Leap runtime 2.3.1 and Leap Unity assets 3.0.0 as linked above. It runs perfectly once, then appears to lock up the Oculus runtime. The goggles just display a message that its waiting for Unity even once Unity has been closed.

All we need access to is the Tool tip position. I'm not using any of the other Leap scripts aside from importing the Leap library, starting a Controllers and then grabbing the tool in a custom script. Thanks again for your assistance and I appreciate the time you've already put in to helping us deal with this issue.

Dylan

[Joe_Ward]

If that's all you need, I'd try keeping things simple and not use our Unity assets at all. You can get the necessary plugin files from the SDK directly (SDK/lib/UnityAssets/Plugins).

To track the tool tip in a script you would just need something like this:

using UnityEngine;
using System.Collections;
using Leap;

public class ToolTracker : MonoBehaviour {
    public Transform LeapSpace; //Set this transform to the virtual position and orientation of the Leap coordinate system. Set scales to .001 to convert from mm to meters
    Controller controller = new Controller();        
   	void Update () {
        Frame frame = controller.Frame();
        if(frame.IsValid){
            for(int t = 0; t < frame.Tools.Count; t++){
                Tool tool = frame.Tools[t];
                Vector tip = tool.TipPosition;
                Vector3 v3Tip = new Vector3(tip.x, tip.y, tip.z);
                Vector3 unityTip = LeapSpace.TransformPoint(v3Tip);
               // Do something with the tip
            }
        }
	}
}

(Note that I haven't tested the above code, there might be typos or other problems.)

[electricmonks]

Hi Joe,

I've made some progress following your advice but am finding a very odd bug that I wonder if it will make sense to you. I have a test case scene that I can run with VR working fine on multiple runs, no problem. However, if I call the Leap plugin (the plugin folder from 2.3.1 as described above)( just calling " using Leap; " ) the script causes the Oculus runtime and Unity to lock up. I get three warnings as well:

1. Compositor failed to initialize.
2. Bad config or LibOVR not found. VR disabled.
3. Oculus HMD not present. VRSettings.enabled set to false.

It appears to freeze up the entire Oculus runtime with a message "Sorry, Unity.exe is taking a while to load. If this issue persists, please take off your headset and check this app on your computer".

The Oculus Library screen will continue to persist. If I try to open up another VR App, eg EVE, I see a dialogue (Looks like you are in the middle of something, ..), then an App Failed to Close dialogue. EVE will start to boot but fail to run. Rebooting Oculus allows EVE to run. Unity remains locked. Rebooting unity just shows the "Please Wait" and "Sorry, Unity.exe is taking a while to close.." messages (in the goggles). This appears as soon as Unity is loaded. Unity also Froze and is status Non Responding in the Task Manager when trying to quit (inconsistently). Quitting unity removes the message in the Goggles and will display the Oculus Library as normal.

This is odd as it will often persist even after I restart Unity and/or Oculus is closed or reset. Once its locked it even breaks projects I open that don't even have the leap. If I open those other projects first they work fine. I normally have to reset my computer to get it working again.

I'm currently testing with the Oculus Runtime 1.6, Unity 5.3.6f1 (the recommended for 1.6) but have encountered with a few other versions of Unity.

I'll keep investigating and share what I find, and I'll post this in the Oculus forums as well. Thanks again for all your help so far!

Best,

Dylan

[Joe_Ward]

It is extremely odd that just including the statement "using Leap;" in a script would have any affect. Using statements essentially just tell the compiler where classes are defined. Possibly, it also causes global static objects to be initialized (I don't know), however, I don't know of anything that would even be remotely likely to cause conflicts with other libraries even if that is the case.

As a test, I tried the simplest possible project created with these steps:

1. Create a new Unity project.
2. Drag the Plugins folder from the libs/UnityAssets folder of the 2.3.1 Leap SDK into the project Assets panel (to import the Leap libraries).
3. Turn on VR Support under the Player settings.

4. Add the following script to the Main Camera:

   using UnityEngine;
   using Leap;
   
   public class ReferenceLeap : MonoBehaviour {
    Controller controller = new Controller();
    void Update () {
      Debug.Log("Frame id " + controller.Frame().Id + " # Hands: " + controller.Frame().Hands.Count + " # tools: " + controller.Frame().Tools.Count);
       }
   }

5. Turn on Tool Tracking in the Leap Motion settings panel (it is off by default).

6. Press Play in Unity.

Everything seemed to work fine and I saw no errors or warnings in the console.

I tried this with both Unity 5.3.5 and 5.4.0. I have the latest Oculus runtime installed on Windows 8.1.

[electricmonks]

Hi Joe,

Thanks for the detailed follow up. One thing I may have forgotten to mention is that it usually will work correctly the first time, then locks up after that. It actually appears to lock up when shutting down the scene. My intuition is that its not unloading properly and thats why it stays busted the second time through. I was able to reproduce the bug as listed above on a different machine (Oculus 1.6, Unity 5.3.5f1). I'm following up with Oculus as well so I'll let you know if something comes from that. Thanks again for the excellent support!

Best,

Dylan

[Joe_Ward]

The Unity editor never unloads libraries after loading them, so if somehow the memory into which the library is loaded gets stomped on by a serious bug, the library would stay in a broken state until you restarted Unity. In my tests, I did not have any problem running the scene multiple times, either.

[electricmonks]

Hi Joe,

Thanks for checking. I realized that it was running fine loading the leap library, it was the call to
Controller leapController = new Controller(); that seems to be gorking it. I'll follow up with the Oculus folks and see if they can shed some light on it. Thanks again for checking it out,My feeling is its an issue with the Oculus runtime that I just happen to be triggering somehow. I've noticed some other weird lockups like this on other devices without the leap around, just not so pronounced or specific. I'll let you know what I discover!

Best,

Dylan

[duchangyu]

Hi Dylan，

Have you ever figured it out? or did you get any clue from Oculus side? Would you please share the link if you communicate with them through forum like this?

Or do you switch to any other devices?

I have same issue with you and stucked