Retroreflective Marker Tracking Script for Unity

[zalo]

Hello fellow devs!

During my own project, I wrote a script that takes in the images provided by the shiny new Image API and outputs the 3D positions of bright retroreflective markers in the form of a Vector3 array.

This allows you to attach retroreflective markers to whatever you like and track them as you would your hands!

Here's an example where I put a sticker on my cellphone camera:

The script can easily be dropped into any project. Make sure you have the Plugin (from LeapDeveloperKit_2.1.0+18736_win\LeapSDK\lib\UnityAssets) as part of your plugins folder, and attach the C# script to any object in your scene to act as "The Leap" parent where all your tracked points will come from.

You can then access the array of blobs from your own script by adding a handle to the receiving object's script with:

public CameraProcessing BlobScript;

And then by dragging the CameraProcessing instance from your "Leap Parent" to that slot.

Access x, y, and z positions with this syntax:

BlobScript.Blobs[0].x

Have fun!

DOWNLOAD HERE

BUY RETROREFLECTIVE MARKERS HERE (bottom of the page)

[zalo]

I was able to do some gun tracking using some markers on a wireless mouse using this script

[samrochon]

This is amazing! My team and I are trying to create a VR game with gun tracking and are currently using this wonderful script. The only problem is it creates a lot of lag, the profiler in unity shows the update function taking up 80% usage. Any ideas on how to fix this?

[zalo]

Ah, that was due to a recent update making the "Rectify" function in the Leap SDK ridiculously slow; I was calling it for each pixel to increase precision. At the cost of a little precision, I can make it so it only calls the function once per blob.

Here's an updated version of the script that should run more quickly on the newer Leap SDKs:
https://dl.dropboxusercontent.com/u/8557629/CameraProcessing.cs

I might have changed some other things, so let me know if it suddenly doesn't work as it did.

Thanks for your support!

[pthor]

Excellent modification and implementation!

[Drysdale]

Hi Zalo i'm fairly new to Leap motion and unity programing and use playmaker in Unity with Oculus Rift for development purposes, your tracking system seem ideal for me as i dont need the Leap fingers just a Vector3 position in time and space to control an already animated fencers hand (sword ), marker tracking may be the key to the accuracy problem , this post is quite dated now, does it still work or is there an updated version ! Best Drysdale

[zalo]

I'm pretty sure it should work with the latest APIs...

The only change they made recently was to have the images access as a property rather than a method, so it theoretically should still work.

If you place the script on an object in the scene, the list of blobs should show up publicly under the script's properties. If you hit play while looking at those properties, you should be able to see the list populate with a few spurious blobs from your hands getting too close.

It's worth testing this before ordering some retroreflective material. If you don't see some blobs pop up or it spits out an error that you can't figure out, let me know. The script is fairly self-contained so it shouldn't break that often...

[Drysdale]

Hi Zalo thanks for your prompt reply !
I tried copy pasting your script to a new script C+ and attaching that to a game object, but just got a bunch of compiler errors so probably done the wrong thing :// .
Do you maybe have a Unity scene with it attached, working which i can then work off of

Best D

[zalo]

Sorry it's taken me so long to test it out.

I'm interested in hearing what kinds of errors you're getting; I just plopped it into a new project with the newest Leap SDK and it seems to be working fine (though I had to change the camera offset from 0.2 to 0.02 and the Blob Size Cutoff to 50000 (I changed the original file to reflect this)).

Here's what I did:
1) Created a new Unity 5.1.1 Project.
2) Installed the Leap Runtime and dragged the Plugins folder from the "Unity Assets" folder in the SDK into my project.
3) Created a new C# script called "CameraProcessing.cs" and pasted the contents of the script into the file.
4) Dragged the script onto a new game object.
5) Enabled "Draw Debug" in the script's properties.
6) Made sure my leap was plugged in and that the runtime was running.
7) Hit play and watched the cool debug lines from the editor viewport...

If you're still having problems somehow, I suppose I can send you the project.

[Drysdale]

Hi Zalo
Thank for the reply!
Did what was stated above,pasted into C sharp script and got the following error message in console :/
andAssets/CameraProcessing.cs.cs(18,1):
error CS1529: A using clause must precede all other namespace elements except extern alias declarations
I'm not having much luck with this.
Thanks for you efforts anyway!
Best D

[zalo]

Dude the file extension on your file is ".cs.cs"... (I suppose I was too literal in the steps)
In Unity's file browser, file extensions are hidden so they're not necessary to include in the name of the file.

Assuming that's not the issue, the script itself still doesn't sound like the problem; that error happens when the keyword "using" is placed in the middle of a file; the script doesn't do this. I suspect something like this would be possible if one of the other files in your project didn't terminate properly and caused the usings at the beginning of the CameraProcessing script to appear in the middle of a file (assuming C# reads files back-to-back as a continuum).

Here's a clean example project; if it happens here then it's a very weird bug indeed.

[Drysdale]

Eureka that seems to work fine now !
Thanks for the project file download !
Now to buy some retro reflective markers
Thanks again .
Best D

[AhmadAmin]

Hi Zalo
I am pretty much amazed by your work, great job.
I was wondering how to add the rotation to my object not just the position ?

Thanks in advance.

[zalo]

If you look at the picture of the mouse earlier in the thread, you can see I have 3 different tracked markers.

Unity has a function called "LookRotation": http://docs.unity3d.com/ScriptReference/Quaternion.LookRotation.html

I basically set:
1) the position equal the marker in the bottom left
2) the forward direction equal to the difference between the bottom left and bottom right markers
3) and the up direction equal to the difference between the bottom left and top left markers

And I can sort which marker is which based on their uneven relative spacing. This gives me full 6 DoF tracking.

I hope that clarifies things for you!

Note: This script is broken in the Orion beta, you have to use it with SDK 2.3.1 and below...

[AhmadAmin]

Thanks a lot Zalo for your reply !
How to know the blob index for each marker ?
Does the script work with Orion but with low accuracy or it doesn't work at all ?

Thanks in advance.

[CaptainNova]

Hi. What camera I can use? I have 4 ps eye cameras. I start script and nothing event. How work with this script?

[zalo]

@AhmadAmin There have been some changes in Orion that I haven't been able to fully compensate for yet... so it's completely broken until I can figure those out... You're welcome to try porting it yourself though (the main issue is in getting good rectification out of the new API's distortion correction...)

@CaptainNova certain elements of the script might work for tracking with the PSEye... but you'll have to rewrite a lot of it yourself to get the same 3D Tracking functionality without a Leap. That said, you're partially in luck. I have another script lying around that will take in webcam data (like that from the PSEye) and do basic blob tracking on it:

https://dl.dropboxusercontent.com/u/8557629/CamTracker.cs

You're on your own for setting it up though; it's been years since I touched the script.

[AhmadAmin]

Thanks a lot Zalo
I wonder how to know the blob index of each marker ?

[zalo]

They're randomly assigned, starting from 0.

If you want to do it like my gun example, you have to re-ID the markers each frame, yourself, by how far away they are from the other markers (after you mount them in such a way that they're all different distances from each other).

[shakirajane]

wow thats pretty good to hear