Detection of ArUco Markers using OpenCV and Leap Motion IR cameras (step by step guide)

[assafr]

Hi,

In this post I will explain how I was able to detect ArUco Markers using Leap Motion using the new Orion API.
This is a sample of an ArUco marker:

After searching the web and this forum and ending up writing some code myself - here is how you do it.

First you need to get a version of OpenCV that support ArUco Markers detection, as I some of the users of this forum don't compile OpenCV on a daily basis here is how to compile OpenCV:
1. Download the main OpenCV repo - https://github.com/opencv/opencv
2. Download a secondary repo that includes the code that finds the markers: https://github.com/opencv/opencv_contrib
3. Open CMake GUI, create a project for OpenCV, set a CMake param named OPENCV_EXTRA_MODULES_PATH to the path of the repo from stage 2 - to a subfolder in that repo named "modules"

4. Set the CMake param "BUILD_EXAMPLES" to true.
5. Compile OpenCV, it compiled.

The next steps are more on the topic of the post:

1. I looked at the OpenCV project named "(sample) detect_markers" as it is a good sample of what I wanted to do.

2. Now I modified the Leap Motion Orion sample named "GLUTImageSample" to show the markers.
   I added this function:
   

   
   
   And called it from "OnImage":
   
   

And this have given me this result when running "GLUTImageSample" (the added thing is the "id=1" text at the marker):

1. Now I tried to use LeapCameraMatrix and LeapDistortionCoeffs but could not - as they are not implemented currently in the Leap Motion SDK and you will get a missing symbol error while linking.
   

   

2. I wanted now to find the 3D position of the corners, I found the following for the "pre-Orion" Leap Motion SDK - Calculate the Direction to an Image Feature
   I translated this to Orion and modified the "GLUTImageSample" sample.
   I added code to get the markers as I did in the other sample, I also a call to OnImage as in the other sample.
   Before getting the 3D position I checked that I have the same amount of markers detected on both cameras, I guess what is needed here is a set of some sort to verify you get the same markers on both cameras, as in many many cases only one camera detects the markers.

Then I added code that gets the 3D position using LeapPixelToRectilinear api:

And was able to see the 4 corners of ArUco Markers (I added the text after taking the screenshot):

So that is about it, Leap Motion is not that good for Detection of ArUco Markers, I am going to try the following to improve the detection:

1. Use ArUco that are more simple, not 6X6 but 4X4 as the IR image is low res.
2. I am going to try to improve the image I get from the IR camera, not sure how, if I will find something I will update this post.
3. I may try to add an external IR light - as the image seems dark when far from the Leap Motion sensor. I will update if I will do it.

I wanted to share the full source code of my modifications to the sample with this forum - but as the samples source code have this as the header of the file - I can't:

/******************************************************************************\
* Copyright (C) 2012-2017 Leap Motion, Inc. All rights reserved.               *
* Leap Motion proprietary and confidential. Not for distribution.              *
* Use subject to the terms of the Leap Motion SDK Agreement available at       *
* https://developer.leapmotion.com/sdk_agreement, or another agreement         *
* between Leap Motion and you, your company or other organization.             *
\******************************************************************************/

Leap Motion should consider removing this header from the samples - as this makes it problematic to share code that is based on the SDK samples between Leap Motion users.

Reference:

1. OpenCV - Detection of ArUco Markers https://docs.opencv.org/trunk/d5/dae/tutorial_aruco_detection.html
2. Leap Motion - Calculate the Direction to an Image Feature - https://developer-archive.leapmotion.com/documentation/cpp/devguide/Leap_Images.html#calculate-the-direction-to-an-image-feature
3. This is the Leap Motion blog post that started me on this topic - as they hint there this is possible - http://blog.leapmotion.com/leapuvc/

[link1375]

Hi thanks for your tutorial. I am working on C# and using the LeapInternal namespace to replicate the 3D positioning with LeapPixelToRectilinear.

private void ComputeMarkerRotationAndPosition(List<Mat>[] cornersArray)
{
    Vector3[] cornerPoints3D = new Vector3[4];

    for (int idIdx = 0;  idIdx < ids.height(); ++idIdx)
    {
        LEAP_VECTOR[] slopes = new LEAP_VECTOR[2];

        int[] camPerspectiveTypes = { (int)eLeapPerspectiveType.eLeapPerspectiveType_stereo_left, (int)eLeapPerspectiveType .eLeapPerspectiveType_stereo_right};

        for (int cornerIdx = 0;  cornerIdx < cornersArray[0][0].height(); ++cornerIdx)
        {

            for (int camIdx = 0; camIdx < camPerspectiveTypes.Length; ++camIdx)
            {
                var corner = cornersArray[camIdx][idIdx];
                var pixelPoint = corner.get(0, cornerIdx);
                var leapVec = new Leap.Vector((float)pixelPoint[0], (float)pixelPoint[1], 0);
                LEAP_VECTOR leapPoint = new LEAP_VECTOR(leapVec);

                slopes[camIdx] = LeapC.LeapPixelToRectilinear(IntPtr.Zero, (eLeapPerspectiveType)camPerspectiveTypes[camIdx], leapPoint);
                print(slopes[camIdx].ToLeapVector().ToVector3().ToString());
            }

            LEAP_VECTOR leftSlope = slopes[0];
            LEAP_VECTOR rightSlope = slopes[1];

            float cameraZ = 40 / (rightSlope.x - leftSlope.x);
            float cameraY = cameraZ * rightSlope.y;
            float cameraX = cameraZ * rightSlope.x - 20;

            cornerPoints3D[cornerIdx] = new Vector3(cameraX, cameraY, cameraZ);
        }
    }

    touchObject.transform.position = cornerPoints3D[0];
}

I have not clue what I should pass for the connection parameter. In C# this function requires an IntPtr and so I passed IntPtr.Zero. The result of print(slopes[camIdx].ToLeapVector().ToVector3().ToString()); is (NaN, NaN, 1.0). Have you any idea what I can do to get right coordinates?