Sdk 2.1 (raw data) get pixel position (xyz)

[paranoio]

So i was waiting for this feature for so long, so now that it is here how can i get a pixel 3d position ?

Using kinect i use the depthStream to get a pixelAt (x,y) and find whats its depth (z)
but couldnt understand how this works with the leapmotion :
https://developer.leapmotion.com/documentation/skeletal/cpp/devguide/Leap_Images.html

im guessing first i need to correct image distortion, then get pixel(x,y) and then use the brightness ?

i dont think brightness is consistent data since light enviroments will affect these values ?

will it be necessary to use the left/rigth images to calculate the 3d position of a pixel ?

[tylerz]

You would most likely want to use OpenCV and supply it the stereo images which it can convert to per pixel depth data using disparity mapping. When you download the OpenCV SDK it comes with examples for processing stereo images but you also need to generate calibration data. Then, I found the following article very useful in selecting parameters to get reasonable results.
http://www.jayrambhia.com/blog/disparity-maps/

[paranoio]

Yes it looks like using opencv is the most simple way to go ,
thought it will be good to not have to add another library to my project.

i will give it a try, thanks.

[flyingattack]

In this case using a library like OpenCV to turn the raw stereo images in to a 3D depth map is necessary since Leap has said that they are taking those raw images and extracting depth information in a way that does not involve a depth map (which is why they can expose the image but not the depth map since one does not exist).

[maddddeline]

Is constructing a disparity map the best way to go from a 2D pixel to 3D coordinate?

Draw Tracking Data over Image shows how to use image.warp() to go from 3D point to 2D pixel. Is there the inverse of this built into the API?

Basically, how would I manually reconstruct the skeletal data from the raw images?

[Joe_Ward]

The inverse of warp() is rectify(). You would have to match up pixels from each of the stereo images and then use rectify() to get the slopes of the lines running from the cameras toward the source of the light for a pixel. You can then find the intersection of those lines in space to get the 3d point.

[Joe_Ward]

A couple of additional points. Two image pixels that form a stereo pair corresponding to the same 3D point will always be on the same horizontal scan line. The distance between the centers of the Leap Motion cameras is 40mm (subject to change in future devices).

[maddddeline]

I think I'm missing a core concept, Joe. Can you clarify?

I've found two corresponding pixels pairs (one from each of the stereo images) and used rectify() to get the two corresponding rays. I'm able to find the intersection of the rays, however, since rectify() only returns a 2D vector, the intersection point will only occur on the XY plane. It seems like I'm misusing the function ...

Any example code would be helpful!

[Joe_Ward]

The vector contains slopes, not positional coordinates. The first coordinate is for the slope to the long axis of the camera, the second angle is to the slope to the short axis.

[maddddeline]

It's still not clear to me how to use 2 slopes (the short and long axis slopes) to define a Ray ... I'm more familiar with vector notation.

So in the API, there's the example code:
Vector slopes = image.rectify(new Vector(horizontal, vertical, 0));
//Use the slope values ...

How do I use the slope values to find the corresponding Ray?
Would I use the formula for a plane z = mx + ny + c, where m is the horizontal slope, n is the vertical slope, and c may be the sensor location on the x-axis(?) ... but again, I'm not sure how this formula gets me to a 3D Ray.

[Joe_Ward]

It is essentially spherical coordinates. Imagine you are the camera. To point to any object in your field of view, you only have to move your arm horizontally by a certain angle and then vertically by another angle. Thus two angles are sufficient for describing the ray.

I'm not enough of a mathematician to go directly from slopes to your plane equation -- I'm not sure what c represents and it seems like you would need additional constraints to go from a plane to a line. However, the wikipedia article linked above does describe how to go from spherical to cartesian coordinates:

r is the length of the ray to your point (this is unknown for one camera)
Ax is the angle found by the horizontal slope
Ay is the angle found with the vertical slope

then:

z = r * cos(Ax)
x = r * sin(Ax) * cos(Ay)
y = r * sin(Ax) * sin(Ay)

Since the rays from the two cameras point to the same 3d point:

Zleft = Zright
Xleft - 20 = Xright + 20 (to account for the offset of the two cameras)
Yleft = Yright

[Joe_Ward]

I had a chance to work out some easier equations last night when my daughter borrowed my computer as a render mule for her video project, leaving me only with a pad of paper....

By definition:

SlopeH = x/z
SlopeV = y/z

From the camera baseline value (40mm) we also know:

xLeft - 20mm = xRight + 20mm

And we know that z is the same for both cameras.

Substituting in the slope values for the two cameras to find z:

z * SlopeHLeft - 20 = z * SlopeHRight + 20

and working things out:

z = 40/(SlopeHRight - SlopeHLeft)
x = z * SlopeHLeft + 20 = z * SlopeHRight - 20
y = z * SlopeVLeft = z * SlopeVRight

Note that the x, y, z coordinates we have been using are relative to the camera coordinate system, not the Leap Motion coordinate system. To get the coordinate in Leap coordinates:

xLeap = xCamera
yLeap = -zCamera
zLeap = yCamera

And a code example. This code takes the finger tips, calculates the pixel in the camera images that where the finger tip is located. It then gets the slopes from those pixel locations using rectify() and then recalculates the 3D positions (kind of pointless except for testing the math from end to end):

Leap::FingerList frameFingers = frame.fingers();
for(Leap::FingerList::const_iterator fl = frameFingers.begin(); fl != frameFingers.end(); fl++)
{
    //Convert finger tip position to a ray from the camera POV
    Leap::Vector tip = (*fl).tipPosition();
    int i = 0;
    float horizontal_slope_left = -(tip.x + 20 * (2 * i - 1))/tip.y;
    float vertical_slope_left = -tip.z/tip.y;
    i = 1;
    float horizontal_slope_right = -(tip.x + 20 * (2 * i - 1))/tip.y;
    float vertical_slope_right = -tip.z/tip.y;

    Leap::Image image_left = frame.images()[0];
    Leap::Image image_right = frame.images()[1];
    Leap::Vector warped_left = image_left.warp(Leap::Vector(horizontal_slope_left, vertical_slope_left,0));
    Leap::Vector warped_right = image_right.warp(Leap::Vector(horizontal_slope_right, vertical_slope_right,0));

    Leap::Vector slopes_left = image_left.rectify(warped_left);
    Leap::Vector slopes_right = image_right.rectify(warped_right);

    //Do the triangulation from the rectify() slopes
    std::cout << "Triangulate with rectified slopes: " << std::endl;
    float z = 40/(slopes_right.x - slopes_left.x);
    float y = z * slopes_right.y;
    float x = z * slopes_right.x - 20;
    Leap::Vector position = Leap::Vector(x, -z, y);
    std::cout << "Tip: " << tip << " calculated: " << position << std::endl;

    std::cout << "#############################" << std::endl;
}

[maddddeline]

Thanks for taking the time to explain this Joe ... extremely helpful!

[dexteritydesign]

How did you find corresponding pixel pairs in two images? I'm also trying to get the 3-D coordinates for every pixel in the steroimage. Thank you!

[Joe_Ward]

How did you find corresponding pixel pairs in two images?

That's the question, isn't it? Look up stereo correspondence. In general, you won't be able to get the 3D coordinates for every pixel in the images because there won't be enough detail to match up points in both pairs. One fact that helps is that two matching points will always be on the same line in both images.

[dexteritydesign]

Are there alternative ways to obtain 3D information of specific pixels? I used cv2 SGBM and the disparity map from undistorted images is very noisy.

[Joe_Ward]

I'm not sure I understand your question. Alternatives to semi-global block matching? I'm not a computer vision scientist, but it seems obvious that there are many alternatives to this particular algorithm.

[dexteritydesign]

Sorry I was confused about how the provided code example works. Am I correct to say that once I have two matching pixels p_L and p_R from images left and right, I can construct vectors V= [x,y,0] with x,y being the horizontal and vertical pixel coordinate in the image and use image.rectify(V) to find the slopes, then use the slopes to find the 3D coordinates? If that is the case I do not really need depth information of the pixels. Or would constructing a disparity map help me find the matching pixels?

[Joe_Ward]

Am I correct to say that once I have two matching pixels p_L and p_R from images left and right, I can construct vectors V= [x,y,0] with x,y being the horizontal and vertical pixel coordinate in the image and use image.rectify(V) to find the slopes, then use the slopes to find the 3D coordinates?

Correct.

Or would constructing a disparity map help me find the matching pixels?

Generating a disparity map, as I understand it, is essentially finding the matching pixels in an image.

[ashish090494]

@Joe_Ward
To get the 3d coordinates using the triangulation, do we need to also un-distort the pixels first and then use rectify() and the triangulation step, or is the undistortion step being done by the rectify() function.
Thanks!