Need Python Image API help

[adreyer]

I'm having trouble working with the new Image API for Python. I basically want to continuosly store the images or even a video from the cameras but don't really know how to achieve that.
I know that I can access the data with frame.images.data but that's where I am stuck. Looks like that gives me a pointer to a Bytearray?!
With that I tried to create images using the Python imaging library and image.frombytes or image.fromarray but I don't get that to work.

I'm new to Python, so anything might help. Could anyone provide me with some lines of code that would solve my problem or help me in other ways?! Google didn't help so far.

Thanks in advance!

[rjoyce]

I'm working on this too. I couldn't get frombytes or fromarray to work either. I've got something working that converts it to a numpy array by manually indexing it (slooooooow) and then using that as the basis for PIL.Image.fromarray

The code looks something like this:

leap_image = frame.images[0]
arr = np.zeros((leap_image.height, leap_image.width))
for i in range(leap_image.width):
    for j in range(leap_image.height):
        arr[j][i] = leap_image.data[(j*leap_image.width)+i]
image = PIL.Image.fromarray(arr)
image = image.convert("RGB")
image.save("test.png")

I'm working on creating a little image viewer that undistorts using an OpenGL shader, but this part is seeming to be much harder than it should be! Using this current method would never work as it's way too slow... 5-10 seconds to copy every frame!

Hopefully someone more experienced in python can help us efficiently get the data.

[Joe_Ward]

It doesn't solve the speed issue, but here is how you can make a PIL.Image object:

def on_frame(self, controller):
    frame = controller.frame()
    leap_image = frame.images[0]

    barray = bytearray(leap_image.width * leap_image.height)
    for d in range(0, leap_image.width * leap_image.height - 1):
        barray[d] = leap_image.data[d]

    image = Image.frombytes("L", (leap_image.width,  leap_image.height), buffer(barray))
    image.save("test.png")

FWIW, the image.data object is not a built-in type, so it doesn't have any interfaces to allow it to interoperate with typical Python functions (see http://www.swig.org/Doc1.3/Python.html#Python_nn48).

[rjoyce]

Thanks Joe, this at least looks cleaner than my numpy way. As you say, it unfortunately will still be too slow for real time analysis.

I'm going to look into trying to call the C OpenGL functions directly which may work to accept the data memory directly, and this may work for other C-backed libraries (e.g. OpenCV). I'm not sure how it would work with PIL though if you need to save an image out, but perhaps there are C libraries that you can use through ctypes that will buffer to a video as the OP wanted.

[rjoyce]

OK I'm on to something, using ctypes I was able to get my glTexImage2D calls to accept it (I haven't been able to get it to work quite right yet with distortion so I'm not sure if it's copying the data correctly yet, but that's another story).

Anyway I went back to PIL to take a look to see if I could get it to accept a ctypes object. Based on this stackoverflow question I was able to get this code to run, but the image is partially corrupt as you can see, so something is going wrong.

Actually I just tested it some more and I realized that sometimes it comes out not as corrupt, so it may be that LeapMotion frees the memory before we use it or something...

Here is the code I'm working with right now:

leap_image = frame.images[0]
imagedata = ctypes.cast(leap_image.data.cast().__long__(), ctypes.POINTER(leap_image.width*leap_image.height*ctypes.c_ubyte)).contents
image = PIL.Image.frombuffer("L", (leap_image.width, leap_image.height), imagedata, "raw", "L", 0, 1)

[Joe_Ward]

On the plus side, it is 100 times faster!

I'm seeing different corruption -- as if a row of pixels isn't starting at the right place in the data.

What would be the ideal data type of image.data for your uses?

[rjoyce]

Joe, It seems like I'm able to get this to an OpenGL texture. I'm having trouble with my shaders at the moment that it doesn't want to paint it out properly, so here is a screenshot from apitrace that shows the texture in the GPU. (The red oval was added to cover my face )

So now I'm worried to say what data type I would like, as it works for my purpose, however it takes a lot of finagling and I still haven't figured out how to get the distortion data copied properly. Maybe swig can output ctypes directly?

When I get a working demo I'll post it, but the relevant part right now is this:

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, leap_image.width, leap_image.height, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, ctypes.c_char_p(leap_image.data.cast().__long__()))

[Bolty]

yes nice work there that getting much less messy

[adreyer]

did you make further progress rjoyce?

[ksmith3412]

I had success using numpy.frombuffer:

leap_image = frame.images[0]
imagedata = ctypes.cast(leap_image.data.cast().__long__(), ctypes.POINTER(leap_image.width * leap_image.height * ctypes.c_ubyte)).contents
image = np.frombuffer(imagedata, dtype = 'uint8')
image.shape = (image.height, image.width)

This will create a 2D numpy array containing the image very quickly and without corruption.

[Joe_Ward]

leap_image.data.cast().long()

I suspect that this is not a valid address. In my tests it appears that this technique gives you data from old frames rather than the newest one, so it is subject to overwriting and corruption. The good news is that this will be fixed in our next release by the addition of Image.data_pointer and Image.distortion_pointer which do give you valid data (and replace the cryptic incantation, leap_image.data.cast().__long__()).

[Joe_Ward]

See the docs of Image.data_pointer for an example on how to convert the image to a ctypes or bumpy array: https://developer.leapmotion.com/documentation/python/api/Leap.Image.html#Leap.Image.data_pointer

[201821030184]

good afternoon,
this give me a error in image = Image.frombytes("L", (leap_image.width, leap_image.height), buffer(barray))
ValueError: tile cannot extend outside image

what's wrong with this?thanks in advance