UnrealEngine4 Plugin

[swhittle]

As a FYI you can see my Animation Blueprint here:

www.dropbox.com/s/pgf3xm4lz9vk1pp/Screenshot%202014-05-19%2018.13.19.png

Sorry if the text is a little blurry, but you can see that I am pulling the data across as-is without any issues. Could you give me an example of what you mean about joints being offset wrong? From what I remember Leap's way of doing bones is not quite as you'd expect if you come from Maya or Max, but given it's been 3 months since I've done much with the plugin I'm a little hazy on details and short on time to switch back to that branch just now to check.

[jaxzin]

That was really helpful because I'm not able to use the data directly. For all Rotators on the left hand the roll is reversed and offset by 180 for me. Once I accounted for that the hand/palm movements matched up. When I added the Transform Bone for the index bones/joints I get this (highlighted with some blue marks):

You can see the 3 bones/joints are offset from the hand and overlapping each other. I'm using the first person skeletal mesh that comes with UE4.

Also what type of object are Thumb, IndexFinger, etc? Your blueprint is cleaner because I need to break each LeapPointable into its LeapBone.

[zalo]

Hmm, I'm trying to port the Plugin to 4.4 and the latest API (the one with images). The vast majority of the changes are just OVERRIDE -> override. The BindDynamicDelegates function gave me a little trouble, but I could get it to compile by the removing the override entirely in its .h

Of course, now I'm suspicious the engine doesn't call the original form of the function, which I suspect is causing trouble when the component tries to register (it doesn't register successfully). This seems to prevent acquisition of frames.

It spits out an error like:

LogActorComponent: UnregisterComponent: (/Engine/Transient.LeapInputComponent_1) Not registered. Aborting.

I suppose I'll have to start over and just try upgrading the 4.3 version to 2.1.0...

[jaxzin]

@zalo you can find a diff of my working copy which includes the override changes as well as a few other tweaks.

https://github.com/jaxzin/UnrealEngine/compare/Oculus-VR:4.3...4.3-leap

I'm using the 4.3.1 branch + Oculus' changes for their 0.4.1 SDK + Leap 2.0.5. Stuff changes fast in just a week so I haven't upgraded to Leap 2.1.0 yet or UE4.4.

[jaxzin]

@swhittle I had a breakthrough tonight using the starter skeletal HeroFPP. With some scaling of the bones, I finally have a working hand. Along with the rotation tweaking I was already doing, it turns out I needed to position the bones using different data points. Specifically Hand.wristPosition() (instead of palmPosition) and Bone.prevJoint() (instead of nextJoint). I exposed those and sent you a pull request.

P.S. Here's the rotation tweaking I needed for the left hand....

...and the right hand....

[turick]

@swhittle @jaxzin I have been reading so much and seeing so many demos with leap motion and Unity... I just came here to gripe about the Leap Motion team only providing support for Unity. However I am very pleasantly surprised to see you guys working on a plugin for UE4.

I'm very interested in getting this working. Has anybody been able to complete a fully functional plugin built for 4.4? Can this be rigged to a full skeleton so the arm moves with the hand?

[jaxzin]

Yup, I got it working with the 4.4 branch last night:

https://github.com/jaxzin/UnrealEngine/tree/4.4-leap

Follow the README and after you build, copy the Leap.dll and Leap.lib to the Binaries dir (the build doesn't currently do it for you).

Also, I have rigged the forearm using the Arm data from the Leap, but I'm new to UE so my next task is to figure out how to animate the elbow to shoulder. Currently my version separates from the skeletal model.

[turick]

@jaxzin That is great news! I'm not sure if I'll get to it tomorrow, but I'll try to get it up and working this weekend for sure!

[turick]

@jaxzin this is actually my first time rebuilding the engine. Will it always be necessary to have a custom engine build or is there anyway this could simply be a plugin in the future?

[zalo]

I just got it working; just take the Source/ThirdParty/LeapMotionLib folder, the plugin in the plugins folder, and copy the .dlls to the binaries folder and you're set.

Thanks jaxzin! Now we just need some Image support...

[turick]

FYI I had some issues with an Oculus library. The info on this page solved it -- basically just had to grab the latest version from the Oculus website: https://answers.unrealengine.com/questions/85773/undefined.html

[turick]

I was able to successfully build the forked engine and launch the editor from Visual Studio. However, if I create a new C++ project, it opens yet another VS instance and everything slows to a crawl. At that point, however, I'm not sure if I should create a new blueprint based off a C++ class or if I need to recreate the BPs shown in this thread?

Also, I tried just copying the 3rd party leap motion lib folder, the plugin folder, and the .dlls, from the forked engine folder to the installed engine folder. Now I can't create any C++ projects... I get an error stating it couldn't create the project file. It does let me create a blueprint project, however. Unfortunately I need a C++ project.

So after launching the rebuilt engine from VS and adding the plugin, I am able to add the controller to a character blueprint. Would any of you be able to share the blueprints / animation blueprints you've created to get the hand working? Or another thing that would be really cool, is a LeapMotion template! So you just fire it up, hit play, and start waving your hands around

[jaxzin]

I'm working on a template, but its really not ready for consumption yet. Maybe by the end of the long weekend I can clean it up for sharing.

[jaxzin]

I started working on Image support last night, but I'm new to Unreal Engine and returning to C++ after 12 years away. I'm having trouble figuring out how to convert the Image data to a UTexture2D. If anyone with more experience wants to help, my branch is here:
https://github.com/jaxzin/UnrealEngine/compare/jaxzin:4.4-leap...add-images

More info on what I'm having trouble with:
https://answers.unrealengine.com/questions/91111/how-to-create-a-utexture2d-outside-of-game-thread.html

[turick]

Completely understand on the template. I'll really be looking forward to trying it out when you get one working and rigged up with a full skeleton! Looks like you've found some help over on the ue4 answer boards with your current issue. Good luck and thanks for what you're doing jaxzin!

[jaxzin]

First buggy but working version with the Images API is available on my fork, tag is leap-images-1.0.

Known issues:

1. Getting the UTexture2D requires a call to a function in the LeapInputComponent (GetLeapImageAsTexture).
2. Distortion maps are 8-bit.
3. The image and distortion maps are first copied to TArrays which is REALLY inefficient but its the only way I could make it work so far.
4. A new UTexture2D instance is created on each frame, I'm trying to figure out how to do this more efficiently.
5. The distortion maps are captured on each frame, this is in line with the API but must be really inefficient in practice if they never change.
6. Compiling a Blueprint with a LeapInputComponent crashes the editor with a memory access violation within the Leap.dll
7. Closing the UE4Editor crashes with a memory access violation within the Leap.dll

Here's my blueprint that uses the GetLeapImageAsTexture function. I haven't had a chance to clean up the project yet.

And a capture of what the textures look like applied to 4 different different planes (left, right, distortion maps on top). I still haven't figured out how to use the distortion map to correct the camera images.

/cc @zalo, @turick

[zalo]

I'll look into implementing dewarp on Tuesday, but if you want to find it sooner, this post has the GLSL code that the Unity example uses to dewarp using the disortion map:

Use of Distortion() to get an OpenCV usable calibration map

[num3ric]

Hi Stephen,

I've been trying to get your add-images branch working with a Leap to no avail. If I enable the plugin and restart my project, I get the following crash:

Any idea what the cause would be?

Cheers,
Eric

Edit: Note that I don't need image support. I tried 4.4-leap but experienced the same crash.

[Joe_Ward]

I still haven't figured out how to use the distortion map to correct the camera images.

I just added a shader dewarp example to the docs: https://developer.leapmotion.com/documentation/skeletal/cpp/devguide/Leap_Images.html#correction-using-shaders

This relies on the availability of a 32-bit floating point texture format. If you can't use that, then you have to encode the distortion map values into the rgba components of an 8-bit RGBA texture (and use two textures for lookup instead of one). The Unity shader example does encode/decode step, but relies on a Unity-specific function/macro DecodeFloatRGBA(). (Still it isn't an uncommon technique for passing extra data to shaders, so there are plenty of examples out there.)

[turick]

I'm in the same boat -- I don't need image support but am very interested in just getting the tracking data rigged to a skeleton. This is an optional add on for my project so it's hard to find the time to dedicate working on this but really hope someone gets a working plugin/template up and going. Still wish the leap motion team would make a decision to officially support this and not just go unity only.