Pilots hands in aviation simulator using LM

Flyingmark 2014-04-10 19:53:31 UTC #1

Hi!,
I Am an airline pilot And 737/320 simulator developer. I Am exploring the potential integration of virtual tools ( like Oculus rift And VR gloves) with realistic airline cabin mock-ups.
I Was looking for a hands tracking solution And I Came across with Motion leap.
My final intention is to implement (together with VR glasses like Oculus Rift) an add on to flight simulator (FSX or Xplane 10) virtual 3D cockpit adding pilot's virtual hands/arms moved with leap motion software. I dont see in your app airspace site Anything in these dirección yet, but I saw the "ORIENTATION" video presentación with the finger-like movements and displays. These software gets quite close to what i Am actually need. Can you confirm this ORIENTATION presentación is not part of the Apps? Can these cyborg- type hands be modified to a human like ones?
Do you work or know of Any software developer that is able to do the Job?
And finally a technical question: is possible to use two motion leap devices at the same time (one below on a horizontal pedestal panel And the other on top of it, just above, or both would interfere each other? The reason behind the need for an upper device, is to be able to track the finger movements when they are down on the very same level where the lower track device is placed, And therefore due to converge tracking area limitations, not been able to be tracked unless an upper device is implemented.
Can you give me some advice?
Many Thanks in advance

Joe_Ward 2014-04-10 22:50:54 UTC #2

Hi Flyingmark,

The Orientation app is included with the Leap Motion software installed with the drivers and service, etc, so it isn't in the Airspace store on its own.

The current API gives you the positions of the hand and the position of the tips of any fingers it detects. The Orientation app uses the same API available to external developers.

You can use a the data from the API to control a human-like hand model. It is challenging to make things move with a life-like appearance, though, given just the finger tips. (I think it can be done, but it isn't a matter of simply moving the model finger tips using the finger tip positions.)

Currently, only one Leap Motion device is supported per computer. Depending on what you are doing, you can use virtual machines (or even multiple physical computers) to get data from multiple devices. Interference is possible if the IR illumination from one device is shining into another device. I've seen experiments that involve mounting the Leap Motion device directly to the Oculus, so that could be another option.

PLeoncini 2014-04-11 09:31:37 UTC #3

Flyingmark,

I'm familiar with articulating a virtual hand by the Leap Motion input, so I can help you on reconstructing hand kinematics from the few device data. Then I don't know whether importing hand parts in form of SimObjects in FSX then sending them pos/orients from outside is an efficient way to operate or if is there a better way to do it.

What I'm not familiar, even if I used to play with FSX, is with what to do with a virtual hand in FSX. My question is: are FSX or X-Plane ready for making a virtual 3D hand model to interact with cockpit panel switches, knobs, or yoke, etc.? I.e. the 3D mesh of a finger phalanx should be collision-checked against a certain panel area or against the 3D model of the object itself - does FSX anything like this? or is there a simpler mechanism to activate (less complex, more effective)?

Anyway just let me know if and how can help you, especially if somebody else take care of the FSX integration mechanisms.

Paolo

Flyingmark 2014-04-11 15:50:59 UTC #4

Thank u both joe and paolo for your quick response.
Paolo, mi email is markdichtl@hotmail.com should you want yo talk with me personally.
Nowadays there are a number of interfaces that allow external hardware ( like trottles and flight yoke/ sidestick in Airbus type and many more..) work perfectly with FSX/XPLANE software. My intention is to also use them to control by real touch the essential flight controls, and to use LM only to replace mouse button click to replicate a finger tip touch on a panel switch. Also some cockpit switches would require a knob switch to be moved left or right with two fingers grasp. But basically these two simple movements would be needed. No complex fisical manipulation( wich is already taken care by real flight controls).
I see some challenges, some of them already mentioned:
In case we can succesfully put in place a set of moveable arms, can we import them as an "add on" to FSX / XPLANE? I.e adding superimposed arms over the already running FSX 3d cockpit background, with only mouse click interaction to the finger tips.This is something i dont know if it 'd work, yet.

Can we create a 3d coodinate matrix mounted around the real fisical cockpit mock-up, and assign to every switch on the panels a given spatial coordinate. Then using LM, based on this cockpit-matrix, move the finger around and when the finger reaches a given switch in the real space, assign this 3d coordinate to the display's 2d coordinate where the switch ( what you see with OCULUS ) is in the virtual cockpit. So when the finger/s reached both in the fisical 3d cockpit and on the 2d virtual cockpit the same switch, you make a mouse click on it to operate it.
The final challenges I see is , when using OCULUS RIFT, to solve the brains difficulty to see and grasp something with 3D glasses and reach it in a 2d visual
Any opinion greatly wellcome!
Mark

Flyingmark 2014-04-12 15:51:45 UTC #5

Searching the web, i just found some Xplane/LM compatibility : http://youtu.be/CnvwuhBO6K4

PLeoncini 2014-04-14 12:19:44 UTC #6

Quite sure, yet don't know details. The idea is to have a 3D model of hand/arm in FSX, and to feed their 3D parts with position/orientation by SimConnect or whatever other FSX-enabled comm library. Yet nobody will prevent it to penetrate cockpit objects though... other than the z-buffer.

Ok, you proposal is to activate the cockpit command by 2D mouse click emulation assuming (to be able) to set up an external 3D->2D mapping. It might work - a further investigation on the FSX add-on integration mechanisms would lead to hopefully find such a function natively within the hosting environment so to leverage the xml cockpit definition which FSX has imported at A/C loading time.

I don't foresee a big problem, yet I could have misunderstood your concern - in your proposal there's no real 2D visual, rather there is the 2D projection of the 3D scene, and the two are coherently aligned (i.e. the graphical and the "perceived" ones are the same).
Conversely if you propose to operate against a 2D plane it would only work for the front cockpit (the one vertical in front of the pilot), not e.g. for the central horizontal deck.
Perhaps our conversation is going to become a bit off-topic for the LeapMotion so we should probably move to private (leoncinip@gmail.com).

Greetings, Paolo.

Beezle 2014-05-30 23:41:48 UTC #7

Hi there,

first of all great that you guys see the same need for this as I do. I was wondering if it would be easier to realise to use the virtual hands as some kind of screen overlay rather than integrating hands as simobjects in FSX.

Also if you guys haven't heard of Prepar3D, it's based on FSX's source code and actively developed by Lockheed Martin and as of v2.2 comes with native Oculus Rift support.

Under the hood it's been largely improved over FSX and 99% if FSX add-ons are compatible.

hesynergy 2014-12-19 13:28:05 UTC #8

This conversation should never have been stopped!

I think the possibility of the LeapMotion interfaced with P3D is much needed not only by the user community but internally by Lockheed Martin who have taken over the Microsoft Flight simulator X (FSX)enterprise.

...frankly I cannot believe that there is not some very intimate communication happening as we speak between LeapMotion and Lockheed Martin's Prepared3D development crew...it's probably hush-hush, not unlike the "Skunk-Works" used to be or perhaps still is.

Take a walk through the sky's of Prepar3d (P3D)

So I would like to re-awaken and continue with the idea of interfacing LeapMotion with the already existing 3-D/2D control surfaces inside the Prepar3D virtual cockpit .

Just so you know, Lockheed Martin has produced the

real F-22 Raptor and the F-35 fighter jets…

Just saying…

Chas

logic28 2015-02-21 23:37:14 UTC #10

This conversation not only should not have stopped, but should restart with fresh inputs.
Just like Flyingmark, I also was thinking of implementing Leap Motion within P3D Flight simulations.
I have worked with motion control, motion capture and track control in Cinematography and it works wonders when you have to attach a virtual object to a real person for the screen arts.
So it would be interesting to know if you guys have had any more info on the subject before I go and duplicate the searches.

here is my email if needed:

fly@tempotechnology.com

Many thanks for your contribution.
Mauro Andreini (Maz)

hesynergy 2015-12-05 16:30:47 UTC #11

Mario!
I am so sorry for having missed your response...the auto alert system here seems to be wanting.

We MUST continue to engender interest in this worthy goal...

Let's keep it going!... I have some programming experience, but I am really rusty.

On another note:
As innovative as the Leap Motion is, I would love to be shown that I am wrong in the assumption that there is not a LM...hmmmm LM = Leap motion= Lockheed Martin....I digress...
I have yet to hear of a reliable Leap Motion emulation of full mouse control.

Why is, where is that???

Chas