Multiple Devices: on getting frames from one device

passervr 2018-12-30 12:08:10 UTC #1

I downloaded the LeapDeveloperKit_4.0.0+52238_win, installed the included software and started working with it using the LeapC library. (The mentioned multidevice Unity Modules were not present in the zip file)

Using Windows 64-bit, it does detect both my Devices (I'm getting a eLeapEventType_Device msg for each device), but then I receive Tracking messages for only one device.

I've worked through the examples (polling/callback etc.) but I did not found any clue how I will get tracking frames from any other camera.

The initialization code I use is the following (derived from the CallbackSample.c):

	ConnectionCallbacks.on_connection = &OnConnect;
	ConnectionCallbacks.on_device_found = &OnDevice;
	ConnectionCallbacks.on_frame = &OnFrame;

	connectionHandle = OpenConnection();
	{
		LEAP_ALLOCATOR allocator = { allocate, deallocate, NULL };
		LeapSetAllocator(*connectionHandle, &allocator);
	}
	LeapSetPolicyFlags(*connectionHandle, eLeapPolicyFlag_Images | eLeapPolicyFlag_MapPoints, 0);

What am I missing here?

jselstad 2018-12-31 15:42:03 UTC #2

A couple things:
1) There’s a policy flag that tells the service you want access to data from multiple devices. The relevant CSharp code for that is here:

github.com

leapmotion/UnityModules/blob/feat-multi-device/Assets/LeapMotion/Core/Plugins/LeapCSharp/Connection.cs#L150


  _leapConnection = IntPtr.Zero;


  Frames = new CircularObjectBuffer<LEAP_TRACKING_EVENT>(_frameBufferLength);
}


private LEAP_ALLOCATOR _pLeapAllocator = new LEAP_ALLOCATOR();


public void Start(bool multiDeviceAware = false) {
  LEAP_CONNECTION_CONFIG config = new LEAP_CONNECTION_CONFIG();
  config.server_namespace = Marshal.StringToHGlobalAnsi("Leap Service");
  config.flags = multiDeviceAware ? (uint)eLeapConnectionFlag.eLeapConnectionFlag_MultipleDevicesAware : 0;
  config.size = (uint)Marshal.SizeOf(config);
  Start(config);
}


public void Start(LEAP_CONNECTION_CONFIG config) {
  if (_isRunning)
    return;


  eLeapRS result;
  if (_leapConnection == IntPtr.Zero) {

2) You must subscribe to each device you want data from. (Ideally you do this once per connection to the service (where the connection key is different for each connection); this allows interpolation to work.)

The logic for that happens here in the Unity Modules:

github.com

leapmotion/UnityModules/blob/feat-multi-device/Assets/LeapMotion/Core/Scripts/LeapServiceProvider.cs#L500




/// <summary>
/// Creates an instance of a Controller, initializing its policy flags and
/// subscribing to its connection event.
/// </summary>
protected void createController() {
  if (_leapController != null) {
    return;
  }


  _leapController = new Controller(_specificSerialNumber.GetHashCode(),  
    _multipleDeviceMode != MultipleDeviceMode.Disabled);
  _leapController.Device += (s, e) => {
    if (_onDeviceSafe != null) {
      _onDeviceSafe(e.Device);
    }
  };


  if (_multipleDeviceMode == MultipleDeviceMode.All) {
    _onDeviceSafe += (d) => {
      _leapController.SubscribeToDeviceEvents(d);

If you do subscribe to all devices from a single connection, you’ll be able to differentiate them via their deviceID (that comes along with the tracking message):

github.com

leapmotion/UnityModules/blob/feat-multi-device/Assets/LeapMotion/Core/Plugins/LeapCSharp/Connection.cs#L285


  LEAP_POLICY_EVENT policy_evt;
  StructMarshal<LEAP_POLICY_EVENT>.PtrToStruct(
    _msg.eventStructPtr, out policy_evt);
  handlePolicyChange(ref policy_evt);
  break;


case eLeapEventType.eLeapEventType_Tracking:
  LEAP_TRACKING_EVENT tracking_evt;
  StructMarshal<LEAP_TRACKING_EVENT>.PtrToStruct(
    _msg.eventStructPtr, out tracking_evt);
  handleTrackingMessage(ref tracking_evt, _msg.deviceID);
  break;
case eLeapEventType.eLeapEventType_LogEvent:
  LEAP_LOG_EVENT log_evt;
  StructMarshal<LEAP_LOG_EVENT>.PtrToStruct(
    _msg.eventStructPtr, out log_evt);
  reportLogMessage(ref log_evt);
  break;
case eLeapEventType.eLeapEventType_DeviceLost:
  LEAP_DEVICE_EVENT device_lost_evt;
  StructMarshal<LEAP_DEVICE_EVENT>.PtrToStruct(

Also, for reference, the UnityModules Multidevice code is in the github repo that the installer is found in:

passervr 2019-01-01 09:44:16 UTC #3

Great! Thank you. I have it working now in both Unity and C. The tricks which did it:

Unity:
I only downloaded the MulticoreDevice Service (LeapDeveloperKit_4.0.0+522238_win) because that is where the blog pointed to. The whole UnityModules GitHub archive (one folder level up) includes the MultiDevice code. Look for the Multi Device (Desktop) scene found in Assets/LeapMotion/Core/Examples.

C:
For multi device tracking, You need to open the connection with eLeapConnectionConfig_MultiDeviceAware:

ConnectionCallbacks.on_connection = &OnConnect;
ConnectionCallbacks.on_device_found = &OnDevice;
ConnectionCallbacks.on_frame = &OnFrame;
//
LEAP_CONNECTION_CONFIG connectionConfig = LEAP_CONNECTION_CONFIG();
connectionConfig.server_namespace = NULL;
connectionConfig.flags = eLeapConnectionConfig_MultiDeviceAware;
connectionConfig.size = sizeof(connectionConfig);
connectionHandle = OpenConnectionWithConfig(&connectionConfig);
{
	LEAP_ALLOCATOR allocator = { allocate, deallocate, NULL };
	LeapSetAllocator(*connectionHandle, &allocator);
}
LeapSetPolicyFlags(*connectionHandle, eLeapPolicyFlag_Images | eLeapPolicyFlag_MapPoints, 0);

Next to that, you need to subscribe to the events for each device, which can be done in the OnDevice function:

static void OnDevice(const LEAP_DEVICE_INFO *props, const LEAP_DEVICE device) {
	LeapSubscribeEvents(*connectionHandle, device);
}

Now the real experimentation can begin!

billmccrary 2019-02-12 16:14:45 UTC #4

Hi - we are very interested in this for VR - should that be doable at this point in time? Since there is no VR example scene for multiple controllers, we're not positive that VR is even supported for this experimental feature.

I pulled down the bits from github and have the desktop scene working with 2 controllers no problem. I don't see a VR sample scene, and I'm struggling to get something to work well (I can get both hands rendering but getting them in sync is not clear).

Can we please get an example VR scene with multiple leap motion controllers? Ideally with 1 desk mounted controller and 1 head mounted.

I'll keep fiddling with it for now and will try to remember to post any success I may have...

passervr 2019-02-13 12:57:04 UTC #5

VR should be no problem I expect, because both devices work independently.

The fusion of the data from both trackers is not available in the SDK, so you need to to this yourself.

In my product (Humanoid Control) I do a lot of sensor fusion, therefore I expect that it won't be a huge problem to do it. I just haven't started working on it.

kameveh934 2020-07-31 12:26:54 UTC #6

Hello passervr, I'm working with four leap motion devices and want to get the best data based on the confidence value of those Hands. But the problem is the confidence value always remains 1 even when no hands are detected. what is the problem here??? Thank You . Your help would be much appreciated.

passervr 2020-07-31 13:09:49 UTC #7

Hi there!
Unfortunatley the confidence level for the hands is always 1 by design. It is a missing feature in the Leap API, see https://developer.leapmotion.com/documentation/v4/group___structs.html#aa0faeb8d4c2f3d042719165273d6abbd

I believe it gave other values in an older implementation of the library but with the current implementation they somehow cannot produce a sensible value any more. Because of backward compatibility it is still there with the static value 1. (That is just my guess, I have no inside information about this)