A while ago, I had to make a networked game, and we used Leap rigged hands for giving users hands. I ran into some trouble trying to get things working across the network right away, but I eventually got everything working. Here's how I got it working.
Important note: This only allows for a visual representation of the rigged hands (they must be rigged hands). This doesn't provide any functionality for working with Interaction Engine.
Initial Setup
To start, make a new Unity project. Import the Leap Orion package.
Drag /Assets/LeapMotion/Prefabs/LeapHandController.prefab into the scene. It should spawn at 0, -0.17, 0.338. If not, place it there.
The camera won't be able to see it from there, so move the camera to 0, 0.379, -0.463. Set the rotation's x value to 5.46. Set the near plane to 0.1 and the far plane to 2.
Getting a Player Prefab
If you're familiar with Unity's UNET tutorial, you'll know about NetworkManager and auto-spawning Player Prefabs. Since we need a Player Prefab, we'll construct one and give it hand-tracking capacity:
Make a new GameObject, call it "HandPlayer".
Select the LeapHandController object. In the Inspector, there should be a gear icon in the upper left of the Transform Component's region. Click it and select 'Copy Component' from the drop-down.
Select "HandPlayer". Right-click the gear icon and select 'Paste Component Values'
Drag the LeapHandController object onto the HandPlayer object to re-parent it.
Nav to /Assets/LeapMotionModules/Hands/Prefabs/HandModelsNonHuman/ in the Project view. Drag the LoPoly_Rigged_Hand_Left and right prefabs into the hierarchy, making them children of HandPlayer.
Select LeapHandController in the hierarchy. In the Inspector, set 'Models Parent' in the Hand Pool component to HandPlayer from the hierarchy.
Expand the Model Pool array variable. Set the Size to 1. Expand 'Element 0' and change "Group Name" from blank to "Hands". Drag the LoPoly_Rigged_Hand_Left object from the scene into the 'left model' variable slot in the Inspector, and drag the LoPoly_Rigged_Hand_Right object into the 'right model' variable slot. Set 'Is Enabled' to true, and leave 'Can Duplicate' as false.
Create an /Assets/Prefabs/ folder if it does not yet exist. Drag HandPlayer from the scene hierarchy into this folder to make a new prefab. We'll be telling the network manager to spawn this later.
Create an /Assets/Scripts/ folder if it does not yet exist. Nav there in the Project view. Right-click in that folder and select Create->C# Script. Name it HandPlayer.cs
Select the HandPlayer object in the scene hierarchy. Drag the HandPlayer.cs script onto the Inspector region near the 'add component' button.
Common Pain Point: Ensuring only one Leap Hand Controller
One problem that you might run into if you just try adapting the Unity UNET tutorial to support Leap hands on your own is if you have multiple LeapHandController objects in the hierarchy at once, all tracking will stop. This error will continue even after you disable all but one of them. To prevent this, we'll need to ensure that our LeapHandController objects (and any related scripts like RiggedHand and RiggedFinger) get removed from net clients before they ever become active. We'll handle that by doing the following:
Select the LeapHandController, LoPoly_Rigged_Hand_Left and LoPoly_Rigged_Hand_Right child objects of HandPlayer. Click the checkbox in the upper left of the Inspector to disable them.
Open Up HandPlayer.cs. Replace everything with the following:
using UnityEngine;
using UnityEngine.Networking;
using System.Collections;
public class HandPlayer : NetworkBehaviour
{
[SerializeField] Transform leftHand;
[SerializeField] Transform rightHand;
EnableEventRelay leftHandEnabler;
EnableEventRelay rightHandEnabler;
[Tooltip("Remove these if we're not the player - they'll just get in the way.")]
[SerializeField] Object[] removeComponentIfNonLocal;
[SerializeField] GameObject[] removeGameObjectsIfNonLocal;
[SerializeField] GameObject[] enableGameObjectsIfLocal;
void Start()
{
if (!isLocalPlayer)
{
for (int i = 0; i < removeGameObjectsIfNonLocal.Length; i++)
{
if (removeGameObjectsIfNonLocal[i] == null) continue;
Destroy(removeGameObjectsIfNonLocal[i]);
}
// we need to get rid everything except the raw transforms
for (int i = 0; i < removeComponentIfNonLocal.Length; i++)
{
if (removeComponentIfNonLocal[i] == null) continue;
Destroy(removeComponentIfNonLocal[i]);
}
// specifically removing rigged fingers if non-local
Leap.Unity.RiggedFinger[] leftHandFingers = leftHand.GetComponentsInChildren
<Leap.Unity.RiggedFinger>(true);
Leap.Unity.RiggedFinger[] rightHandFingers = rightHand.GetComponentsInChildren
<Leap.Unity.RiggedFinger>(true);
if (leftHandFingers != null)
{
for (int i = 0; i < leftHandFingers.Length; i++) Destroy(leftHandFingers[i]);
}
if (rightHandFingers != null)
{
for (int i = 0; i < rightHandFingers.Length; i++) Destroy(rightHandFingers[i]);
}
gameObject.tag = "Untagged";
gameObject.name = "Player (Remote)";
}
else
{
gameObject.name = "Player (Local)";
leftHandEnabler = leftHand.gameObject.AddComponent<EnableEventRelay>();
leftHandEnabler.Enabled = new UnityEngine.Events.UnityEvent();
leftHandEnabler.Disabled = new UnityEngine.Events.UnityEvent();
leftHandEnabler.Enabled.AddListener(CmdLeftHandEnable);
leftHandEnabler.Disabled.AddListener(CmdLeftHandDisable);
rightHandEnabler = rightHand.gameObject.AddComponent<EnableEventRelay>();
rightHandEnabler.Enabled = new UnityEngine.Events.UnityEvent();
rightHandEnabler.Disabled = new UnityEngine.Events.UnityEvent();
rightHandEnabler.Enabled.AddListener(CmdRightHandEnable);
rightHandEnabler.Disabled.AddListener(CmdRightHandDisable);
for (int i = 0; i < enableGameObjectsIfLocal.Length; i++) enableGameObjectsIfLocal[i].SetActive(true);
}
leftHand.gameObject.SetActive(true);
rightHand.gameObject.SetActive(true);
}
#region Hand Enabling/Disabling
[Command]
void CmdLeftHandEnable()
{
if (isClient) leftHand.gameObject.SetActive(true);
else RpcLeftHandEnable();
}
[ClientRpc]
void RpcLeftHandEnable()
{
leftHand.gameObject.SetActive(true);
}
[Command]
void CmdLeftHandDisable()
{
if (isClient && isLocalPlayer) leftHand.gameObject.SetActive(false);
else RpcLeftHandDisable();
}
[ClientRpc]
void RpcLeftHandDisable()
{
leftHand.gameObject.SetActive(false);
Debug.Log("Left Hand Disable");
}
// right hand
[Command]
void CmdRightHandEnable()
{
if (isClient && isLocalPlayer) rightHand.gameObject.SetActive(true);
else RpcRightHandEnable();
}
[ClientRpc]
void RpcRightHandEnable()
{
rightHand.gameObject.SetActive(true);
Debug.Log("Rght Hand Enable");
}
[Command]
void CmdRightHandDisable()
{
if (isClient && isLocalPlayer) rightHand.gameObject.SetActive(false);
else RpcRightHandDisable();
}
[ClientRpc]
void RpcRightHandDisable()
{
rightHand.gameObject.SetActive(false);
Debug.Log("Right hand Disable");
}
#endregion
}
The code here is fairly straightforward, but to sum it up quickly, the Start() method handles all of the initialization. If the instance of HandPlayer does not belong to the local client, it will destroy the contents of the specified component and gameobject arrays. (We'll specify these components and GameObjects later.) It also will go through provided left and right hand objects and remove all the rigged fingers. It's faster to do this in code than to feed each one into the removeComponentIfNonLocal array manually in the inspector. It will also handily re-name the player objects to indicate if they are local or remote players.
Hide your code editor and go back to the scene view. Assign the Left hand and Right hand variables in the HandPlayer component to LoPoly_Rigged_Hand_Left and LoPoly_Rigged_Hand_Right gameObjects respectively. Drag the LeapHandController object into the 'Remove Game Objects If Non Local' array. We're most of the way to having the auto-clean setup, but the next bit requires some tricky editor work. With the HandPlayer object selected in the Inspector, click the lock icon just above the Inspector pane. You'll notice that the Inspector now no longer changes, no matter what you select in the scene hierarchy.
Since the goal here is to drag components from one object onto variable slots in another, we'll need another Inspector window. Above the Game View window, in the upper right corner you should see a small button with a graphic that has three lines and a tiny arrow. Click that button and a drop-down menu should show up. Click Add Tab->Inspector and you'll get a second Inspector!
In the scene hierarchy, select the LoPoly_Rigged_Hand_Left object. Click & drag the Animator component in the left Inspector (Hotspot for dragging components is the Name bar) onto the 'Remove Component If Non Local' variable of the right Inspector. Do this with the Rigged Hand and HandEnableDisable components as well.
Select the LoPoly_Rigged_Hand_Right gameobject. Click and drag the RiggedHand and HandEnableDisable components into the 'Remove Component if Non Local' variable of the right Inspector. We're all set with our second inspector, so click the arrow and lines icon you used to create the new inspector tab. This time, click 'Close Tab'. It's right under Maximize. Unlock the remaining inspector using the lock icon.
Also, we will want the LeapHandController object to be enabled if the player is local, so select the HandPlayer object in the scene hierarchy. Drag the LeapHandController child object onto the 'Enable Game Objects if Local' array.
We're all set with this object, so with it still selected, click the 'Apply' button in the Inspector to take all of our changes to the HandPlayer prefab and commit them to the prefab. Save the sene, then delete the HandPlayer object from the scene.
Setting up the NetworkManager
Create a new GameObject in the scene and name it "NetworkManager". Add a NetworkManager component and a NetworkManagerHUD component.
Select the HandPlayer prefab in the project view, and add a NetworkIdentity component, as well as a NetworkTransform component.
Back in the scene, select the NetworkManager again, and in the SpawnInfo section, drag the HandPlayer prefab into the NetworkManager component's 'Player Prefab' field.
Getting Transforms Across the Network
So now, our program is technically networked, although it doesn't do anything meaningful yet. To get it to transmit the hand data across the network. We've already added a NetworkTransform to the HandPlayer object, which is the beginning of this but there is much more to do. Unity will not automatically send child transforms of a NetworkTransform across the network (This is an optimization, not a bug). To get those transforms across the network we'll need to add NetworkTransformChild components. These components have a caveat that developers new to UNET might not intuit - they MUST be placed on the same GameObject as their corresponding NetworkTransform.
Now it should be noted that according to Unity Technologies, placing a bunch of NetworkTransformChild components on a GameObject for an entire skeleton is not the most efficient way of doing things. They even provide a NetworkSkeleton.cs file to do so. However, I was completely unable to get this approach working. So I did it the 'bad way'.
Normally, applying one NetworkTransformChild per bone would be an obnoxious process. However, I made a simple editor script to automate it. Make an /Assets/Editor/ folder if it does not yet exist and make a new script called NetworkSkeletalBuilder.cs. Give it the following code:
using UnityEngine;
using UnityEngine.Networking;
using UnityEditor;
using System.Collections;
using System.Collections.Generic;
public class NetworkSkeletalBuilder : EditorWindow
{
[MenuItem("GameObject/Network/Skeletal Builder")]
public static void ShowWindow()
{
EditorWindow.GetWindow<NetworkSkeletalBuilder>();
}
GameObject selection;
NetworkTransform netTransform;
bool multiSelectFail = false;
[SerializeField] Transform skeletalRoot;
void OnGUI()
{
if (selection != null)
{
if (multiSelectFail)
{
EditorGUILayout.LabelField("Not compatible with multi-selection!");
}
else
{
if (netTransform == null)
{
EditorGUILayout.LabelField(selection.name);
EditorGUILayout.LabelField("Object must have a NetworkTransform.");
if (GUILayout.Button("Try finding one further up the tree."))
{
NetworkTransform transformCandidate = selection.GetComponentInParent<NetworkTransform>();
}
if (GUILayout.Button("Add one to this object."))
{
netTransform = selection.AddComponent<NetworkTransform>();
}
}
else
{
skeletalRoot = (Transform)EditorGUILayout.ObjectField("skeleton root", skeletalRoot, typeof(Transform), true);
if (skeletalRoot == null)
{
EditorGUILayout.LabelField("Please assign a skeleton root object before trying to assign network transform children");
}
else
{
if (GUILayout.Button("Build tree"))
{
BuildTree(skeletalRoot);
}
}
if (GUILayout.Button("DELETE ALL CHILDREN"))
{
NetworkTransformChild[] children = selection.GetComponents<NetworkTransformChild>();
for (int i = 0; i < children.Length; i++) DestroyImmediate(children[i]);
}
}
}
}
else
{
EditorGUILayout.LabelField("To use this script, you must have a single GameObject selected with a NetworkTransform component.");
if(GUILayout.Button("Refresh"))
{
GetSelectionComponents();
}
}
}
void BuildTree(Transform skeletalRoot)
{
List<NetworkTransformChild> comprehensiveList = new List<NetworkTransformChild>();
comprehensiveList.AddRange(selection.GetComponentsInChildren<NetworkTransformChild>());
EnsurePathToParent(skeletalRoot, netTransform.transform, ref comprehensiveList);
ApplyNetTransChild(skeletalRoot, netTransform.transform, ref comprehensiveList);
}
void EnsurePathToParent(Transform child, Transform root, ref List<NetworkTransformChild> childList)
{
if (child.GetInstanceID() == root.GetInstanceID()) return;
if (!ChildListContains(child, childList))
{
NetworkTransformChild newChild = root.gameObject.AddComponent<NetworkTransformChild>();
newChild.target = child;
childList.Add(newChild);
}
EnsurePathToParent(child.parent, root, ref childList);
}
private static bool ChildListContains(Transform child, List<NetworkTransformChild> childList)
{
bool childListContainsTarget = false;
for (int i = 0; i < childList.Count; i++)
{
if (childList[i].target == null)
{
Debug.Log("Network Transform Child had no target!");
Debug.Break();
}
else
{
if (childList[i].target.GetInstanceID() == child.GetInstanceID())
{
childListContainsTarget = true;
break;
}
}
}
return childListContainsTarget;
}
void ApplyNetTransChild(Transform child, Transform root, ref List<NetworkTransformChild> childList)
{
if (!ChildListContains(child, childList))
{
NetworkTransformChild newChild = root.gameObject.AddComponent<NetworkTransformChild>();
newChild.target = child;
childList.Add(newChild);
}
if(child.childCount > 0)
{
for(int i=0; i < child.childCount; i++)
{
ApplyNetTransChild(child.GetChild(i), root, ref childList);
}
}
}
void GetSelectionComponents()
{
selection = null;
multiSelectFail = false;
if (Selection.gameObjects.Length == 0) return;
if (Selection.gameObjects.Length > 1)
{
multiSelectFail = true;
return;
}
selection = Selection.gameObjects[0];
netTransform = selection.GetComponent<NetworkTransform>();
}
void OnSelectionChange()
{
GetSelectionComponents();
}
}
To use it, we'll need to place our HandPlayer prefab back in the scene. Select the HandPlayer gameObject in the scene, then click GameObject->Network->Skeletal Builder. A new window should pop up. Click 'Refresh' if you get a warning about not having a selection. Ignore the DELETE ALL CHILDREN button for now - it is there in case you want an easy way of clearing all the NetworkTransformChild components on an object - useful for debugging. Expand the HandPlayer object, then expand the LoPoly_Rigged_Hand_Left object. Drag the L_Wrist object onto the 'skeleton root' field in the window, then click 'Build Tree'.
Expand the LoPoly_Rigged_Hand_Right object, then drag R_Wrist onto the skeletaon root field in the window. Click 'Build Tree'
Click 'Apply' in the inspector to apply the changes to the HandPlayer prefab. Save the scene, then delete the HandPlayer object from the scene.
Getting Enable/Disable State Across The Network
You might have noticed, if you tried playing the program as it currently stands, that the hands don't appear to anyone except the local client. To remedy this, we'll need to alert the other clients that the hands should be enabled and disabled when tracking data becomes valid or invalid.
Make a new script in /Assets/Scripts/ and name it EnableEventRelay. The entire file should be this:
using UnityEngine;
using UnityEngine.Events;
using System.Collections;
public class EnableEventRelay : MonoBehaviour
{
public UnityEvent Enabled;
public UnityEvent Disabled;
void OnEnable()
{
Enabled.Invoke();
}
void OnDisable()
{
Disabled.Invoke();
}
}
Now, we'll need to make another script. Name this one NetHandEnableDisable. Set it to the following code:
using UnityEngine;
using UnityEngine.Networking;
using System.Collections;
using Leap.Unity;
public class NetHandEnableDisable : NetworkBehaviour
{
[SerializeField] GameObject leftHand;
[SerializeField] GameObject rightHand;
// Use this for initialization
void Start ()
{
if(isLocalPlayer)
{
Debug.Log("Added local player events");
}
}
}
We're going to need to temporarily add the HandPlayer prefab back to the scene so we can make some changes to it. Do so now. Once you've done that, expand it so that we can get at the LoPoly_Rigged_Hand_Left and LoPoly_Rigged_Hand_Right objects. Drag them into the NetHandEnableDisable component's left and right hand slots respectively. Click apply to apply the changes, then delete the HandPlayer object from the scene hierarchy again. Save the scene.
We're ready to do our first attempt at a networked test. Click File->Build Settings and wait for the compile to finish. Once the build is ready, load one windowed instance of it. Click on 'Lan Host', and notice how this instance is the host and hand tracking works.
Now, start up another instance of the program. This time click LAN Client. Notice that the host's hands now appear visible to us as the client. That's it! You've officially got skeletal hands tracking across the network.
If you want to skip this and just download a working example project, go here: https://github.com/jcorvinus/LeapNetTutorial
NOTE: Client/server/command RPCS work the way you expect, with one minor caveat: client rpc calls are NOT mirrored to the server, so for clients with local authority (like our hand enable/disable script), clientRPC calls will need to be mirrored to the server
