Form & Function 3D: Leap Motion Application for Undergraduate Anatomy Education

[Bren]

Hi all,

I am a student in the Biomedical Communications program at the University of Toronto. I just want to share a screencast of my Masters research project using the Leap Motion Controller. This is me interacting with a virtual cat heart I modeled from CT data.

Here's a short blurb from my proposal:

In any 3D virtual environment, selecting objects with a mouse becomes difficult as the 3D scene becomes densely populated and structures are occluded. In industries such as game development, designers can work around this limitation by creating open environments with large amounts of space between objects. Biocommunicators are not afforded the same luxury since models of anatomy represent environments in which there is no true empty space. Organs, vessels, and nerves always sit flush with adjacent structures. Consequently, accessing occluded objects requires a user to either distort his or her point of view, or remove obstructions entirely. Although individual structures can be observed, important information about the spatial relationships between different anatomic structures is lost.

For students in undergraduate anatomy courses, understanding spatial relationships is an essential learning goal. Traditionally, these relationships have been taught using dissections, however, rising costs and decreased availability of specimens have led many educational institutions to transition into using interactive, virtual anatomic models. Furthermore, even when dissections are available, spatial information of deep structures, such as the vessels of the circulatory system, are often impossible to preserve.

The goal of my project is to develop a NUI-based educational tool to help undergraduate anatomy students understand the spatial relationships of the circulatory system. By integrating a gesture-based NUI with accurate 3D anatomic models, students will have the freedom to explore individual structures while maintaining the fidelity of spatial information.

My project is quickly wrapping up and I would love to get feedback from the developer community. You can read a bit more about my project at http://brendanpolley.com

[TheoArmour]

Brendan

Beautiful work!

Good 3D interaction and it looks like you've got some 2D menus working. Certainly a lot of effort.

Did you ever think of using two hands? One hand to move and rotate the heart and the other to point or dissect or whatever.

Organs, vessels, and nerves always sit flush with adjacent structures. Consequently, accessing occluded objects requires a user to either distort his or her point of view, or remove obstructions entirely. Although individual structures can be observed, important information about the spatial relationships between different anatomic structures is lost.

Bingo!

This is going to be the fun part of the near future. Using shaders in new ways. Inventing 3D magnifying glasses. Devising ways of augmenting unreality and more.

Looks like you have a good grasp on this.

Got any code you can show?

Theo

[Bren]

Hi Theo,

Thanks for your feedback! I completely agree with your statements about where we need to be heading in the near future. The biggest challenge associated with my project was building in strong visual cues to covey the depth of objects in a 3D environment. For example, the photo below shows my heart model using Unity's built-in Transparency shader (left) and the custom transparency shader I eventually made (right).

The built-in transparency does not do a great job at respecting strong depth cues like occlusion, cast shadows, and reflected light. My shader relies heavily on rim lighting and layers of culling masks to preserve contours.

---

I did quite a bit of usability testing on using two hands compared to one, and complex vs simple gestures. In the end, the two handed interactions were viewed as strenuous and it was difficult to get smooth transitions between navigation and selection tasks without distorting the view of the model. Interestingly, even with nav and selection functions simultaneously enabled, students tended to focus on one task at a time; either manipulating the position of the model, or identifying specific structures, but rarely both at the same time. In the end, the one hand approach was well received, easiest to learn, and resulted in the fastest workflow.

---

I'm happy to share some of my code and other design strategies. Is there anything in particular you would be interested in?

[matthewtytel]

Very cool.
Yea that rim lighting makes it it a lot easier to see inside the organs. Mind sharing the shader?

I also like what you did with the target zooming in combination with the light source when you're close.
I've been doing a bit of experimenting on visual depth feedback with shadows but it's not perfect.

[Bren]

Thanks Matthew!

For the shader, I expanded on a fantastic Unity Asset called Nice Transparency: https://www.assetstore.unity3d.com/#/content/7793

This took care of the occlusion/artifact problem I got with the stock Unity shader. I editing the shader to include rim lighting capabilities. There's a good example for making rim lighting shaders in Unity's documentation: http://docs.unity3d.com/Documentation/Components/SL-SurfaceShaderExamples.html

Then it was a matter of adjusting the rim light power to make certain structures more salient. In my case, I wanted the inner structures to draw focus so the power is almost double that of the outer surface.

I hope this helps. Unfortunately, I can't post my shader since it's built on a purchased asset but hopefully you can follow similar steps to get the look you want.

Best of luck!

[tylerz]

I love the look and feel of the project and its interaction!

With the speed of interaction, it is hard to discern how each of the feedback UI components perform. Namely, there are four triangles around the cursor sometimes, sometimes not, sometimes they expand, sometimes they contract. Can you elaborate on how they work?

I figured out that hand depth is controlling the green cursor sphere, which is a green light source, illuminating material in its vicinity. Simply pointing causes the organ or component under the cursor to provide context information and become selected with a highlighting. Anything else I missed?

Fantastic work. Look forward to discussing further.

Cheers,

Tyler

[Bren]

Thanks for your feedback, Tyler.

The triangles around the cursor respond to the proximity of the cursor to selectable objects. A ray is cast directly forward from the cursor and the distance from any collision controls the triangles. The closer the collision, the closer the triangles move toward the cursor. The triangles persist if an object is selected but there are other, deeper objects that can be selected if the user moves the cursor forward. In these cases, the triangles lose their fill colour. The idea was to use the triangles to guide the user to the surfaces of selectable objects but also encourage them to explore deeper areas.

I have a separate script that detects if the cursor is contained within the mesh of an object and that triggers the highlight and label events.

I hope that explains things well. The UI still needs some work. Let me know if I can answer any other questions. I hope to post the final build soon so people can actually try it out.

[Bren]

Thanks for all the great feedback. Here's an updated demo video. The app now includes models of two, three, and four chambered hearts, blood flow simulations, and a quiz mode.

Hoping to submit to Airspace soon.

Form & Function 3D Trailer: http://youtu.be/llkfKLAy2Rg

[drunkenminion]

That looks really great! Thanks for sharing. Is that the Leap V2 tracking?

[matthewtytel]

damn looks so good..

[Kothe]

Hi Bren.
Congratulations for your work.
I love your Depth Sensing HUD. Unfortunately I have no real programmer skills, I use PlayMaker for Unity to avoid coding. So I am far away to make it work. I´m waiting to Hutong Games, developers of PlayMaker, to create custom
actions for the Leap Motion.
It is too much if I ask for a step by step set up or the simplest example in Unity ? Cause I really got lost just with the
few steps you told.

Any help will be very apreciated.

Thx

[Bren]

Hi Kothe,

I'm happy to help out. I started building Form & Function 3D back in the V1 days of Leap Motion. A lot has changed since then and there are lot more resources available for developers just getting started. If you haven't done so already, you should grab yourself a copy of the Leap Motion Assets from the Unity Asset Store. There are some great examples in there to get you started.

If you want to make your own depth sensing HUD you could probably set up something similar using PlayMaker. The meat of my sample code is in the castRayForward() function. I haven't used PlayMaker but I believe you can set up Raycast states. So if you cast a ray forward from the Finger TipPosition you can then measure the distance between the TipPosition and any collider hit by the raycast with RaycastHit.distance. Once you get hooked into this, you can make any kind of visual representation of depth you please.

I hope this helps. Let me know if you have any other questions. If I can ever find the time, maybe I'll submit my whole UI system with HUD to the Asset Store so others can use it. Best of luck!

[Kothe]

Hi Bren,
Thankyou very much. I will work harder on this.

Merry Christmas and a Happy New Year!

Best Regards

[haukesa]

Hi @Bren may I ask how you got the swipe gesture working this well? It is one of the few Leap moves that do feel intuitive and I would need it to control a virtual slideshow. Could you share the code and how did you program the slider dots? Do you use an approach like this to detect allover movement: http://thomaswinther.no/techtalk/leap-motion-full-hand-gesture

It would be indeed great of you to submit just the HUD to the Assets store. I am trying to recreate also the selection with the raycast as you described.

[Bren]

HI @haukesa , happy to share. Here's the meat of the script I used to detect swipes. It's very similar to the examples in the docs but I only grab the first gesture registered in the gesturelist and I added a cool down of 0.75 sec.

void Update () {

	GestureList _gestureList = m_leapController.Frame().Gestures();

	if(_gestureList.Count > 0){
			Gesture gesture = _gestureList [0];

			switch (gesture.Type) {

			case Gesture.GestureType.TYPE_SWIPE:
				SwipeGesture swipe = new SwipeGesture (gesture);

				if(Time.time - _lastGestureTime > 0.75f && swipe.Direction.x < 0){
					rotateMenuLeft(); // rotates menu to left
					_lastGestureTime = Time.time;
				}
				else if(Time.time - _lastGestureTime > 0.75f && swipe.Direction.x > 0){
					rotateMenuRight(); // rotates menu to right
					_lastGestureTime = Time.time;
				}
				else{
					Debug.Log("No swipe direction");
				}
				break;
			default:
				print ("Unknown gesture type.");
				break;
			}
		}
	}
}

I also played with the minimum velocity required for swipes to register by changing the config settings of the controller (i.e. yourController.Config.SetFloat("Gesture.Swipe.MinVelocity", 3000f);)

Hope this helps! Let me know if you have any other questions.

[esabo]

Sorry for posting so late...

can you please explain where and how should I declare the _lastGestureTime variable?

[Bren]

No problem. Sorry about that. I should have mentioned _lastGestureTime is just a global float.

I set the cooldown time between gestures to 0.75 seconds but feel free to play around with this value and see what feels right for your users.

[esabo]

Thanks for your help.

[faradinaannisa]

Hi, what a great project! How did you make the menu UI? Would you mind sharing the code?