I ran into this cool little device that looks tailor made for Leap Motion controlled robotics.
http://www.pololu.com/product/1350
It connects to the USB port for Windows and Linux (there does not appear to be support for Macs yet) and allows you to control up to six servo motors with C++ programs. It actually looks pretty straight forward. I can already imagine how easy it would be to use two servos with the Hello World and Polou's intro program that would beat a physical drum to the up and down movement of your hands. I think I could do that in about an hour. A simple robotic arm could be built with four servos that follows the movements of your hand by simply tracking the XYZ motion of the hand and a fist to grab an object by closing the claw servo.
My simplistic idea is to point a web cam over the robotic arm, make the Leap Motion control program small, and simple watch the web cam video to move the hand and pick stuff up, move it somewhere else and drop it.