Record a chess game using leap motion

[user-254204206]

Before I buy an LM, I need to know if it is capable of the following app (I am a C# developer).

1. I want to place LM on the side of a 20" x 20" chess board, with 3.5" pieces. Will the LM track a hand anywhere directly above the board, down to (say) and inch and above to (say 6")? If the inverted pyramid does not cover this volume (with the LM at the side the board), can I turn LM and mount it on a small tripod to get the coverage I need?

2. Is it feasible to detect where a hand picks up a piece (i.e. the location with enough accuracy to detect what square on the board the piece was), and similarly detect the square where the piece was released? If I can get the position of the piece (while in the hand) just before it is released (i.e. detect when the hand is now empty) that would probably work.

I already have logic to detect if the move was valid, given the source and destination squares (according to the rules of chess), so can resolve some ambiguity if more than one move was possible (due to tracking imprecision).

The purpose of this app: I run a chess club for kids who do not want to record their moves on paper. I want to record the game so that I can go over it with the kids. I have written lots of chess programs and have years of experience. I just need to know if LM can do this. What are the major problems and what suggestions do you have for solving them?

I have some other apps in mind too. I really need to know the precise dimensions of the inverted pyramid and whether it is possible to mount it on a tripod, tilting it to cover a volume that is not directly above the unit. The reason is that I need to track the hand very close to the table and somewhat far from the LM unit.

[upretirementman]

Why not place your chess board as a near transparent graphic on glass and have the Leap Unit below that. There is a smaller setup that uses a cube as space which is located inside the inverted cone space, I am told. It's in the Leap API.

BTW, a movie camera would also do what you are needing.

[user-254204206]

I need to use existing (tournament approved) chess boards and sets. I want to record the moves in a form a chess engine on the computer can understand. Immediately after the game we can go to any move in the game and ask the engine for the best move. We can play a few moves against the engine before we revert back to the game and continue the analysis. I have all the chess programming in hand: given a source and destination square, the logic is written. I just want to capture the moves via LM, find the source and destination square with logic I would write. The rest is done. I am beginning to doubt that LM can cover the entire board (20" x 20"), even if mounted on a small tripod, and track the hand as it grabs and releases a piece in such a way that the source and destination square can be determined.

[upretirementman]

My monitor is 20 inches across. I held a bottle of Bayer Low Dose over the Leap Unit and it does not register at all anywhere because it cannot see any fingers or finger like objects in the view. They have to be horizontal not vertical over it for it to see the object. Perhaps if the chess board was displayed on the monitor with a camera, where you could point to beginning and ending positions you might get something working. Take a look at some chess programs and see how they do it by clicking and dragging each piece around on screen chess board.

[user-254204206]

Thanks for the input. Could I ask you for a favor? Layout a 20" by 20" square. Put any object at each corner (a coin will do). Put the LM to the side of the square (i.e. outside the square, on any edge, say to the right). Pick up each coin and place it in the center. Did LM detect the hand motion at each corner? If not, could you turn the LM on its side and raise it a bit, so that it points down on the "board" (still keeping it to the side of the "board"). See if LM tracks your hand from each corner to the center. If you could, try picking up an object (something about 2.5" high, pointy like a chess piece). How does the LM track the object (while in the hand, from corner to center). Do you think it is precise enough for me to detect the position of the object when the hand picked it up and when it released it? Thanks!

[lm_dedelhart]

A cautionary note - recording hands is easy -- however the pieces themselves will throw a LOT of noise at the Leap Motion Detector, which is better at finding hands in open air than in a jungle of objects.

If you were to model the chess board in 3d (unity, three.js) and JUST have the hands to detect, you'd likely have more success.

[upretirementman]

As you can see my setup for testing Leap use in Chess Schools. My recommendation is a solid connection between Leap Unit and chess board. Perhaps build a one piece that holds the Leap unit and the chess board so that neither move in relationship to each other. Before playing the first time, you have to initialize each location on the board with software. Software must have limits stated for each square, wherein when one touches the top of a piece in a square anywhere, the software know which square is activated and equivalent location dictionary values are known. Storage of beginning square and ending square is saved in a dictionary during game: moves.update(begin,end) (python)

I used my finger on top of piece to display location on screen. Software can record as location of chess piece. The height of each piece will be different. As using Leap on side height is really location horizontally. Possible software conversion to up/down/sideways correlation will be needed.

Download Using Leap with Chess set movie

[upretirementman]

You initialize each square by touching the top of the piece with your finger.

"Touch top of chess piece at square a1 now for 3 seconds and press enter"
"coordinates of a1 initialized"

For testing, you have the software print out the coordinates.

[user-254204206]

Thanks! If you can confirm that your setup covers a 20" by 20" board I will undertake this project. The chess sets and boards in our club cannot be changed (standard USCF sets). The calibration (as you suggest) can be undertaken before a game begins. I will have to use a small tripod and clamp for the LM, and place it far enough away from the board so that it will not be accidentally jostled by the players. Do you think LM can cover the 20" x 20" board accurately enough for the source and destination squares to be determined (via calibration and translation software you suggest)?

I appreciate your experiment. If you are a chess player you might be interested in the (free) software I have written: https://sites.google.com/site/fredm/ Site also has links to pictures of our club so you can see what I am dealing with.

[upretirementman]

OK, I tried something else. I mounted the Leap unit upside down about 11 inches above the board. It could track my finger down to 1 inch above the board! and about 20 inches side to side. So I imagine using velcro to attach it underneath a shelf and it would work if the board is underneath in an unmovable location ( perhaps set in a 1/2 inch framework around it to hold it in place. and you could switch from one to another if needed. )

I used the 3D Blender open source app to display the board on-screen. I would need some software to reverse the directions since it's mounted upside down. IE: when I moved my finger to the right, it displayed to the left.

x_dir_finger = - x_dir_finger is all that is needed to change display direction.
y_dir_finger = -y_dir_finger if other direction is opposite that is needed

[user-254204206]

This is interesting. Please remember that I must make the system unobtrusive. The players, as much as possible, should not have to change their behavior. They will not look at the computer screen, and cannot have their view of the board/pieces obstructed at all. Many of the players are just starting out, young kids, and I don't want to confuse them by giving them a different experience than when playing over the board. Only when the game is over would I direct their attention to the computer screen as we go over the game.

I think your experiments prove that the coverage of LM is, at best, 10" on either side of the unit. Thus putting the LM (even on a small tripod) somewhere to the side of a 20" board (where it cannot be knocked accidentally by a player) does not seem feasible. I guess the coverage is just not there. Thanks for your input -- it was very helpful.

BTW, I looked at the Kinect SDK. The unit is much more expensive, but would work nicely if it were not for its size (it is too big). If LM were to provide the same interface (Kinect delivers all objects in its view) and/or increase the LM coverage, it would support many more applications. They might add a standard tripod screw while they are at it.

[upretirementman]

Well I never said they had to use the screen whatsoever, in fact it could be programmed to be invisible. The display is not needed for playing, only for testing. The side view will not reach 20 inches total, only the upside-down mounting does. There is a Leap Motion book out now, covering C++ code, Java, and some MacOS, but mostly Windows.
It's called Leap Motion Development Essentials by Mischa Spiegelmock. Published by PACKT. It's the only Leap book out at the present time.

Blender is not needed also, just create your code in C++ and add to your own chess engine. BTW, there are many chess engines available for Blender for Windows. Just do a search for Blender Chess.

[upretirementman]

Try raising the height of the up-side-down mounting to expand your sphere of influence. I used the stated height because I have a two shelf unit screwed to my computer desk so I can work standing up for a while, as I had developed back problems because of poor posture.

That idea about the tripod mount is interesting. It could be made with MakerBot as a case that holds the Leap unit and has a place on the bottom to insert a tripod mounting nut. Perhaps that would bring about many more apps of various sorts.

Here is a Leap case created with a MakerBot 3D printer:

Another view of Leap Case: