Tried with pre-undistored images, calibration behavior is the same with even bigger error.
This is the result I've got:
Some parts of images was aligned properly, but apparently amount of distortion is more than OpenCV can handle. If anyone was able to do successful calibration, please let me know. For now, I will try to do detection in plugin and use rectify() function from Leap Motion SDK.