Looks like we sorted out the issue.
We are using Unity 2020.2.3 and were trying to simply test out the controller interaction of the Varjo XR3 without the headset (remote development). I only have the Rift S controllers available with me and wanted to see whether they could work as a reference point.
Seems like not, and the Leap Motion Package for Varjo isnt required for controller integration, only hand tracking integration (which makes sense).
Thank you for getting back to me