A few ideas:
Make the mesh a child of the gameobject with a collider, rigidbody and interactionbehaviour
Have an event listener that adjusts rotation OnGraspedMotion, OnGrasped, or on GraspStay
Use one object as a handle (with rigidbody, Interaction) and put update the position of your model (not childed) OnUpdate to match the handle.