Light probes and point lights do not have a notion of direction, unlike for example spot lights, so rotating them has no effect. This is the reason why the rotation gizmo is available only for spot lights, area lights and sun lights. For these light types you will notice a switch at the bottom of the screen allowing you to change between translation and rotation.