It turned out to be a very small value in the Camera Near Plane (0.01) that I think XR Rig sets by default - or maybe i lowered it time ago.
My guess is rounding issues lead to these artifacts when calculating the culling if the light is inside a structure. (This is just my shot in the dark - I'm sure you experts can have a better explanation)
Increasing camera near plane to 0.1 solves the issue. Tested in many different scenes with random lights and distances and the issue was gone.
An interesting observation - the point where everything goes dark shows this incremental clipping around the problematic rotation... keeping rotating the camera eventually un-clips it from the opposite side.
