The collisions in the VR mode are off, so that the camera movement always matches your head movement. If your head moved while the camera did not it could cause motion sickness.
If you move through the scene using teleporting (triggered with the index finger), the teleport won’t allow you to pass through walls, though.