That some reason is motion sickness. There has to be consistency with your perception, else it develops into compounding vection feelings. It tend not to apply for vehicular controls hence sim usage.
That varies from person to person. I have played games with keyboard and mouse in VR (e.g. Polynomial 2, or the unofficial GTA 5 VR mod), and it works great for me.
