> fringe cases (how many people actually plug in a controller and a mouse at the same time..?)

There are good reasons to have both, and no reason to unplug either of them except to work around a buggy game.